This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Regular expression (regexp) parser
The AxoSyslog application can parse fields from a message with the help of regular expressions. This can be also achieved with the match() filter, by setting the store-matches flag, but the regexp-parser() offers more flexibility, like multiple patterns and setting the prefix of the created name-value pairs.
Note
The
regexp-parser() can create additional name-value pairs only if “named capture groups” are used in the regular expression, for example
(?<test_field>\w+). For more information, see “named capture groups” in
PCRE documentation.
For more information about regular expressions in AxoSyslog, see Regular expressions.
See also the related FilterX function regexp_search(), which returns the matches instead of creating name-value pairs, and takes a single pattern instead of a list.
For example:
Declaration:
parser p_regexp {
regexp-parser(
patterns( ... )
);
};
Example: Using a regexp-parser()
In the following example, the incoming log message is the following:
Apr 20 11:09:46 test_field -> test_value
The regexp-parser inserts the .regexp. prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log line connects the source, the parser and the destination.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser(p_regexp);
destination(d_file);
};
You can also define the parser inline in the log path.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser{
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination(d_file);
};
You can set multiple patterns:
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$", ".*other_format: (?<foo>.*)$")
prefix(".regexp.")
);
};
1 - Options of Regular expression parsers
The Regular expression parser has the following options.
flags()
|
|
| Synopsis: |
flags(“ignore-case” “unicode”) |
| Mandatory: |
no |
Description: The flags to apply for the parser. The available flags depend on the type() of the regular expression. For details, see Options of regular expressions.
internal()
|
|
| Accepted values: |
yes, no |
| Default: |
no |
Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.
patterns()
|
|
| Synopsis: |
patterns(“pattern1” “pattern2”) |
| Mandatory: |
yes |
Description: The regular expression patterns that you want to find a match. regexp-parser() supports multiple patterns, and stops the processing at the first successful match.
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser p_regexp{
regexp-parser(
patterns( ... )
prefix("myprefix.")
);
};
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
type()
|
|
| Synopsis: |
pcre, string, glob |
| Mandatory: |
no, defaults to pcre |
Description: Sets how the patterns() expressions are interpreted: as Perl Compatible Regular Expressions (pcre, used by default), literal string searches (string), or glob patterns without regular expression support (glob).