PostgreSQL csvlog

Available in AxoSyslog version 4.5.0 and later.

This parser processes messages in the PostgreSQL csvlog format. The following sample message is a multi-line message with embedded NL characters. This is a single, multi-line log entry that starts with the timestamp.

2023-08-08 12:05:52.805 UTC,,,22113,,64d22fa0.5661,1,,2023-08-08 12:05:52 UTC,23/74060,0,LOG,00000,"automatic vacuum of table ""tablename"": index scans: 0
pages: 0 removed, 4 remain, 0 skipped due to pins, 0 skipped frozen
tuples: 114 removed, 268 remain, 0 are dead but not yet removable, oldest xmin: 149738000
buffer usage: 97 hits, 0 misses, 6 dirtied
avg read rate: 0.000 MB/s, avg write rate: 114.609 MB/s
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,,,""

The postgresql-csvlog-parser() extracts the information from this message into a set of name-value pairs. By default, the name-value pairs have the .pgsql prefix.

@version: current

log {
    source { file("/var/log/pgsql.log" follow-freq(1) flags(no-parse)); };
    parser { postgresql-csvlog-parser() };
    destination { ... };
};

The postgresql-csvlog-parser() driver is actually a reusable configuration snippet configured to parse log messages using the csv-parser(). For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

FilterX has no PostgreSQL parser. Since this driver builds on the CSV parser, the related parse_csv() function is the closest building block.

Options

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->

on-type-error()

Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.

Accepts the same values as the on-error() global option.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)