metrics-probe

Available in AxoSyslog version 4.1.1 and newer.

metrics-probe() is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages.

You can configure the name of the keys and the labels. Note that the keys are automatically prefixed with the syslogng_ string. You can use templates in the values of the labels.

See also the equivalent FilterX function, update_metric(), which takes the counter name as its first argument instead of in a key() option.

The minimal configuration creates counters with the key syslogng_classified_events_total and labels app, host, program and source. For example:

parser p_metrics_probe {
  metrics-probe();

  # Same as:
  #
  # metrics-probe(
  #   key("classified_events_total")
  #   labels(
  #     "app" => "${APP}"
  #     "host" => "${HOST}"
  #     "program" => "${PROGRAM}"
  #     "source" => "${SOURCE}"
  #   )
  # );
};

This configuration results in counters like:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

You can query the metrics by running the following command:

syslog-ng-ctl stats prometheus

For example, the following metrics-probe() parser creates a counter called syslogng_custom_key that counts messages that have their custom_label_name_1 field set to foobar, and for these messages it creates separate counters based on the value of the custom_label_name_2 field.

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

This configuration results in counters like:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

Starting with AxoSyslog 4.4, you can create dynamic labels as well.

Options

The metrics-probe() parser has the following options. In addition to the options listed here, metrics-probe() accepts the value-pairs() options — such as cast(), exclude(), include-bytes(), key(), pair(), rekey(), and scope() — inside its labels() option. For details, see Dynamic labels.

frac-digits()

Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.

increment()

Available in AxoSyslog version 4.2 and newer.

Sets a template, which resolves to a number that defines the increment of the counter. The following example defines a counter called syslogng_input_event_bytes_total, and increases its value with the size of the incoming message (in bytes).

metrics-probe(
    key("input_event_bytes_total")
    labels(
        "cluster" => "`cluster-name`"
        "driver" => "kubernetes"
        "id" => "${SOURCE}"
        "namespace" => "${`prefix`namespace_name}"
        "pod" => "${`prefix`pod_name}"
    )
    increment("${RAWMSG_SIZE}")
);

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

key()

The name of the counter to create. Note that the value of this option is always prefixed with syslogng_, so for example key("my-custom-key") becomes syslogng_my-custom-key.

labels()

The labels used to create separate counters, based on the fields of the messages processed by metrics-probe(). Use the following format:

labels(
    "name-of-the-label-in-the-output" => "field-of-the-message"
)

Default value:

labels(
    "app" => "${APP}"
    "host" => "${HOST}"
    "program" => "${PROGRAM}"
    "source" => "${SOURCE}"
)

This results in counters like:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3

Dynamic labels

Available in AxoSyslog 4.4 and newer.

Dynamic labelling allows you to use every available value-pairs() options in the labels, for example, key(), rekey(), pair(), or scope().

For example:

metrics-probe(
  key("foo")
  labels(
    "static-label" => "bar"
    key(".my_prefix.*" rekey(shift-levels(1)))
  )
);

syslogng_foo{static_label="bar",my_prefix_baz="anotherlabel",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4

level()

Available in AxoSyslog version 4.2 and newer.

Sets the stats level of the generated metrics.

Note: Drivers configured with internal(yes) register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using metrics-probe().

local-time-zone()

Description: Sets the timezone that AxoSyslog uses when it expands a timestamp macro in the increment() or labels() templates as a local time.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

on-error()

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().

• drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
• drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
• fallback-to-string: Convert the property to string and log an error message to the internal() source.
• silently-drop-message: Drop the entire message silently, without logging the error.
• silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
• silently-fallback-to-string: Convert the property to string silently, without logging the error.

send-time-zone()

Description: Sets the timezone that AxoSyslog uses when it expands a timestamp macro in the increment() or labels() templates. The time-zone() option is an alias of send-time-zone().

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

template-escape()

Description: Turns on escaping for the ', ", and backspace characters in the expanded value of the increment() and labels() templates.

time-zone()

Description: Alias of send-time-zone().

ts-format()

Description: Overrides the global ts-format() option for the timestamp macros expanded in the increment() and labels() templates.