elasticsearch-http: Send messages to Elasticsearch HTTP Bulk API

Version 3.21 of AxoSyslog can directly post log messages to an Elasticsearch deployment using the Elasticsearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols.

HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.

Declaration:

   d_elasticsearch_http {
        elasticsearch-http(
            index("<elasticsearch-index-to-store-messages>")
            url("https://your-elasticsearch-server1:9200/_bulk")
            type("<type-of-the-index>")
        );
    };

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

Use an empty string to omit the type from the index: type(""). For example, you need to do that when using Elasticsearch 7 or newer, and you use a mapping in Elasticsearch to modify the type of the data.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.

<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

For more detailed information about these environment variables, see the libcurl documentation.

Example: Sending log data to Elasticsearch

The following example defines an elasticsearch-http() destination, with only the required options.

   destination d_elasticsearch_http {
        elasticsearch-http(
            index("<name-of-the-index>")
            type("<type-of-the-index>")
            url("http://my-elastic-server:9200/_bulk")
        );
    };
    
    
    log {
        source(s_file);
        destination(d_elasticsearch_http);
        flags(flow-control);
    };

The following example uses mutually-authenticated HTTPS connection, templated index, and also sets the type() and some other options.

   destination d_elasticsearch_https {
        elasticsearch-http(
            url("https://node01.example.com:9200/_bulk")
            index("test-${YEAR}${MONTH}${DAY}")
            time-zone("UTC")
            type("test")
            workers(4)
            batch-lines(16)
            timeout(10)
            tls(
                ca-file("ca.pem")
                cert-file("syslog_ng.crt.pem")
                key-file("syslog_ng.key.pem")
                peer-verify(yes)
            )
        );
    };

This driver is actually a reusable configuration snippet configured to send log messages using the tcp() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.