Merge branch 'master' into s3_fixes

_data/external_links.yml

@@ -608,6 +608,11 @@ gh-es:
   url: https://github.com/syslog-ng/syslog-ng/tree/master/scl/elasticsearch
   title: [ "Elasticsearch configuration snippet on GitHub" ]
 
+gh-es-ds:
+  id: gh-es-ds
+  url: https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elastic-datastream.conf
+  title: [ "Elasticsearch datastream configuration snippet on GitHub" ]
+
 gh-fortigate:
   id: gh-fortigate
   url: https://github.com/syslog-ng/syslog-ng/blob/master/scl/fortigate/fortigate.conf
@@ -796,6 +801,11 @@ ebpf:
   url: https://ebpf.io/
   title: [ "Extended Berkeley Packet Filter" ]
 
+elastic-ds:
+  id: elastic-ds
+  url: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
+  title: [ "Elasticsearch data streams" ]
+
 gcp-ser:
   id: gcp-ser
   url: https://cloud.google.com/iam/docs/service-account-overview

doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md

@@ -71,6 +71,10 @@ destination {
 
 *Description:* This option configures the upper limit of in-flight gRPC requests per worker. It is advisd to set this value in the range of 10s or 100s when there are a high number of clients sending simultaneously. In an optimzed solution, the number of `workers()` and `concurrent-requests()` is greater than or equal to the number of clients. However, this can cause an increase in memory usage.
 
+## keep-hostname()
+
+The `syslog-ng-otlp()` and `opentelemetry()` sources ignore this option and use the hostname from the message as the `${HOST}`.
+
 ## log-fetch-limit()
 
 |   Type:|       number|

doc/_admin-guide/060_Sources/175_syslog-otlp/000_otlp_source_options.md

@@ -129,7 +129,9 @@ source {
 
 {% include doc/admin-guide/options/host-override.md %}
 
-{% include doc/admin-guide/options/keep-hostname.md %}
+## keep-hostname()
+
+The `syslog-ng-otlp()` and `opentelemetry()` sources ignore this option and use the hostname from the message as the `${HOST}`.
 
 {% include doc/admin-guide/options/keep-timestamp.md %}
 

doc/_admin-guide/060_Sources/180_System/README.md

@@ -50,7 +50,10 @@ such messages without losing any information to CIM-aware applications
 |              | Note that on Linux, the so-rcvbuf() option of the system() source is automatically set to 8192.        |
 |              | If the host is running under systemd, {{ site.product.short_name }} reads directly from the systemd journal file using the systemd-journal() source.                        |
 |              | If the kernel of the host is version 3.5 or newer, and /dev/kmsg is seekable, {{ site.product.short_name }} will use that instead of /proc/kmsg, using the multi-line-mode(indented), keep-timestamp(no), and the format(linux-kmsg)options.                      |
-|              | If {{ site.product.short_name }} is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.                                 |
+|              |If {{ site.product.short_name }} is running in a jail or a Linux Container (LXC), it will not read from the `/dev/kmsg` or `/proc/kmsg` files.
+|              |With systemd: `systemd-journal();`
+|              |Without systemd, on kernel 3.5 or newer: `unix-dgram("/dev/log"); file("/dev/kmsg" program-override("kernel") flags(kernel) format("linux-kmsg") keep-timestamp(no));`
+|              |Without systemd, on kernels older than 3.5: `unix-dgram("/dev/log"); file("/proc/kmsg" program-override("kernel") flags(kernel) keep-timestamp(no));`
 | macOS        |     file("/var/log/system.log" follow-freq(1));      |
 |              | **NOTE:** Starting with version 3.7, the {{ site.product.short_name }} system() driver automatically extracts the msgid  from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the **extract-solaris-msgid()** parser. You can find the exact source of the Solaris parser on GitHub.|
 | NetBSD       |     unix-dgram("/var/run/log");                      |

doc/_admin-guide/060_Sources/220_unix-stream_unix-dgram/README.md

@@ -4,10 +4,9 @@ short_title: unix-stream, unix-dgram
 id: adm-src-unix
 description: >-
     The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and
-    start listening on it for messages. The unix-stream() driver is
-    primarily used on Linux and uses SOCK_STREAM semantics (connection
-    oriented, no messages are lost), while unix-dgram() is used on BSDs and
-    uses SOCK_DGRAM semantics: this may result in lost local messages if
+    start listening on it for messages. On Linux both the unix-stream() and unix-dgram() drivers are used and are always reliable. The unix-stream() driver uses SOCK_STREAM semantics (connection
+    oriented, no messages are lost),
+    while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may result in lost local messages if
     the system is overloaded.
 ---
 

doc/_admin-guide/070_Destinations/035_elasticsearch-datastream/README.md

@@ -0,0 +1,31 @@
+---
+title: 'elasticsearch-datastream: Elasticsearch data streams'
+short_title: elasticsearch-datastream
+id: adm-dest-es-datastream
+description: >-
+    From {{ site.product.short_name }} 4.8 and later versions, you  can send messages and metrics to Elasticsearch data streams to store your log and metrics data as time series data.
+---
+
+**Declaration**
+
+```config
+destination d_elastic_data_stream {
+  elasticsearch-datastream(
+    url("https://elastic-endpoint:9200/my-data-stream/_bulk")
+    user("elastic")
+    password("ba253DOn434Tc0pY22OI")
+  );
+};
+```
+
+This driver is a reusable configuration snippet configured to send log messages using the http() driver using a template. You can find the Elasticsearch datastream configuration snippet on GitHub.
+
+## Prerequisites
+
+* An account for Elasticsearch datastreams with a username and a password.
+
+## Options
+
+Elasticsearch datastream is an HTTP based driver, hence it utilizes the HTTP destination options.
+
+> *Copyright © 2024 Axoflow*

doc/_admin-guide/100_TLS-encrypted_message_transfer/002_Mutual_authentication_using_TLS/000_Configuring_mutual_TLS_client.md

@@ -17,28 +17,28 @@ syslog() driver):
 1. Create an X.509 certificate for the {{ site.product.short_name }} client.
 
 2. Copy the certificate (for example, client_cert.pem) and the
-    matching private key (for example, client.key) to the syslog-ng
+    matching private key (for example, client.key) to the {{ site.product.short_name }}
     client host, for example, into the
     /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must
-    be a valid X.509 certificate in PEM format. If you want to use a
-    password-protected key, see
-    Password-protected keys.
+    be a valid X.509 certificate in PEM format. The key must be in PEM format.
+    If you want to use a password-protected key, see Password-protected keys.
 
-3. Copy the CA certificate of the Certificate Authority (for example,
-    cacert.pem) that issued the certificate of the {{ site.product.short_name }} server (or
-    the self-signed certificate of the {{ site.product.short_name }} server) to the
-    {{ site.product.short_name }} client hosts, for example, into the
+3. Copy the CA certificate (for example, cacert.pem) of the Certificate
+    Authority that issued the certificate of the {{ site.product.short_name }} server
+    (or the self-signed certificate of the {{ site.product.short_name }} server) to the
+    {{ site.product.short_name }} client host, for example, into the
     /opt/syslog-ng/etc/syslog-ng/ca.d directory.
 
-    Issue the following command on the certificate: **openssl x509
-    -noout -hash -in cacert.pem** The result is a hash (for example,
+    If you wish to use the ca-dir() option, instead of the ca-file(), in the
+    {{ site.product.short_name }} configuration file (step 4.) then
+    - issue the following command on the certificate:\
+    `openssl x509 -noout -hash -in cacert.pem`\
+    The result is a hash (for example,
     6d2962a8), a series of alphanumeric characters based on the
     Distinguished Name of the certificate.
-
-    Issue the following command to create a symbolic link to the
+    - issue the following command to create a symbolic link to the
     certificate that uses the hash returned by the previous command and
-    the **.0** suffix.
-
+    the **.0** suffix:\
     `ln -s cacert.pem 6d2962a8.0`
 
 4. Add a destination statement to the {{ site.product.short_name }} configuration file that
@@ -49,6 +49,9 @@ syslog() driver):
     Include the client\'s certificate and private key in the tls()
     options.
 
+    For the details of the available tls() options, see
+    TLS options.
+
     Example: A destination statement using mutual authentication
 
     The following destination encrypts the log messages using TLS and

doc/_admin-guide/100_TLS-encrypted_message_transfer/002_Mutual_authentication_using_TLS/001_Configuring_mutual_TLS_server.md

@@ -11,38 +11,37 @@ Complete the following steps on the {{ site.product.short_name }} server:
 
 ## Steps
 
-1. Copy the certificate (for example, syslog-ng.cert) of the syslog-ng
-    server to the {{ site.product.short_name }} server host, for example, into the
+1. Create an X.509 certificate for the {{ site.product.short_name }} server.
+
+2. Copy the certificate (for example, syslog-ng.cert) and the
+    matching private key (for example, syslog-ng.key) to the {{ site.product.short_name }}
+    server host, for example, into the
     /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must
-    be a valid X.509 certificate in PEM format.
+    be a valid X.509 certificate in PEM format. The key must be in PEM format.
+    If you want to use a password-protected key, see Password-protected keys.
 
-2. Copy the CA certificate (for example, cacert.pem) of the Certificate
+3. Copy the CA certificate (for example, cacert.pem) of the Certificate
     Authority that issued the certificate of the {{ site.product.short_name }} clients to
-    the {{ site.product.short_name }} server, for example, into the
+    the {{ site.product.short_name }} server host, for example, into the
     /opt/syslog-ng/etc/syslog-ng/ca.d directory.
 
-    Issue the following command on the certificate: **openssl x509
-    -noout -hash -in cacert.pem** The result is a hash (for example,
+    If you wish to use the ca-dir() option, instead of the ca-file(), in the
+    {{ site.product.short_name }} configuration file (step 4.) then
+    - issue the following command on the certificate:\
+    `openssl x509 -noout -hash -in cacert.pem`\
+    The result is a hash (for example,
     6d2962a8), a series of alphanumeric characters based on the
     Distinguished Name of the certificate.
-
-    Issue the following command to create a symbolic link to the
+    - issue the following command to create a symbolic link to the
     certificate that uses the hash returned by the previous command and
-    the **.0** suffix.
-
+    the **.0** suffix:\
     `ln -s cacert.pem 6d2962a8.0`
 
-3. Copy the private key (for example, syslog-ng.key) matching the
-    certificate of the {{ site.product.short_name }} server to the {{ site.product.short_name }} server host,
-    for example, into the /opt/syslog-ng/etc/syslog-ng/key.d directory.
-    The key must be in PEM format. If you want to use a
-    password-protected key, see Password-protected keys.
-
 4. Add a source statement to the {{ site.product.short_name }} configuration file that uses
     the tls( key-file(key_file_fullpathname)
     cert-file(cert_file_fullpathname) ) option and specify the key and
     certificate files. The source must use the source driver (network()
-    or syslog()) matching the destination driver used by the syslog-ng
+    or syslog()) matching the destination driver used by the {{ site.product.short_name }}
     client. Also specify the directory storing the certificate of the CA
     that issued the client's certificate.
 
@@ -52,7 +51,7 @@ Complete the following steps on the {{ site.product.short_name }} server:
     Example: A source statement using TLS
 
     The following source receives log messages encrypted using TLS,
-    arriving to the 1999/TCP port of any interface of the syslog-ng
+    arriving to the 1999/TCP port of any interface of the {{ site.product.short_name }}
     server.
 
     ```config

doc/_admin-guide/110_Template_and_rewrite/000_Customize_message_format/004_Macros_of_syslog-ng.md

@@ -503,6 +503,7 @@ following macros are available in {{ site.product.short_name }} version 3.9 and
 - .tls.x509_o: The value of the Organization field.
 
 - .tls.x509_ou: The value of the Organization Unit field.
+- .tls.x509_fp: The key fingerprint of the peer, when the trusted-key() option is used. Available in {{ site.product.short_name }} 4.8 and later versions.
 
 ## ${UNIQID}
 

doc/_admin-guide/120_Parser/002_Parsing_key-value_pairs/000_kv_parser_options.md

@@ -1,6 +1,6 @@
 ---
 title: Options of key=value parsers
-parser: kv-parser
+parser: kv
 prefix: kv
 id: adm-parser-kv-opt
 description: >-