diff --git a/_includes/doc/admin-guide/batch-size.md b/_includes/doc/admin-guide/batch-size.md new file mode 100644 index 00000000..432f2bd1 --- /dev/null +++ b/_includes/doc/admin-guide/batch-size.md @@ -0,0 +1,19 @@ +## Batch size + +The batch-bytes(), batch-lines(), and batch-timeout() options of the +destination determine how many log messages syslog-ng OSE sends in a +batch. The batch-lines() option determines the maximum number of +messages syslog-ng OSE puts in a batch in. This can be limited based on +size and time: + +- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even + if the number of messages in the batch is less than batch-lines(). + That way the destination receives every message in a timely manner + even if suddenly there are no more messages. + +- syslog-ng OSE sends the batch if the total size of the messages in + the batch reaches batch-bytes() bytes. + +To increase the performance of the destination, increase the number of +worker threads for the destination using the workers() option, or adjust +the batch-bytes(), batch-lines(), batch-timeout() options. diff --git a/doc/_admin-guide/070_Destinations/030_Elasticsearch-http/000_Batch_mode_and_load_balancing.md b/doc/_admin-guide/070_Destinations/030_Elasticsearch-http/000_Batch_mode_and_load_balancing.md index 7cbf6587..4353e277 100644 --- a/doc/_admin-guide/070_Destinations/030_Elasticsearch-http/000_Batch_mode_and_load_balancing.md +++ b/doc/_admin-guide/070_Destinations/030_Elasticsearch-http/000_Batch_mode_and_load_balancing.md @@ -9,25 +9,7 @@ description: >- the following section. --- -## Batch size - -The batch-bytes(), batch-lines(), and batch-timeout() options of the -destination determine how many log messages syslog-ng OSE sends in a -batch. The batch-lines() option determines the maximum number of -messages syslog-ng OSE puts in a batch in. This can be limited based on -size and time: - -- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even - if the number of messages in the batch is less than batch-lines(). - That way the destination receives every message in a timely manner - even if suddenly there are no more messages. - -- syslog-ng OSE sends the batch if the total size of the messages in - the batch reaches batch-bytes() bytes. - -To increase the performance of the destination, increase the number of -worker threads for the destination using the workers() option, or adjust -the batch-bytes(), batch-lines(), batch-timeout() options. +{% include doc/admin-guide/batch-size.md %} ### Example: HTTP batch mode diff --git a/doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md b/doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md index 26d71a3e..9080052a 100644 --- a/doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md +++ b/doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md @@ -8,14 +8,7 @@ description: >- see the following section. --- -## Batch size - -The `batch-bytes()`, `batch-lines()`, and `batch-timeout()` options specify how many log messages syslog-ng OSE sends in a batch. The `batch-lines()` option defines the maximum number of messages syslog-ng OSE includes in a batch in. This can be limited based on size and time: - -* syslog-ng OSE sends a batch in time intervals defined by `batch-timeout()` in milliseconds, even if the number of messages in the batch is less than the amount defined in `batch-lines()`. This way the destination receives every message in a properly even if the messages cease. -* syslog-ng OSE sends a batch if the total size of the messages in the batch reaches the amount specified in `batch-bytes()` in bytes. - -To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options. +{% include doc/admin-guide/batch-size.md %} ### Example: HTTP batch mode @@ -35,18 +28,7 @@ In the following example, a batch containing 100 messages, or a maximum of 512 k ## Load balancing between multiple indexers -In syslog-ng OSE version 3.19 and later versions, multiple URLs can be specified, for example, `url("site1" "site2")`. In this case, syslog-ng OSE sends log messages to the defined URLs in a load-balanced method. This means that syslog-ng OSE forwards each message to a single URL. For example, this can be used to send the messages to a set of ingestion nodes or indexers of a SIEM solution if a single node cannot handle the load. The order of the messages arriving to the servers can differ from the order syslog-ng OSE has received them. Use load-balancing only if a server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the resulting order of the messages is incorrect. - -![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** If multiple URLs are set in the url() option, set the persist-name() option as well to avoid data loss. -{: .notice--warning} - -In syslog-ng OSE version 3.22 and later versions, any of the following formats can be used to specify multiple URLs: - -```config - url("server1", "server2", "server3"); # comma-separated strings - url("server1" "server2" "server3"); # space-separated strings - url("server1 server2 server3"); # space-separated within a single string -``` +{% include doc/admin-guide/load-balancing.md %} ### Example: HTTP load balancing diff --git a/doc/_admin-guide/070_Destinations/210_Redis/000_Batch_mode_in_Redis.md b/doc/_admin-guide/070_Destinations/210_Redis/000_Batch_mode_in_Redis.md index b9991c2d..4a9b7b2f 100644 --- a/doc/_admin-guide/070_Destinations/210_Redis/000_Batch_mode_in_Redis.md +++ b/doc/_admin-guide/070_Destinations/210_Redis/000_Batch_mode_in_Redis.md @@ -6,22 +6,7 @@ description: >- help of Redis's pipelining feature. --- -## Batch size - -The batch-lines(), batch-lines(), and batch-timeout() options of the -destination determine how many log messages syslog-ng OSE sends in a -batch. The batch-lines() option determines the maximum number of -messages syslog-ng OSE puts in a batch in. This can be limited based on -size and time: - -syslog-ng OSE sends a batch every batch-timeout() milliseconds, even if -the number of messages in the batch is less than batch-lines(). That way -the destination receives every message in a timely manner even if -suddenly there are no more messages. - -To increase the performance of the destination, increase the number of -worker threads for the destination using the workers() option, or adjust -the batch-lines() and/or batch-timeout() options. +{% include doc/admin-guide/batch-size.md %} ### Example: Redis batch mode