Skip to content

Commit

Permalink
Separated Batch size block into reusable snippet (#70)
Browse files Browse the repository at this point in the history
Fixes: #68

Unification of "Batch size" blocks with a snippet. Also replaced "Load
balancing" with existing snippet.
  • Loading branch information
HofiOne authored May 28, 2024
2 parents ea5c7d8 + 4057d11 commit 4ea7b35
Show file tree
Hide file tree
Showing 4 changed files with 23 additions and 55 deletions.
19 changes: 19 additions & 0 deletions _includes/doc/admin-guide/batch-size.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## Batch size

The batch-bytes(), batch-lines(), and batch-timeout() options of the
destination determine how many log messages syslog-ng OSE sends in a
batch. The batch-lines() option determines the maximum number of
messages syslog-ng OSE puts in a batch in. This can be limited based on
size and time:

- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even
if the number of messages in the batch is less than batch-lines().
That way the destination receives every message in a timely manner
even if suddenly there are no more messages.

- syslog-ng OSE sends the batch if the total size of the messages in
the batch reaches batch-bytes() bytes.

To increase the performance of the destination, increase the number of
worker threads for the destination using the workers() option, or adjust
the batch-bytes(), batch-lines(), batch-timeout() options.
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,7 @@ description: >-
the following section.
---

## Batch size

The batch-bytes(), batch-lines(), and batch-timeout() options of the
destination determine how many log messages syslog-ng OSE sends in a
batch. The batch-lines() option determines the maximum number of
messages syslog-ng OSE puts in a batch in. This can be limited based on
size and time:

- syslog-ng OSE sends a batch every batch-timeout() milliseconds, even
if the number of messages in the batch is less than batch-lines().
That way the destination receives every message in a timely manner
even if suddenly there are no more messages.

- syslog-ng OSE sends the batch if the total size of the messages in
the batch reaches batch-bytes() bytes.

To increase the performance of the destination, increase the number of
worker threads for the destination using the workers() option, or adjust
the batch-bytes(), batch-lines(), batch-timeout() options.
{% include doc/admin-guide/batch-size.md %}

### Example: HTTP batch mode

Expand Down
22 changes: 2 additions & 20 deletions doc/_admin-guide/070_Destinations/155_opensearch/000_batch-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,7 @@ description: >-
see the following section.
---

## Batch size

The `batch-bytes()`, `batch-lines()`, and `batch-timeout()` options specify how many log messages syslog-ng OSE sends in a batch. The `batch-lines()` option defines the maximum number of messages syslog-ng OSE includes in a batch in. This can be limited based on size and time:

* syslog-ng OSE sends a batch in time intervals defined by `batch-timeout()` in milliseconds, even if the number of messages in the batch is less than the amount defined in `batch-lines()`. This way the destination receives every message in a properly even if the messages cease.
* syslog-ng OSE sends a batch if the total size of the messages in the batch reaches the amount specified in `batch-bytes()` in bytes.

To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.
{% include doc/admin-guide/batch-size.md %}

### Example: HTTP batch mode

Expand All @@ -35,18 +28,7 @@ In the following example, a batch containing 100 messages, or a maximum of 512 k

## Load balancing between multiple indexers

In syslog-ng OSE version 3.19 and later versions, multiple URLs can be specified, for example, `url("site1" "site2")`. In this case, syslog-ng OSE sends log messages to the defined URLs in a load-balanced method. This means that syslog-ng OSE forwards each message to a single URL. For example, this can be used to send the messages to a set of ingestion nodes or indexers of a SIEM solution if a single node cannot handle the load. The order of the messages arriving to the servers can differ from the order syslog-ng OSE has received them. Use load-balancing only if a server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the resulting order of the messages is incorrect.

![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** If multiple URLs are set in the url() option, set the persist-name() option as well to avoid data loss.
{: .notice--warning}

In syslog-ng OSE version 3.22 and later versions, any of the following formats can be used to specify multiple URLs:

```config
url("server1", "server2", "server3"); # comma-separated strings
url("server1" "server2" "server3"); # space-separated strings
url("server1 server2 server3"); # space-separated within a single string
```
{% include doc/admin-guide/load-balancing.md %}

### Example: HTTP load balancing

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,7 @@ description: >-
help of Redis's pipelining feature.
---

## Batch size

The batch-lines(), batch-lines(), and batch-timeout() options of the
destination determine how many log messages syslog-ng OSE sends in a
batch. The batch-lines() option determines the maximum number of
messages syslog-ng OSE puts in a batch in. This can be limited based on
size and time:

syslog-ng OSE sends a batch every batch-timeout() milliseconds, even if
the number of messages in the batch is less than batch-lines(). That way
the destination receives every message in a timely manner even if
suddenly there are no more messages.

To increase the performance of the destination, increase the number of
worker threads for the destination using the workers() option, or adjust
the batch-lines() and/or batch-timeout() options.
{% include doc/admin-guide/batch-size.md %}

### Example: Redis batch mode

Expand Down

0 comments on commit 4ea7b35

Please sign in to comment.