Q&A Please assess my Vector config for per-message loadbalancing - split messages and TCP backpressure

[ma-mueh]

Question

Hi

We've implemented Vector as our solution for per Syslog message load balancing due to our environment consisting of high volume and very low volume senders (in the testing phase mostly Fortinet Fortigate), which a L3/L4 load balancer could not adequatly distribute to our Elastic agent forwarders. In total we expect around 200k EPS continously which adds up to 20MB/s.

We've got 1500 senders which send to two vector instances through a round robin L3 load balancer. Vector in turn has 60 sinks which point to Elastic Agents.

Source devices -> load balancer -> 2 Vector -> 60 Elastic Agents.

As the only thing we need is even distribution of messages our Vector config was planned to be as simple as possible and leave the original message untouched. With that we have two issues:

1. The vector host can't handle the load (TCP zero window size messages in tcpdumps) - TCP Receive Buffers are full, send Buffers are empty
2. Some messages arrive in Elastic as partials of the original message

We've optet to octet_counting because we are not interested in any parsing of the message. However, we've seen log messages indicating issues with the built-in utf8 check for that particular framing method.

On the host we've tuned our TCP config so it should theoretically be able to handle the load:

ulimit 262144
sysctl net.core.somaxconn=8192
sysctl net.ipv4.tcp_max_syn_backlog=8192
sysctl net.core.rmem_max=16777216
sysctl net.core.wmem_max=16777216

Vector runs on 8 CPUs with about 30% load on each.

I haven't found much in terms of similar setups and would greatly appreciate if someone could tell me if we're completely off track with our thoughts recarding the vector configuration with the goal we want to achieve or if it's simply a limitation of what vector can handle (other discussions talk about 70MB/s throughput) and we need to scale horizontally. In terms of the broken messages I assume that's because Fortigate sends its Logs with wrong octet counts, currently I have no idea how we could fix that.

Vector Config

sources:
  src_fortigate:
    type: socket
    mode: tcp
    address: 0.0.0.0:7010
    connection_limit: 4096
    receive_buffer_bytes: 16777216
    keepalive:
      time_secs: 60
    decoding:
      codec: bytes
    framing:
      method: octet_counting
transforms:
  shard_fortigate:
    type: remap
    inputs:
    - src_fortigate
    source: |
      .shard = random_int(0, 60)
  route_fortigate:
    type: route
    inputs:
    - shard_fortigate
    reroute_unmatched: false
    route:
      s0: .shard == 0
[...]
sinks:
  sink_fortigate_00:
    type: socket
    inputs:
    - route_fortigate.s0
    mode: tcp
    address: <redacted>:7010
    encoding:
      codec: text
    keepalive:
      time_secs: 60
    send_buffer_bytes: 4194304
    buffer:
      type: memory
      max_events: 262144
      when_full: block
    healthcheck:
      enabled: true
    framing:
      method: newline_delimited

Vector Logs

2026-09-18T13:17:40.278127Z ERROR source{component_kind="source" component_id=src_fortigate component_type=socket}:connection{peer_addr=<redacted>:12053}: codecs::internal_events: Failed framing bytes. error=FramingError(Connection timed out (os error 110)) error_code="decoder_frame" error_type="parser_failed" stage="processing"
2026-09-18T12:55:40.953972Z ERROR source{component_kind="source" component_id=src_fortigate component_type=socket}:connection{peer_addr=<redacted>:10851}: codecs::internal_events: Failed framing bytes. error=FramingError(Connection reset by peer (os error 104)) error_code="decoder_frame" error_type="parser_failed" stage="processing"
2026-09-18T12:55:40.953770Z ERROR source{component_kind="source" component_id=src_fortigate component_type=socket}:connection{peer_addr=<redacted>:10851}: codecs::internal_events: Internal log [Failed framing bytes.] has been suppressed 3 times.
2026-09-18T12:47:41.893136Z ERROR source{component_kind="source" component_id=src_fortigate component_type=socket}:connection{peer_addr=<redacted>:11513}: codecs::internal_events: Internal log [Failed framing bytes.] is being suppressed to avoid flooding.
2026-09-18T12:47:41.131128Z ERROR source{component_kind="source" component_id=src_fortigate component_type=socket}:connection{peer_addr=<redacted>:11513}: codecs::internal_events: Failed framing bytes. error=Unable to decode message as UTF8 error_code="decoder_frame" error_type="parser_failed" stage="processing"

[identity-wael]

I would separate message framing from backpressure before increasing capacity. The posted errors don't yet prove that Fortigate is sending incorrect lengths.

decoding.codec: bytes avoids parsing the syslog payload, but framing.method: octet_counting still interprets message boundaries. For a captured TCP stream, verify the decimal length prefix against the number of bytes following the separating space, not Unicode characters and not individual TCP packets. TCP segments need reassembly before doing that check. A bad length can shift subsequent boundaries; invalid UTF-8 or a connection reset can also have other causes. The socket source reference distinguishes framing from decoding.

Your output also changes framing to newline_delimited. If an input payload contains an embedded newline, the downstream newline-framed receiver can treat one input event as multiple messages. Compare one sanitized offending payload before Vector, immediately after source decoding, and at the Elastic input. That distinguishes an input-frame problem from an output delimiter problem. Confirm the receiver's framing and maximum-message-size settings too.

For throughput, random sharding is not load-aware: events assigned to a slow/full sink are not automatically reassigned to a healthier one. With when_full: block, backpressure can propagate to the source while CPU remains partly idle; Vector documents this in its buffering model. Empty kernel send buffers don't tell you whether Vector's internal sink buffers are full.

A useful isolated test is the same input/route setup with fast local consumers, then deliberately slow one consumer while watching per-sink buffer occupancy and sent-event rates. If that reproduces the source zero-window condition, you have downstream/topology backpressure rather than evidence of an overall 8-core throughput ceiling.

Finally, 60 × 262144 permits about 15.7 million buffered events per instance. That is an event-count limit, not a byte budget; include the in-memory event representation when sizing RAM. Bigger buffers only buy time if a downstream consumer cannot sustain its assigned rate.

[ma-mueh]

Thank you very much for these pointers, we're working on getting the before and after captures for the framing method. I strongly suspect bad lengths in the packets, as the messages in the stream I've seen so far are simple key/value strings. However, I can't completely rule out a random newline in a comment being the culprit, we didn't think of that.

We've mostly been working on the throughput problem these last few days and made some headway in isolating the root cause but are still not fully there yet. At least the tcp zero window messages are gone due to scaling horizontally and optimizing the TCP buffers to match sender/ vector/ linux host better.

During our tests we've seen that heavy senders can:

• get the logs to Vector
• the receive queue on the vector host fills (cpu and memory are utilized at around 50%)
• the heavy sender starts to drop logs locally when the receive queue is full on Linux.

The sink buffer metrics reported by Vector do not show any sink backpressure during periods where drops on some senders occured.
We've checked vector_buffer_received_bytes_total - vector_buffer_sent_bytes_total and vector_buffer_received_events_total -vector_buffer_sent_events_total which hopefully is the right approach. So we've discarded when_full: block being the root cause.

With the existing documentation I don't know where else to look for backpressure in Vector, our analysis points to vector itself not being able to move the events from the source to the sink fast enough for individual heavy senders (continuous 2.2MB/s during slow hours, more during peak). Could moving these individual hosts to dedicated sources be a valid approach to improve performance?