Q&A VRL assert*() rate limit?

[ozacas]

Question

Hi,

I had a pipeline use assert() functions to catch oddball logs at scale which are not handled due to unusual combination of fields. However, these functions with vector 0.5x does not appear to be rate limited in a manner similar to log() which caused docker container logs to nearly fill some sink's capacity.

Just wondering if this is by design? Is it discouraged to use assert functions in the pipeline and only use them in unit tests?

Vector Config

    if unusual_log {
        assert(false, "Dropping  unusual log")
    }

Vector Logs

No response

[thomasqueirozb]

This is likely related to the changes made in #24024/#24050. Are you able to confirm that this was not the case in v0.50.0? Do you have an full example config and the expected output from when it was working as expected vs now?

[ozacas]

I am still working to understand what is going on - there's quite a few pieces involved:

• docker run alpine:latest tar -cf ... to stdout via pipe as part of a container volume backup procedure
• docker daemon.json log-driver = journald (confirmed with docker inspect against the container)
• docker community edition v28.1 or later
• vector 0.51.x (or later) configured to read from journald as a source

With these basic components, I am seeing millions of logs from stdout of the alpine container, representing the tarball content as captured by vector via journald. Something is capturing more than desired, but I was trying to drop the logs quickly in the vector pipeline. Although the container run could be done with --log-driver none or similar, I was hoping to be more selective.

As a result, I was trying to filter logs using assert(false) but this doesnt seem to be subject to rate limiting for downstream sinks, leaving millions of assert errors into the local vector debug sink. This in turn filled the underlying filesystem.

I'll give an earlier version a try, but unfortunately the pipeline uses recent released functions like basename() and friends...

[ozacas]

i was able to try vector noble 0.50.0-1 - same behaviour observed in terms of pipe output being sent from alpine container to vector. Seems to be a docker/journald related issue. I expected that piped output would not be captured.

I ran the container with: docker run --name junk --log-driver=journald -v /var/cache:/data alpine:latest tar -zcf - /data | cat > /dev/null

And relevant vector config:

sources:
  # local sources
  vector_logs:
    type: internal_logs

  systemd_journal_log:
    type: journald

produces non-sensical logs such as the tap capture file attached.

tap.txt