vector_buffer_events & vector_buffer_byte_size metrics stuck on non-zero value

[rayan-chehab]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

After upgrading from version 0.44.0 to 0.50.0, I started experiencing issues with Vector's vector_buffer_events and vector_buffer_byte_size metrics coming from the internal_metrics source getting stuck on non-zero values.

I was able to verify that this was an issue with metrics and not events actually being stuck by computing the actual number of events in buffer using the following PromQL query

vector_buffer_received_events_total - vector_buffer_sent_events_total

Using the vector top command, the Events In and the Events Out values were equal for the sink that had its vector_buffer_events metric stuck at 625.

Additionally, I went further into debugging this and looked into the CSV files that were being output by the sink. The result was that there were no files nor data missing from them.

Configuration

api:
  enabled: true

sources:
  metrics_source:
    type: prometheus_scrape
    endpoints:
      - <SENSITIVE>
    scrape_timeout_secs: 15
    scrape_interval_secs: 60

  internal_metrics_source:
    type: internal_metrics

transforms:
  metric_to_log_transform:
    type: metric_to_log
    inputs:
      - metrics_source
  
  add_epoch_timestamp_transform:
    type: remap
    inputs:
      - metric_to_log_transform
    source: |
      .epoch_timestamp = to_unix_timestamp!(.timestamp)

  remove_metrics_host_label_transform:
    type: remap
    inputs:
      - internal_metrics_source
    source: |
      del(.tags.host)

sinks:
  csv_file_sink:
    type: file
    inputs:
      - add_epoch_timestamp_transform
    path: /opt/enrichment/{{ tags.environment }}/{{ epoch_timestamp }}.csv
    encoding:
      codec: csv
      csv:
        fields:
          - <SENSITIVE>
          - tags.environment

  prometheus_sink:
    type: prometheus_exporter
    inputs:
      - remove_metrics_host_label_transform
    address: 0.0.0.0:9598
    flush_period_secs: 86400

Version

0.50.0