How does disk buffer cleanup work in Vector when using EFS with stateless pods

[wangzhihaocom]

Hi Vector team
I’m trying to better understand how Vector disk buffers work internally, especially buffer cleanup behavior, when running Vector in Kubernetes with EFS-backed persistence and Spot instances.

Environment / Setup

Deployment type: Kubernetes Deployment
Helm role: Stateless-Aggregator
Autoscaling: Enabled (pods are ephemeral)
Node type: AWS Spot instances
Persistence: AWS EFS mounted to Vector data_dir

Purpose:

Handle frequent pod termination (Spot interruptions, rescheduling)
Avoid data loss during abrupt shutdowns
Sinks: ClickHouse
Buffer type: disk
Acknowledgements: true
Each pod uses a pod-specific data_dir to avoid concurrent writers:

Filesystem layout (inside a running pod)

df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          30G  6.6G   24G  22% /
127.0.0.1:/     8.0E   92M  8.0E   1% /vector-data-dir

EFS directory structure

/vector-data-dir/
├── pod-vector-6b5d85f886-qvnm5
│   └── buffer
│       └── v2
│           ├── clickhouse_syslog
│           │   ├── buffer-data-0.dat
│           │   ├── buffer.db
│           │   └── buffer.lock
│           └── clickhouse_trafficlog
└── pod-vector-6b5d85f886-v48hb

Observed Behavior:

Data is successfully written to ClickHouse

Incoming traffic has been stopped

However:

buffer-data-0.dat .This file size dose not decreasing,.. (Does this filie store all the buffer)

Disk usage does not immediately decrease

Files remain even when there is no more traffic

Questions

1. Disk buffer lifecycle

What exactly is stored in the buffer-data-0*.dat files? Do they contain all of the buffer data?

I stopped the traffic, but the file size did not decrease. Should the file size decrease when there is no incoming traffic?

2. Handling Spot instance termination

If a pod is terminated because the Spot instance is taken away, what is the recommended way to handle unflushed buffer data?

3. Current design approach
In my current setup:

When a Spot instance is terminated, the old pod is killed.

A new pod is created.

I copy the old pod’s buffer data from:

vector-data-dir/pod-vector-${OLD_POD}/buffer/v2/<sink>

to the new pod’s directory:

vector-data-dir/pod-vector-${NEW_POD}/buffer/v2/<sink>

The idea is that the new pod will continue flushing the old pod’s buffer data.

Is this the correct approach for handling buffer persistence when using Spot instances?

Any help is very appreciated

[wangzhihaocom]

anyone has any idea?