Reduce S3 LIST operations by introducing manifest files for LTX replication levels #1172
Description
Feature Description
Reduce the number of S3 LIST operations by introducing manifest files that describe the expected LTX files for each replication level (e.g., L0 and L1). Readers such as litestream restore could fetch the manifest instead of repeatedly listing the bucket.
Use Cases
Litestream currently polls the replica frequently (default every second), which results in a large number of S3 LIST operations to discover new LTX files.
In one of my applications, I observed approximately:
- ~100k LIST operations
- ~4k GET operations
Since LIST operations are more expensive and scale poorly, this significantly increases storage API costs.
The goal is to reduce LIST operations and replace them with fewer, more efficient GET requests.
Attempted Solutions
Currently using Litestream with default polling configuration. The only way to reduce LIST operations is to increase the polling interval, which increases replication latency and restore detection time.
No clean workaround exists to maintain low latency while minimizing LIST costs.
Proposal
The writer could periodically upload a manifest file (for example, JSON) containing the expected set of LTX files for each level (L0/L1).
Readers would:
- Fetch the manifest via GET
- Determine which LTX files exist
- Avoid issuing frequent LIST requests
This would significantly reduce S3 API costs and improve scalability, especially for frequently polling replicas.