Feature Description

Reduce the number of S3 LIST operations by introducing manifest files that describe the expected LTX files for each replication level (e.g., L0 and L1). Readers such as litestream restore could fetch the manifest instead of repeatedly listing the bucket.

Use Cases

Litestream currently polls the replica frequently (default every second), which results in a large number of S3 LIST operations to discover new LTX files.

In one of my applications, I observed approximately:

• ~100k LIST operations
• ~4k GET operations

Since LIST operations are more expensive and scale poorly, this significantly increases storage API costs.

The goal is to reduce LIST operations and replace them with fewer, more efficient GET requests.

Attempted Solutions

Currently using Litestream with default polling configuration. The only way to reduce LIST operations is to increase the polling interval, which increases replication latency and restore detection time.

No clean workaround exists to maintain low latency while minimizing LIST costs.

Proposal

The writer could periodically upload a manifest file (for example, JSON) containing the expected set of LTX files for each level (L0/L1).

Readers would:

• Fetch the manifest via GET
• Determine which LTX files exist
• Avoid issuing frequent LIST requests

This would significantly reduce S3 API costs and improve scalability, especially for frequently polling replicas.