L0 retention monitor LISTs the replica for every database on every tick, with no change-detection guard

[stomybexy]

Summary

Store.monitorL0Retention issues one object-store LIST per database, per tick, unconditionally — before anything checks whether that database was written to. On a fleet of mostly-idle databases this is the dominant term in the storage-API bill, and it is paid entirely for nothing.

This is distinct from #1172 (which is about the read/follow path) and from #1210/#1211 (which reduce idle CPU and syscalls; PR #1211 touches db.go and replica.go, not store.go, so it does not remove any retention LIST). #1171 reports a symptom of the same family without isolating a cause.

The path

In v0.5.14:

Store.monitorL0Retention           store.go:606
    ticks every L0RetentionCheckInterval (DefaultL0RetentionCheckInterval = 15s),
    iterates every open DB, no change-detection guard

  → DB.EnforceL0RetentionByTime    db.go:2545
      returns early only when db.L0Retention <= 0

    → db.Replica.Client.LTXFiles(ctx, 1, 0, false)    db.go:2555
        issued immediately, before maxL1TXID is known

      → newFileIterator → s3.NewListObjectsV2Paginator    s3/replica_client.go:1405

At the default 15s check interval that is 5760 LIST requests per database per day, whether or not the database received a single write.

Why this looks like an oversight rather than a design

Two things in the same codebase already do the right thing, which is what makes this stand out:

1. The compaction monitors do not pay this cost. Store.CompactDB reaches the remote through DB.MaxLTXFileInfo (db.go:2783), which memoises per level in db.maxLTXFileInfos and — for levels > 0 — is never invalidated in steady state. So the L1/L2/L3 monitors are effectively free once warm. The retention monitor is the one loop that goes to the network every tick.

2. The replication path already skips on "nothing changed." DB.Sync bails with sync: skip / no new wal pages when sz == 0 (db.go:1913). The project clearly holds the principle that an idle database should not generate work; the retention monitor just does not apply it.

So this is not "the timer model is expensive" — it is one unguarded loop paying for all the others.

Impact

The cost is strictly linear in the number of databases and completely independent of traffic, which makes it a fixed floor per database rather than a usage cost. That is benign for a single database and becomes the whole bill for the multi-database / multi-tenant deployments that v0.5's directory mode invites — precisely the shape where a per-database fixed timer is worst.

Operators can only mitigate it by lengthening l0-retention-check-interval, which trades away the promptness of L0 sweeping (and therefore restore speed) to buy down a cost that should not exist at all.

Suggested fix

Give the retention monitor the same "has anything changed?" guard the sync path already has, before the LIST:

• Skip a database when no L0 file has been written since its last successful retention pass — the writer already knows its own last L0 TXID locally, so the check needs no network round-trip; or
• reuse/extend the maxLTXFileInfos memoisation for the L1 lookup in EnforceL0RetentionByTime, invalidating it when the L1 compactor writes.

Either keeps the current retention semantics exactly: a database with no new L0 files has nothing to expire, so the LIST cannot change the outcome.

Environment

• litestream v0.5.14 (paths above are from that tag; the same code is present in v0.5.15 and v0.5.16 — neither release changes it)
• Backend: S3
• Deployment: many databases in one process, the large majority write-idle for long stretches