10 000 databases in one process: fatal error: thread exhaustion (Go 10000-thread limit) at startup

[stomybexy]

Summary

litestream replicate does not start with 10 000 databases in its config. It dies during boot with the Go runtime's hard thread ceiling:

runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion

5 000 databases run fine on the same machine, same config, same binary. The wall is somewhere between the two, and it is a crash at startup, not a degradation.

Reproduction

• litestream v0.5.14, linux/arm64, 4 vCPU / 32 GB
• N empty WAL-mode SQLite databases, one replica: each
• file:// replicas (no network backend involved)
• Housekeeping cadences set explicitly: l0-retention: 3h, l0-retention-check-interval: 30m, levels: 30m/2h/6h, snapshot: 6h/72h, sync-interval: 1s

databases	result
500	runs, 216 MB RSS, 3 006 fds
1 000	runs, 404 MB RSS, 6 006 fds
2 500	runs, 955 MB RSS, 15 006 fds
5 000	runs, 1 848 MB RSS, 30 006 fds
10 000	fatal error: thread exhaustion

30 019 goroutines at the point of the crash. Memory was not a factor — 31 GB were free.

Confirmed on two independent runs.

What it is not

Not file descriptors. That was my first hypothesis and it is wrong: the second run set ulimit -n 200000 and the outcome was identical. File descriptors scale at a steady 6 per database (3 006 / 6 006 / 15 006 / 30 006), so 10 000 needs ~60 000 — comfortably under that limit.

Not memory. RSS is close to linear at ~0.36 MB per database and the crash happens with tens of GB available.

Why it seems worth reporting

1. The failure mode is abrupt. There is no slow degradation to notice: replication for every database stops at once, at startup. For a backup tool that is an unusually bad shape — the discovery moment tends to be a restore.
2. An operator cannot raise the limit. It is not a ulimit or a sysctl — debug.SetMaxThreads defaults to 10 000 inside the process, so the ceiling is only reachable from litestream's own code.
3. v0.5's directory-watching mode invites this scale. It makes many-database deployments the natural configuration, and nothing warns that a ceiling exists.
4. Related but different: Directory Watcher w/ ~400 DBs crashes litestream #1051 reported instability around 383 databases (since fixed). Same region of the product, different symptom.

Possible directions

debug.SetMaxThreads is a one-line raise, though it only moves the wall. The more durable fix is presumably fewer OS threads held per database — the goroutines that block in syscalls are what consume them (the crash dump is full of modernc.org/sqlite.interruptOnDone select frames).

Even documenting a supported maximum would help: right now the only way to find it is to hit it.

Environment

• litestream v0.5.14 (fatal error reproduced twice)
• linux/arm64, Amazon Linux 2023
• Deployment shape: many databases in one process, the large majority write-idle