[bug] 0000000.log in data/ grows continuously under idle, larger than other data files

[joewiz]

[This issue was filed in response to a request by @duncdrum during PR #6145 testing.]

Symptom

While testing PR #6145 (Jetty 11→12 upgrade + WebSocket eval), @duncdrum observed a continuously-growing log file at exist/data/0000000.log:

• File grows without apparent bound, even when the database is idle
• After a few minutes of idle time it's already larger than anything else in the data/ folder
• File is locked while running, so contents could not be inspected from outside the JVM

Context

This is not caused by the Jetty 12 upgrade or the WebSocket changes — it is independent of those, surfaced incidentally during #6145 testing. The file is in data/, not in the eXist log directory configured by log4j2, so it sits outside the normal log-management path.

Hypothesis

The filename pattern (0000000.log — leading zeros, no rolling-suffix) is consistent with the BFile / journal log style used by eXist's storage layer (collection cache journal, broker pool, or transaction log). Likely candidates to investigate:

• Transaction log without bounded rotation
• Cache journal accumulating without checkpoint
• A debug-level writer left enabled in the storage layer

Requested

• Identify which subsystem owns 0000000.log
• Confirm whether the unbounded growth is a regression or pre-existing behavior
• Decide on appropriate rotation / size cap / opt-out

Reference

PR #6145 review thread: #6145

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Investigated. Not a leak — it's startup accumulation that never rotates. Posting findings + options before implementing anything, since the choice between fixes has a design trade-off worth core-dev steering.

Reproduction (vanilla existdb/existdb:latest container, 2026-05-17)

Fresh container, no client activity:

Time since start	0000000000.log size	Other largest data file
~5 min	12.9 MB	dom.dbx at 7.8 MB
+60 s (steady-state idle)	+1.4 KB	—

So duncdrum's observation reproduces exactly: after a few minutes the journal is larger than anything else in data/. But:

• Steady-state idle growth is ~1.4 KB/min (just the periodic Sync.MAJOR checkpoint entries, every 120 s per db-connection.pool.sync-period).
• The 13 MB came from startup-phase writes — AutoDeploymentTrigger installs 9 XAR packages on first boot (functx, semver-xq, packageservice, dashboard, eXide, templating, exist-documentation, fundocs, function-documentation), each creating collections + writing documents + building indexes. All of that hits the journal.

Why it never shrinks

Code path (current develop):

• Journal.flushToLog() (line 434) triggers pool.triggerCheckpoint() only when channel.size() >= journalSizeLimit (default DEFAULT_MAX_SIZE = 100 MB).
• BrokerPool.sync() (line 1505) calls transactionManager.checkpoint(checkpoint) with checkpoint = false under normal periodic syncs. false writes a Checkpoint entry but doesn't switch journal files.
• checkpoint = true is set only by enterServiceMode() (manual) and triggerCheckpoint() (called by the 100 MB ceiling above + consistency-check shutdowns).

So the 13 MB of startup writes sit in 0000000000.log indefinitely. Subsequent activity gradually adds to it, but rotation only happens after the 100 MB ceiling. On a typical install with light write activity, a single journal file can accumulate for weeks/months.

The filename 0000000000.log (per Journal.getFileName(): %010x.log) is journal file number 0 — the first one ever created. We never moved to 0000000001.log because the ceiling wasn't reached.

This is by design rather than a regression. But it has real costs:

• Perception — looks like a leak to anyone inspecting data/ on an apparently-idle instance.
• Post-crash recovery time — RecoveryManager replays from the last Checkpoint entry; with no rotation, replay can grow toward 100 MB.
• Startup writes never reclaimed — even though the XARs are persistent and don't need re-deploying, their original install transactions stay in the journal until the file finally rotates.

Three fix options (no preference; flagging trade-offs)

1. Post-startup checkpoint with switchFiles=true. Add a hook so BrokerPool triggers a checkpoint-and-rotate once startup completes (after AutoDeploymentTrigger and any other system tasks finish). Net effect: the 13 MB of startup writes get consolidated into a closed journal that's then removed (per the RemoveRunnable path in Journal.checkpoint()), and the live journal restarts near-empty. Minimal API surface, surgical, addresses both the perception complaint and the recovery-time cost. Doesn't change the steady-state rotation contract.

2. Lower DEFAULT_MAX_SIZE from 100 MB to 10–25 MB. More frequent rotation overall. Admins can still tune via db-connection.recovery.size-limit. Cons: marginally higher I/O on busy installs; might surface latent issues with rotation behavior under load that aren't exercised today; doesn't specifically address the startup-accumulation case.

3. Document the expected behavior. No code change; update Journal JavaDoc and admin docs to make the 100 MB ceiling and rotation cadence visible. Cons: doesn't help anyone who hits the same observation duncdrum did and reasonably reports it.

My lean: option 1, optionally combined with 3. Option 1 is surgical, solves the actual user-visible problem, and meaningfully improves post-crash recovery. Option 2 is more invasive for less direct benefit. Option 3 alone doesn't reduce the support burden.

A natural place to hook option 1 would be BrokerPool.postInit() or a new StartupTrigger priority that runs last, after AutoDeploymentTrigger has finished. Happy to draft a PR for whichever direction maintainers prefer — or for an option not on this list.

[duncdrum]

Thanks for the analysis. I don't see any negatives for calling rotate on as part of successful start up.

We would need to see what happens during a recovery run. But 1 + 3 sound good to me.

[duncdrum]

I m also wondering if containers should run in yolo mode by default?

Rebuilding or restarting an ephemeral container seems to negate the advantages of the log file, leaving us with just IO overhead for no gain.