ADR-324 policy receipt ledger grows without bound until all MCP tools hit policy-state-lock-timeout

[sparkling]

Summary

ADR-324 stores every MCP authorization decision in .claude-flow/policy/state.json. The receipt array has no retention, segmentation, or checkpointing. Every tool call then:

1. reads and parses the complete state;
2. clones the complete receipt history;
3. verifies the complete hash chain from sequence 0;
4. pretty-serializes and atomically rewrites the complete state;
5. does most of that while holding .claude-flow/policy/state.lock.

Lock wait is fixed at 5 seconds. Once a real project accumulates enough receipts, one valid transaction takes longer than another caller is allowed to wait. The result is a cross-cutting outage: unrelated memory_search, hooks_route, hooks_pre_task, and other MCP calls fail with policy-state-lock-timeout.

This is not an AgentDB/WAL defect and should not be repaired by touching .swarm/memory.db.

Reproduction and observed evidence

Environment:

• published ruflo@3.38.21
• @claude-flow/security@3.0.0-alpha.14
• Node 24.14.1
• Linux
• policy mode legacy, no configured rules, budgets, or approvals

Six long-running projects on one host currently have:

Project	state.json	receipts
semantic-builder	193,280,467 bytes	133,598
semantic-control	128,981,429 bytes	89,209
semantic-query	118,963,331 bytes	82,274
oxigraph	92,556,373 bytes	63,983
semantic-fabric	76,671,433 bytes	53,018
semantic-modelling	36,073,068 bytes	24,956

The failure is recurrent in Codex session evidence from at least 2026-08-15 through 2026-09-02. A failing call waits almost exactly five seconds and returns:

Mcp error: -32603: Failed to execute MCP tool 'memory_search':
policy-state-lock-timeout

The lock file was absent immediately after the failure, so this is not a permanently stale lock. It is normal live contention against a transaction whose duration has exceeded the waiter budget.

Read-only timing against the existing, valid ledgers (no state or DB mutation):

semantic-query, 82,285 receipts

• read: 285.5 ms
• parse: 270.5 ms
• full verifyLedger(): 1,488.3 ms
• exportState() structured clone: 944.8 ms
• compact stringify: 379.9 ms
• process RSS: 555 MB for verify/clone

semantic-builder, 133,612 receipts

• read + parse: 899.8 ms
• full verifyLedger(): 2,263.6 ms
• exportState() structured clone: 1,424.2 ms
• pretty stringify: 771.2 ms
• total before the atomic disk write completes: 5.49 seconds
• process RSS: 1.21 GB

A second caller therefore cannot satisfy the current five-second lock budget even when the first caller is healthy.

Concurrency amplifies this. A long-lived Codex app-server currently retains four ruflo mcp start children per active project. Unlike the orphans fixed by #2234, they retain a non-PID-1 app-server parent, so the parent-death watchdog does not reap them. The unbounded/O(n) transaction remains a bug with one client; multiple clients simply reach the failure earlier.

Source cause

Current v3/@claude-flow/cli/src/services/policy-runtime.ts:

• LOCK_WAIT_MS = 5_000;
• withPolicyTransaction() acquires the lock, loads the whole state, runs the operation, calls engine.exportState(), calls full engine.verifyLedger(), and rewrites the complete JSON state;
• writeJsonAtomic() pretty-prints the complete object;
• acquireLock() catches every error as contention and releases by unconditionally unlinking the path, without an ownership nonce.

Current v3/@claude-flow/security/src/policy/engine.ts:

• evaluate() appends one receipt to an unbounded array;
• exportState() structured-clones the full array;
• verifyLedger() re-hashes the entire chain from receipt zero on every call.

ADR-324 is Accepted, dated 2026-07-28, and says implementation is complete. It explicitly promises “a small locked state transaction.” Its cited microbenchmark excludes filesystem lock contention and disk latency and does not exercise a mature ledger, so it does not cover this production shape.

Issue #2917 already warns that receipt design must avoid unbounded ledger growth, but it concerns per-candidate trust and does not track this existing runtime outage.

Required behavior

Do not solve this by truncating receipts, disabling receipts in legacy mode, deleting policy state, increasing the timeout alone, or weakening full-audit verification. Existing evidence must remain verifiable.

Recommended architecture

Separate mutable policy state from receipt history:

1. Keep mode, rules, budgets, usage, approvals, and a small authenticated ledger head/checkpoint in the transactional state.
2. Store receipts in append-only bounded segments (or an equivalent transactional store), each binding:
   • start/end sequence;
   • previous segment/receipt hash;
   • final receipt hash;
   • segment content digest;
   • key id/authentication when configured.
3. Append and verify only the new suffix on the hot path. Run full historical verification through explicit policy verify/audit, traversing every immutable segment.
4. Rotate by receipt count and byte size, not just count.
5. Use a crash-consistent journal/commit protocol so a receipt, budget consumption, approval use, and ledger-head update are one recoverable transaction. A crash must neither lose authority evidence nor permit budget/approval replay.
6. Make lock ownership explicit with a random nonce; only the owner may unlink its lock. Treat only EEXIST as contention. Surface permission, path-type, and I/O failures directly.
7. Keep a bounded wait with owner/age diagnostics, but do not use a larger timeout as the scalability fix.
8. Report receipt count, on-disk bytes, segment count, oldest/newest sequence, and migration state in policy status / doctor.

Existing-ledger migration

Migration must be data preserving and fail closed:

1. Read and fully verify the original ledger before writing.
2. Build segments in a temporary sibling directory.
3. Verify every original sequence/hash and the new segment manifest.
4. Preserve the exact original file as a recoverable archive until promotion is sealed.
5. Atomically promote only after all verification succeeds.
6. Update the external trust anchor coherently for observe/enforce projects.
7. On any interruption, retain the original authoritative state and ignore incomplete staging.
8. Prove the audit command returns the same receipt count, first/last sequence, and final hash before and after migration.

Regression tests

• 0, 1, 1,000, 100,000, and 1,000,000 receipts;
• 1, 4, and 16 concurrent MCP clients;
• bounded hot-path latency and memory independent of total history;
• rotation at both count and byte thresholds;
• crash injection before/after append, mutable-state write, manifest write, and promotion;
• tamper one receipt, segment, manifest, head, and HMAC anchor;
• approval and budget replay after every crash point;
• migration of unsigned legacy, signed observe, and signed enforce state;
• full audit across segments;
• lock ownership race, stale-owner recovery, permissions, symlinks, and non-regular lock paths;
• a release test that grows a real published artifact past the rotation threshold and calls multiple ordinary MCP tools concurrently.

Related

• MCP servers orphan on every Claude Code restart — ~1 GB leaked, and a stale server silently serves the tool surface on superseded code #2234: parent-death cleanup for true PID-1 MCP orphans; does not cover children retained by a long-lived Codex app-server.
• Design proposal: policy-governed per-candidate memory retrieval trust #2917: per-candidate policy receipts, which would accelerate this growth.
• [Dream Cycle 2026-08-31] security: MCP governance policy never enforced at runtime (opt-in fix, ACCEPT) + intelligence,swarm scan #3138: runtime policy enforcement rollout.
• init-generated guidance and project skills contradict MCP-first contract and emit stale tool calls #3153: generated MCP-first guidance; using MCP correctly currently makes this policy bottleneck more visible.