Data integrity: one bug wiped a live store, and there are no backups

[asim]

What happened

wallets.json was written by two different maps — the credit ledger (wallets) and the key store (userWallets) — because a rename moved the second onto a filename the first already owned. Each save replaced the other's contents wholesale. A live account's 560 credits read zero, and every wallet lost its address and private key from that file.

Fixed in 56bef80. Nothing was lost, but only by luck:

• the keys survived in trade_wallets.json, which is only ever read, never written
• the balances survived in transactions.json, which neither writer touches and which happens to record the balance after every transaction

Neither was a designed safety net. Change either accident and the loss is permanent, because the only backup is a VM snapshot from the previous day.

Why it was possible

Every store is a whole-file JSON blob, rewritten in full on each change. internal/data.writeAtomic makes that crash-safe — temp file, fsync, rename — so a power cut cannot truncate a file. It does nothing about a logic error: a write of the wrong map is atomic, durable, and complete.

So the blast radius of any bug touching a store is the entire store, in one write, with no history.

Plan, in priority order

1. Backups — nothing exists today

• mu backup producing a timestamped tar.gz of ~/.mu/data
• Scheduled daily, and automatically before a deploy or restart, which is when new code first touches the stores
• Off-box copy — the S3_* settings are already configured for file storage and can hold these
• Retention: daily for a fortnight, weekly for a quarter
• mu restore <archive> and a documented drill, because an untested backup is not a backup

2. Keep the previous generation of every store

Highest value per line of code. These files are small; keeping the last few versions is nearly free.

• writeAtomic rotates: x.json → x.json.1 → x.json.2, keeping the last 3
• Turns "destroyed" into "copy the previous generation back"
• Would have made today a thirty-second recovery instead of an investigation

3. Refuse a write that destroys a store

The check that would have prevented this outright, rather than caught it afterwards.

• Before saving, compare the new record count against what is on disk
• Refuse when a non-empty store would become empty, or shrinks by more than a set proportion, unless the caller explicitly opts in
• Log loudly and keep the old file
• Today's write was an all-empty map over ~500 accounts — the crudest possible version of this stops it

4. Detection

• Test that no store file is written from two different values (56bef80)
• Log a record count per store at startup, so a collapse is visible in the log rather than in a support message
• Warn when a store has shrunk substantially since the last run

5. Write down what is reconstructable from what

• docs/DATA.md: every store, its owner, what depends on it, and what it can be rebuilt from
• Balances rebuild from transactions.json — implemented, and it should be documented rather than rediscovered
• Anything with no reconstruction path is a candidate for an append-only log beside it

6. Longer term: shrink the blast radius

Whole-file rewrite is the underlying hazard. Two directions, not both:

• Append-only logs for anything that is money, with state derived from the log. transactions.json already works this way, which is exactly why it survived.
• SQLite with WAL, which gives transactional writes, a real backup command, and no whole-file rewrite. A large change; worth considering when the JSON stores next cause pain.

The honest part

The collision arrived in aa574e8, but I made it destructive by adding a write to GetOrCreateWallet's repair path without checking what else wrote that filename. walletsFile = "wallets.json" was on screen at the time. A one-line grep would have shown two owners.

Item 3 exists because process did not catch it and would not have. The write should have been impossible, not merely inadvisable.