Multicore safety protect shared mutable state with mutexes and atomics

[balat]

Summary

This PR adds multicore safety to shared mutable state using mutexes and atomics. This fixes at least one multicore test failure (test_concurrent_low crashes when append_exn is not protected by a mutex).

Thread-safety fixes

• Protect Buffer.t in append_only_file.ml with an Eio.Mutex — the race between append_exn and flush is triggered by test_concurrent_low without the fix
• Replace mutable suffix/prefix fields in file_manager.ml with Atomic.t to prevent use-after-close during GC swap
• Protect Hashtbl.t in dict.ml with an Eio.Mutex (guards refill which yields on I/O before mutating)
• Replace mutable chunks array in chunked_suffix.ml with Atomic.t for consistent snapshots during concurrent reads
• Fix TOCTOU on during_batch/running_gc checks in store.ml by moving them inside Repo.lock
• Prevent deadlock when calling GC or split from a batch(~lock:true) callback
• Replace global ref with Atomic.t in watch.ml
• Protect global Hashtbl.t with Eio.Mutex in irmin_fs.ml and irmin_mem.ml

Cleanup (from lyrm review checklist on PR #2149)

• Remove unused Mutate constructor from Metrics.update_mode — incompatible with the Atomic.compare_and_set retry loop, and never instantiated by any caller
• Remove redundant | e -> raise e exception re-raises in irmin_fs_unix.ml, inode.ml, tree.ml
• Update outdated Lwt references in indexable_intf.ml and node_intf.ml documentation
• Migrate irmin.mli code examples (custom_merge, sync) from Lwt to direct-style
• Improve frightening comments in watch.ml / watch_intf.ml: replace "Big Yikes" with actionable error, document stream_iter limitation, explain set_watch_switch workaround
• Review all Atomic.set vs compare_and_set usage: no correctness issues found

Not addressed in this PR

• f () |> function cleanup (~33 occurrences) — mechanical, better as a dedicated PR
• eio_pool.ml vs Eio.Pool — not redundant (provides post-failure check and pool clearing)
• Pin-depends removal (index, irmin-watcher) — tracked in separate PRs

Performance

Benchmarked with bench_inline (50k contents, 5k reads), stabilised with taskset -c 0 and pre-built binaries — no measurable regression (write +1.6%, read p50 +1.0%, within noise).

Test plan

• dune runtest test/irmin-pack/ — 252 tests pass
• Removing the append_only_file mutex causes test_concurrent_low to crash, confirming the race is real
• dune build src/irmin/ src/irmin-pack/ src/irmin-fs/ compiles cleanly

Please read each commit individually.

[balat]

An anlysis of the remaining tasks from @lyrm's list, by Claude, gives the following text:

Notes on lyrm's review checklist (PR #2149)

Items investigated but not changed

• progress dependency in src/irmin-pack/unix/dune: not redundant. The progress library is actively used in Utils (progress bars), traverse_pack_file.ml (Progress.Line.Using_int63), and checks.ml (Object_counter). No action needed.

• ?lock parameter in batch: actively used. Only irmin-pack/io/store.ml implements the locking behaviour (Eio.Mutex.use_rw when ~lock:true), and it is called with ~lock:true from Commit.v in store.ml:180. Other backends ignore it (?lock:_), which is fine — they don't need it. Not dead code.

• eio_pool.ml vs Eio.Pool: not redundant. The custom pool provides two features that Eio.Pool lacks: (1) a check callback for asynchronous validation after a use fails, and (2) a clear function that tracks in-use resources. Used for mkdir_pool (size 1, serialising directory creation) and openfile_pool (size 200, fd reuse) in irmin_fs_unix.ml.

• Atomic.set vs compare_and_set review: all uses of Atomic.set in src/ are either inside a mutex, unconditional overwrites on caches, or one-shot initialisations. No correctness issue found. pack_key.ml and metrics.ml already correctly use compare_and_set with retry loops where needed.

• test/irmin-pack/data/version_1_large/README.md: contains Lwt-style code but documents the program that generated the test data with irmin-pack.2.10.2. It is a historical record and should not be updated.

Items deferred to other PRs

• f () |> function → match f () with (~33 occurrences): mechanical cleanup, better as a dedicated PR
• Pin-depends removal (index, irmin-watcher): tracked separately (see PR Integrate eio watcher #2336 for irmin-watcher)