fix(chroma): retire the HNSW bloat guard — it corrupts the index it was meant to protect

[joshuafontany]

Closes #2106.

Who is asking

Your mission opens: "Memory is identity. When an AI forgets everything between conversations, it
cannot build real understanding." We hold that same thesis, which is why our shape looks the way it
does. We create instances of your code as libraries in ours:

  sensoriums/memory/content          ← a parallel memory capability to ~/.mempalace
  sensoriums/mark-twain/content
  sensoriums/kumulipo/content
  ~/.mempalace                       ← the operator's own install, when present

Three sovereign content palaces, one per sensorium (extendable), plus whatever the operator already runs. Peers,
never satellites. We consume mempalace as a library from our own NDJSON holder processes, one per
plane, with the embedding contract inverted: vectors arrive on the wire, so no model loads in the
store process. Everything below comes out of running that shape hard.

What this changes

① Retire _HNSW_BLOAT_GUARD, adopt chromadb's own write defaults. The guard pinned
batch_size=2 / sync_threshold=2 to answer #1579. On chromadb 1.5.9 it buys nothing (a 5-record
collection at 100/1000 reads back whole from a fresh process, and _segment_appears_healthy returns
True) and it costs correctness at scale (a 26k-drawer mine fires ~13,000 full segment rewrites,
wedging the compactor and opening 13,000 torn-persist windows). Measurements and repro in #2106.

The knobs move into _hnsw_creation_metadata(), which centralizes them so a multi-collection palace
can tune per collection. One finding worth more than the refactor: it keeps the legacy
metadata= keys deliberately. The modern configuration= API stores the same parameters in
configuration_json and leaves collection.metadata empty — which silently blinds the divergence
guard, the cosine-space detector (ChromaCollection.distance_metric), and _read_sync_threshold,
all of which read collection_metadata.

② Isolate a failed upsert sub-batch instead of aborting the file. Same subsystem, same load: one
bad sub-batch currently loses every record after it in that file. It rides along because it shows up
in exactly the conditions ① is about; say the word and I will split it out.

A note on the rebase

develop grew a second create_collection site (**_HNSW_BLOAT_GUARD) after the branch point.
The cherry-pick did not touch it, and Python would not have caught the dangling name until that path
ran — so it is re-pointed to _HNSW_WRITE_DEFAULTS inside the first commit. Both commits are
self-consistent: neither leaves a reference to a name it removed.

Verification

Rebased onto develop @ aa89bd8.

• ruff check . → clean · ruff format --check . → 192 files already formatted
• pytest tests/test_backends.py tests/test_hnsw_capacity.py tests/test_miner.py → 264 passed
  (clean develop in the same worktree: 262 — the +2 are the tests added here)
• pytest tests/ --ignore=tests/benchmarks → 3398 passed, 31 skipped, 5 failed

On those 5 failures, stated plainly: all five are
test_init_filters_sys_path_from_leaked_pythonpath, and they fail identically on clean develop
in the same directory — I ran that control precisely because this test reacts to where it runs
rather than to the code under it. Zero regressions from this branch; the failures come from a leaked
PYTHONPATH in my environment, which is the very thing the test asserts about.

Local ruff reads 0.15.20 against your pinned 0.15.14, so please let CI have the final word on lint.

[fatkobra]

I agree with retiring the 2/2 HNSW thresholds; the 100/1000 direction matches ChromaDB’s documented defaults and should substantially reduce write amplification.

I do not think the second commit is safe as written, however.

After a middle upsert batch fails, later successful batches still carry the current source_mtime and normalize_version. On the next mine, file_already_mined() returns true when it finds any current drawer group for the source, so the logged “re-run to retry the missing spans” path is unlikely to happen. The file can remain permanently partial.

There is also a closet consistency problem: drawer_ids is generated from every chunk, including failed batches, while all_metas contains only successful batches. That can create pointers to missing drawers and misalign locator metadata.

Separately, _hnsw_creation_metadata() appears to be defined but not used by the actual collection-creation paths, which continue to expand _HNSW_WRITE_DEFAULTS directly.