Skip to content

Commit bdfcfe1

Browse files
paddymulclaude
andcommitted
docs: reconcile staged plan with merged PR #57 / #50
#48 fixed (PR #57): alias bookkeeping now catalog.yaml keys (alias_map/ alias_history), #53 sidecar rejected, #48 closed. Mark stage-1 item 1 done. Surfacing partly done — remainder tracked as #56 (catalog_registered dies in _run_and_record; set_alias discards add_alias bool). #52 (reset divergence) is now the live stage-1 fix; PR #50 Group B stays red until it lands. Reframe #49 per #54: authored state belongs in catalog.yaml (fix durability, not relocate). Rebuild script unblocked on the #48 side. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent e55a888 commit bdfcfe1

1 file changed

Lines changed: 53 additions & 63 deletions

File tree

plans/staged-catalog-perf-reactive.md

Lines changed: 53 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -34,75 +34,65 @@ then produces a clean parking catalog for stage 2 to measure.
3434
durable catalog cannot reproduce and reports success. Compounded by #48 today,
3535
but independent: any failed/interrupted `add` (which `checkpoint_catalog` already
3636
anticipates, `catalog_state.py:388-390`) triggers it after #48 is fixed.
37-
- **#49** — authored state (charts, post_processing, …) stored as extra
38-
`catalog.yaml` keys is dropped when xorq rebuilds `catalog.yaml` from
39-
entries+aliases on a merge. **Deprioritised out of stage 1's critical path:**
40-
latent only under multi-user `pull`/sync, which does not happen single-user /
41-
`--no-sync`. Stays filed; not fixed this stage.
37+
- **#49 / #54** — authored state (charts, post_processing, stats, notebook, …)
38+
stored as `catalog.yaml` keys is dropped when xorq rebuilds `catalog.yaml` from
39+
entries+aliases on a merge. **Resolution direction settled by #54 (and the #57
40+
precedent):** this state is shareable catalog content and *belongs* in
41+
`catalog.yaml` so it clones and versions with the catalog — #49's proposed
42+
relocation to an out-of-repo store would stop that. So the fix is on the
43+
*durability* side (make the keys survive xorq's merge-rebuild — e.g. per-entry
44+
state into the entry metadata sidecar, which xorq treats as opaque), not
45+
relocation. **Deprioritised out of stage 1's critical path:** the drop is latent
46+
only under multi-user `pull`/sync, which does not happen single-user /
47+
`--no-sync`. Stays filed (#49, #54); not fixed this stage.
4248

4349
### Work
4450

45-
1. **#48 fix — fold the bookkeeping into `catalog.yaml` (decided 2026-06-14).**
46-
Stop writing `aliases.json` / `alias_history.json` as separate files; store
47-
their content as tallyman keys in `catalog.yaml`, the same way every other
48-
tallyman-owned thing already is (`post_processing`, `stats`, `charts`,
49-
`display_configs`, `notebook`, `entry_hashes`, `result_cache`,
50-
`compute_cache`). The two files are the lone anomaly that breaks #48; folding
51-
them in removes them from the tracked file set, so `assert_consistency` (which
52-
checks the file *set*, not `catalog.yaml`'s contents, catalog.py:768-785)
53-
passes and `xorq catalog add` stops no-opping. xorq's add/add-alias
54-
read-modify-write the whole dict and round-trip unknown keys on the `--no-sync`
55-
path tallyman always uses.
56-
- **Why this over a separate file/repo.** `catalog.yaml` is git-tracked and
57-
checkpoint-committed, so `git reset --hard` already rolls alias state back —
58-
no sibling repo, no lockstep commits, no bespoke reconciliation, and
59-
`test_reset_rolls_back_alias_state` stays green via git. Strictly less code
60-
than the alternatives.
61-
- **Known caveat (deferred, not regressed).** `catalog.yaml` keys are what xorq
62-
drops when it rebuilds the file from entries+aliases on a merge-conflict
63-
`pull` (#49). Latent under single-user / `--no-sync`, and the same exposure
64-
the authored state already accepts. The multi-user fix is to move all
65-
project-global state out of `catalog.yaml` together (#48 + #49 jointly) —
66-
`catalog.yaml`-now defers that without foreclosing it.
67-
- **Optional refinement.** `set_alias` already calls `_xcat.add_alias`
68-
(`aliases.py:111`), so xorq's own `aliases` key holds `{alias: latest_hash}`
69-
and `aliases.json` partly duplicates it. Folding only `alias_history` into a
70-
tallyman key (and reading current aliases from xorq's native `aliases`)
71-
shrinks tallyman bookkeeping and survives merges for the current-alias half.
72-
- Audit the full set of non-xorq files the checkpoint commits under
73-
`catalog_dir`, not just the two named, in case anything else needs the same
74-
treatment.
75-
- The deferred `test_reset_rolls_back_alias_state` guard (PR #50) ships *with*
76-
this fix — the rollback path changes, so the guard travels with the change
77-
per the TDD rule.
78-
79-
2. **#52 fix.** Add an explicit cross-view reconciliation + assertion at the end of
80-
`reset_to`: compare the tracked `entries/*.zip` set against `entry_hashes`,
81-
re-register (or fail loudly on) any pointer whose recipe is missing, and assert
82-
`Catalog.from_kwargs(init=False)` opens. Turns the silent divergence observable.
83-
84-
3. **#48 secondary — surface the silent swallow.** `add_entry` is best-effort and
85-
discards its result (`xorq_catalog.py:63-68`, `build.py`); expose a
86-
`BuildResult.catalog_registered`-style signal, log failures at ERROR, and emit a
87-
one-time companion warning. PR #50's Group D pins this.
88-
89-
4. **Reproducible parking-catalog rebuild script.** No migration/repair of the
90-
existing broken catalog — rebuild from scratch through the *fixed* machinery.
91-
One script drives the parking sources through `catalog_create`/`catalog_revise`
92-
to produce the canonical catalog, and emits both corpus variants for stage 2
93-
(full ≈10M+ rows, truncated ≈1M). This replaces "repair" on the task list and
94-
doubles as the stage-2 corpus generator. Depends on #48 + #52 being fixed first
95-
(otherwise the rebuild is born broken again).
51+
1. **✅ DONE (PR #57, merged) — #48 fix, alias bookkeeping in `catalog.yaml`.**
52+
`aliases.py` now reads/writes `alias_map` / `alias_history` as `catalog.yaml`
53+
keys via `catalog_state` (no `aliases.json`, no out-of-repo sidecar — #53's
54+
alternative was rejected). The tracked file set is back to xorq's expected set,
55+
`assert_consistency` passes, `xorq catalog add` stops no-opping. `reset_to`'s
56+
`git reset` rolls the alias keys back with the rest of `catalog.yaml` — no
57+
separate reconcile path. **#48 closed.** `test_reset_rolls_back_alias_state`
58+
shipped with the fix and guards the rollback; the non-xorq-fileset audit
59+
landed too (`_TRACKED` is `catalog.yaml` only).
60+
61+
2. **OPEN — #52 fix (the live stage-1 work item).** Not touched by #57: that fix
62+
reconciles *alias state*, not the xorq `.zip`-vs-`entry_hashes` divergence. Add
63+
an explicit cross-view reconciliation + assertion at the end of `reset_to`:
64+
compare the tracked `entries/*.zip` set against `entry_hashes`, re-register (or
65+
fail loudly on) any pointer whose recipe is missing, and assert
66+
`Catalog.from_kwargs(init=False)` opens. Turns the silent divergence
67+
observable. (PR #50's Group B is the failing-test half, already on `main`.)
68+
69+
3. **PARTLY DONE — surface the silent swallow.** #57 exposed
70+
`BuildResult.catalog_registered` and logs the failed entry-`add` at ERROR. The
71+
remainder is **#56 (open):** `catalog_registered` is computed but dies in
72+
`_run_and_record` (never propagated to the caller / companion), and `set_alias`
73+
still discards `add_alias`'s bool, so a failed *alias* add stays swallowed.
74+
Finish #56 to actually surface both failures.
75+
76+
4. **OPEN — reproducible parking-catalog rebuild script.** No migration/repair —
77+
rebuild from scratch through the now-fixed machinery. One script drives the
78+
parking sources through `catalog_create`/`catalog_revise` to produce the
79+
canonical catalog and emits both corpus variants for stage 2 (full ≈10M+ rows,
80+
truncated ≈1M). Doubles as the stage-2 corpus generator. Now unblocked on the
81+
#48 side; still wants #52 fixed first so a back/forward rebuild stays durable.
82+
**Blocking input:** confirm the parking source parquet(s) still exist / where.
9683

9784
### Tests / acceptance
9885

99-
- PR #50's failing integration tests (Groups A/B/D) go green.
100-
- One end-to-end demo-walk invariant test: genesis → multiple creates → revise →
101-
reset back → forward → back, asserting at every step that the catalog opens,
102-
every recipe is durable as a tracked `.zip`, alias state is correct, and (where
103-
in scope) authored state survives.
104-
- TDD split per house rule: failing tests (PR #50) seen red on CI first, then the
105-
fixes; the alias-rollback guard ships with the fix commit.
86+
- ✅ PR #50's failing integration tests (Groups A/B/D) landed on `main`
87+
(`80d2e9b`); #57 made the #48 ones (A, and D rewritten) pass and added Group F
88+
(alias registration durability). The TDD red-then-green split is satisfied for
89+
#48.
90+
- **Still red / unwritten:** Group B (reset divergence) stays failing until #52 is
91+
fixed — that's the next fix to land.
92+
- One end-to-end demo-walk invariant test still wanted: genesis → multiple creates
93+
→ revise → reset back → forward → back, asserting at every step that the catalog
94+
opens, every recipe is durable as a tracked `.zip`, alias state is correct, and
95+
(where in scope) authored state survives. Lands with the #52 fix.
10696

10797
---
10898

0 commit comments

Comments
 (0)