Skip to content

docs: reactive catalog recalc & staleness — research snapshot#51

Open
paddymul wants to merge 9 commits into
mainfrom
docs/reactive-catalog-recalc
Open

docs: reactive catalog recalc & staleness — research snapshot#51
paddymul wants to merge 9 commits into
mainfrom
docs/reactive-catalog-recalc

Conversation

@paddymul

@paddymul paddymul commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Docs-only. Adds plans/adr-reactive-catalog-recalc.md, a research snapshot of the recalc / staleness design review. No code.

Intent

A tallyman project is a few raw sources and many derived expressions, and changes don't reach dependents — they go silently stale, because (1) from_catalog dependencies are by-value (B pins the version of A it was built on, io.py:87) and (2) xorq invalidates lazily at execute() with no propagation, while tallyman's SnapshotStrategy keys are content-blind. The goal is on-demand recompute (a "recalc" / scan-on-load) that keeps each entry reproducible and recoverable.

Guiding principle: use xorq natively wherever possible; patch only when xorq has no way, or its way is too slow for interactive use.

What the doc captures

  • Research — what xorq does natively (cache strategies, pull-based invalidation, HashingTag provenance, deep-loading catalog composition, the git-backed catalog) vs what tallyman does today (anonymous result.parquet reads, path-matched DAG, source_identity).
  • Answered questions — follow-alias/pin-hash semantics; follow-through-renames; unbind for multi-parent children; archive-with-derived-overlay over hard delete; the git-backed + bullpen recovery model; and the chaining mechanism: a HashingTag-tagged read of the materialized result (records the edge natively, keeps the graph shallow).
  • Verified findings — the tagged-read spine works (0.019s build, 0.20s head off a 263MB result, edge recovered via walk_nodes); xorq composition deep-loads (bind.py:115); and tallyman commits aliases.json/alias_history.json into the xorq catalog repo, so xorq's assert_consistency fails and catalog add silently no-ops for every entry after the first — entry recipes never reach git #48.
  • Open questions — source-identity mode (cas vs salt), cascade/transaction semantics, archive overlay, trigger model, determinism, deep-chain build-cost measurement.

Related

🤖 Generated with Claude Code

@paddymul paddymul force-pushed the docs/reactive-catalog-recalc branch from 16c99c0 to bdfcfe1 Compare June 15, 2026 02:06
@paddymul paddymul mentioned this pull request Jun 15, 2026
Closed
paddymul and others added 9 commits June 18, 2026 13:07
@paddymul @claude
docs: ADR — reactive catalog recalc & staleness research snapshot
1f2c296 
Captures intent, the xorq-native vs tallyman mapping, decisions reached, and
open questions from the recalc/staleness design review. No code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: staged plan — catalog coexistence, perf lifecycle, reactive recalc
5e95aa7 
Sequences the reactive-catalog work into three gated stages:

- Stage 1: close the tallyman/xorq catalog coexistence cluster (#48, #52;
  #49 deprioritized for single-user/--no-sync), rebuild-from-scratch script
  doubling as the stage-2 corpus generator, end-to-end demo-walk invariant.
- Stage 2: reframed around the expression lifecycle (build/load/execute
  counts as the explanatory variable, cache key strategy second-order);
  consolidate perf instrumentation behind configurable log levels; two-tier
  page-load harness (Playwright ground truth + python proxy, calibrated);
  full/truncated parking corpora. The harness is the instrument approaches
  are compared with.
- Stage 3: reactive design gated on stage-2 numbers; tagged-read spine
  landable early.

Open: stage-1 bookkeeping-file location (recommend sibling tallyman_state
repo). Relates to #48, #52; builds on the reactive-catalog ADR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: decide #48 bookkeeping storage — fold aliases into catalog.yaml
e5394c4 
Retire the sibling tallyman_state repo recommendation. aliases.json /
alias_history.json are the lone tallyman state stored as separate files;
every other tallyman key already lives in catalog.yaml. Folding them in
removes them from the tracked file set so assert_consistency passes (it
checks the file set, not catalog.yaml contents), preserves git-based reset
rollback for free, and is less code than a separate file/repo. Caveat:
inherits #49's merge-rebuild fragility, deferred under single-user/--no-sync.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: reconcile staged plan with merged PR #57 / #50
2f06e3c 
#48 fixed (PR #57): alias bookkeeping now catalog.yaml keys (alias_map/
alias_history), #53 sidecar rejected, #48 closed. Mark stage-1 item 1 done.
Surfacing partly done — remainder tracked as #56 (catalog_registered dies in
_run_and_record; set_alias discards add_alias bool). #52 (reset divergence)
is now the live stage-1 fix; PR #50 Group B stays red until it lands. Reframe
#49 per #54: authored state belongs in catalog.yaml (fix durability, not
relocate). Rebuild script unblocked on the #48 side.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: reconcile plan with merged PR #58; file Stage 2 perf issues #59/#…
c7460b4 
…60

#52 closed (PR #58): reset_to now asserts tracked recipe set == entry_hashes
and raises on divergence (fail-loud, settling the recovery-policy question).
Stage 1's catalog-coexistence prerequisites (#48, #52, #49) are all closed.
Cross-link Stage 2 deliverables to new issues: #59 (two-tier perf harness)
and #60 (configurable perf instrumentation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: reconcile ADR set with #74 (coherence pass); file reactive prer…
88a174a 
…equisites #84-#89

Resolutions from the 2026-06-16 design review reconciling the reactive-recalc plan with merged PR #74: the materialization boundary is policy (structural cache_worthy stays live, measured value-per-byte runs in shadow, flip on data — #30 softened from remove-now); the cross-entry DAG is restorable now via a manifest parents field (only the follow-the-concept layer stays deferred); cas is a prerequisite, not an open question (recompute-on-cold-read); read-intent rides the same parents field {hash, ref, follow} with previous_version disambiguation; determinism is detect-and-warn (structural via a hash predicate, execution via a lint + #83).

Touches all four ADRs in plans/. Prerequisites filed #84-#88, epic #89; measurement #82, soundness #83.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: reconcile reactive-recalc plans with PRs #103/#104
c55762e 
PR #103 (native tallyman_core.catalog store) and PR #104 (removed the
on-demand result.parquet layer) merged to main; merge main in and update
the reactive plan set to match the current tree.

- adr-reactive-catalog-recalc.md: add dedicated "Reconciliation with PR
  #104" and "PR #103" sections; correct the deleted ensure_result /
  result.parquet-survives, salt-mode, and catalog_parents-regex claims;
  re-point io.py / result_cache.py / lineage.py / aliases.py citations.
- staged-catalog-perf-reactive.md: Stage 1 bug cluster dissolved-by-
  construction; rebuild script OPEN -> PARTLY DONE; disk-usage/inspector
  follow-ups closed; split the parents-DAG prerequisite (capture done,
  reactive consumer remains); refresh the dependency diagram.
- adr-result-cache-cost-rubric.md / adr-source-identity-content-hash.md:
  strike ensure_result, fix the now-false "from_catalog reads
  result.parquet" claim, repoint drifted line numbers.

Attribution corrected after git archaeology: the manifest `parents` edge
+ parent_capture (#84) shipped via PR #91, and the cache-admission
instrumentation + compute_cache disk counting (#87) via PR #94 -- both
ahead of #103, not part of it. #103 only makes the parents edge durable
across reset (manifest.json in the recipe-zip member set).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: reconcile reactive plan with post-#103 state; pin no-migration …
af2f960 
…corpus decision

Verified the #103/#104 reconciliation against the merged tree (all code
citations accurate bar one drift, fixed: execute_seconds is manifest.py:38,
not :47-50) and brought the plan current with what shipped since:

- Deliverable 1: prose lifecycle model + refreshed cache map already shipped as
  docs/expression-lifecycle.md + docs/caching.md (PR #47); fix the stale
  "docs/caching-architecture branch" reference.
- Deliverable 2: record #59/#64 closed and both perf tiers shipped; remaining =
  regression-gate promotion (#45), #82 deep-chain measurement (gates Stage 3).
- Stage 3: determinism lint shipped (#88) -> mark PARTIAL not open; note the
  #80/#96 dangling-path failure is mitigated by the per-call path.exists
  self-heal; name epic #89; add the recompute-substrate holes the consumer
  inherits (#75/#76/#77/#97/#79/#81).

Corpus: per the project no-migration rule, build the stage-2 corpus FRESH in
native format from the source parquets, not by converting the pre-#103 on-disk
catalogs. rebuild_native_catalog.py is a migration convenience, not on the
critical path; its alias-revision pinning limit is out of scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul @claude
docs: file reactive consumer plan — staleness detection & recalc, seq…
b0dc991 
…uenced

Adds plans/reactive-staleness-recalc.md: the consumer side of epic #89 —
dependents reader, staleness predicates, recalc action, MCP/companion surfaces.

Leads with a "what remains" roadmap verified against the tree (2026-06-18):
land PR #124 (#80/#96) then PR #121 (#88), fix #115 (cold reconstruction is not
content-faithful — the one genuine hard blocker), then build the consumer. The
substrate is otherwise complete: #82/#83/#85 and substrate holes
#75/#76/#79/#81/#97 are all closed. Names the land-with quality items
(#111/#117/#118) and the deferred/out-of-scope set (#125/#77/archive-delete).

Corrects five code-level details against the source: enumerate via
build.list_entries (not read_tallyman_state, which misses uncheckpointed
entries); digest_for is stat-memoized (needs TALLYMAN_SOURCE_REHASH=1 for a
sound source-drift check); cache_clear lives on cached_result_expr;
build_and_persist takes recipe text and does not set the alias or checkpoint;
io.py:83 / parent_capture.py:9 still name the deleted lineage.catalog_parents.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@paddymul paddymul force-pushed the docs/reactive-catalog-recalc branch from a1a7127 to b0dc991 Compare June 18, 2026 19:11
@paddymul paddymul mentioned this pull request Jun 18, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@paddymul