Skip to content

reset_to never reconciles the git-tracked xorq catalog against tallyman pointers — back→forward reset restores an entry the durable catalog cannot reproduce, and reports success #52

Description

@paddymul

Problem

reset_to reconciles two different views of "what entries exist" with two
different mechanisms that are never checked against each other
(src/tallyman_core/catalog_state.py:413-429):

  • The git-tracked xorq catalog (entries/<hash>.zip +
    metadata/<hash>.zip.metadata.yaml, plus catalog.yaml's entries/aliases)
    is reconciled solely by git reset --hard <commit> (:422). git can only
    restore a recipe zip that was actually committed.
  • The untracked tallyman artifacts (the entries/<hash>/ build dirs,
    result_cache, compute_cache) are reconciled against the entry_hashes /
    result_cache / compute_cache pointer lists in catalog.yaml via the
    bullpen: prune_entries (:248), prune_result_cache/prune_compute_cache
    (:280-285) retire to the bullpen, restore_from_bullpen (:288-314) copies
    back.

reset_to runs materialize → prune_* → restore_from_bullpen and never
re-validates the git-tracked xorq recipe set against the tallyman pointers, nor
re-runs xorq catalog add for an entry whose pointer is present but whose .zip
is not. So when the recipe set is incomplete, the tallyman view (restored by
copy from the bullpen) claims entries that the durable xorq catalog cannot
reproduce, and reset returns success.

Trace from the real demo flow (genesis → catalog_create → catalog_create → reset_to), driven through the MCP tools so the checkpoint wrapper fires as in
production:

[after both]   dirs=[023fb..,49b47..]  tracked_zips=[49b47..zip]   head=2
[reset->s1]    dirs=[49b47..]          tracked_zips=[49b47..zip]   head=1
[forward->s2]  dirs=[023fb..,49b47..]  tracked_zips=[49b47..zip]   head=2

After back→forward the tallyman view shows two entries; the durable xorq catalog
still holds one zip. The 023fb recipe round-trips through the bullpen as a
build-dir copy but is never regained as a committed recipe.

Impact

The bullpen masks the loss instead of surfacing it: the tallyman entry comes
back, so reset looks correct, while "git-backed, recompute from history" is
quietly false for that entry — its only copy is the evictable bullpen dir, and
the content-addressed .zip it should recompute from does not exist. The
recovery model the bullpen exists to support is the thing being undermined.

This is most often triggered today by #48 (every xorq catalog add after the
first silently no-ops, so the zips were never committed). It is a distinct cause,
not the same bug: even with #48 fixed, checkpoint_catalog already anticipates a
commit that does not land (returns None on index.lock contention with xorq's
own add, :388-390) and add_entry is best-effort (xorq_catalog.py:63-68).
Any interrupted or failed add leaves a dir + pointer with no zip, and
reset_to will mask it the same way, because the missing-cross-view-check is the
real gap.

Suggested fix

Give reset_to an explicit reconciliation between the git-tracked xorq catalog
and the tallyman pointers after git reset --hard, before it reports success:
compare the tracked entries/*.zip set against entry_hashes, and for any
pointer whose recipe is missing, either re-register it (xorq catalog add from
the restored build dir) or fail loudly rather than returning a catalog whose two
views disagree. A consistency assertion at the end of reset_to (tracked-zip set
== entry_hashes, and Catalog.from_kwargs(init=False) opens) would turn the
silent divergence into an observable error.

Context

Found while writing the failing integration tests in #50 (catalog↔xorq
coexistence and reset round-trips), main @ a18b14f, xorq 0.3.29 (1f46392e).
Related to #48 (recipes never committed) and #49 (authored state dropped on
merge) — all three are tallyman and the xorq catalog disagreeing about what is
durable — but a distinct cause: reset-time reconciliation has no cross-view
check here, vs consistency-check rejection (#48) and merge-time reconstruction
(#49). Group B of #50 pins this end to end.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions