Skip to content

Commit 46698ee

Browse files
paddymulclaude
andcommitted
docs: reflect PR #137 — result digest as a row multiset, tallyman_read_csv
Rewrite the Result digest concept: it is now snapshot_file_digest (SHA-256 of the baked snapshot parquet, ordered by original_row_order) — a row multiset, recorded for worthy entries only; cheap entries record none. Replaces the per-row repr()+sha256 description. Index the three docs PR #137 added: the canonical-ordering ADR (under ADRs) and the two digest-investigation findings (datafusion scan order, result_digest vs xorq staleness). Note tallyman_read_csv as the fourth in-code data-sourcing helper in the MCP authoring section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 6a74f4b commit 46698ee

2 files changed

Lines changed: 31 additions & 5 deletions

File tree

docs/architecture.md

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -211,11 +211,17 @@ unique by construction, Buckaroo sessions are keyed globally by content hash,
211211
not per project. The source-identity mode (`off` / `cas` / `salt`, default
212212
`cas`) is decided in [adr-source-identity-content-hash.md](../plans/adr-source-identity-content-hash.md).
213213

214-
**Result digest.** A second identity axis. The content hash keys the expression
215-
graph; the `result_digest` (a content hash of the executed rows, in order) keys
216-
the executed bytes. A mismatch during a cache self-heal points at execution
217-
nondeterminism (sampling, `now()`, unordered limits, impure UDFs) rather than a
218-
structural change.
214+
**Result digest.** A second identity axis, recorded for *worthy* (snapshot-baking)
215+
entries only. The content hash keys the expression graph; the `result_digest`
216+
keys the executed *bytes* as a row **multiset**, not a sequence. It is the
217+
SHA-256 of the entry's baked snapshot parquet (`snapshot_file_digest`), which the
218+
bake writes after sorting on a synthetic `original_row_order` and pinning the
219+
parquet write settings, so the bytes are reproducible run-to-run and the file
220+
hash is order-insensitive. Cheap, row-preserving entries record no digest — they
221+
have no snapshot to hash and recompute live. A mismatch when an evicted snapshot
222+
self-heals points at execution nondeterminism (sampling, `now()`, an impure UDF,
223+
source drift), not the unordered-scan row reshuffling the canonical ordering now
224+
absorbs. Design: [adr-result-digest-canonical-ordering.md](../plans/adr-result-digest-canonical-ordering.md).
219225

220226
**Alias and V_n versions.** An alias is a named, mutable pointer (for example
221227
`sales`) to the latest content hash of a logical entry. Each alias carries an
@@ -385,6 +391,11 @@ when in doubt, the code wins.
385391
a *proposed* cost-vs-size cache rubric. **Partially stale / not adopted:** the
386392
structural `cache_worthy` admission test it proposes to remove is still the
387393
live gatekeeper, and `ensure_result` it names was removed (#73).
394+
- [adr-result-digest-canonical-ordering.md](../plans/adr-result-digest-canonical-ordering.md)
395+
`result_digest` as a row multiset via a canonically-ordered snapshot hash,
396+
replacing the per-row Python digest (#137). **Current** (implemented: the
397+
digest is now `snapshot_file_digest`, and `tallyman_read_csv` injects
398+
`original_row_order`).
388399

389400
### Plans (`plans/`)
390401

@@ -412,6 +423,14 @@ when in doubt, the code wins.
412423

413424
### Research notes / experiment logs (`plans/`, `demo/`) — point-in-time records
414425

426+
The digest investigation behind #137 (current):
427+
[datafusion-scan-order-findings.md](../plans/datafusion-scan-order-findings.md)
428+
— why a parallel datafusion scan emits rows in a different order each run, and
429+
the polars-ingest decision — and
430+
[result-digest-vs-xorq-staleness.md](../plans/result-digest-vs-xorq-staleness.md)
431+
— why `result_digest` can't reuse xorq's content-aware cache staleness.
432+
433+
Older, historical:
415434
[eda-prompt-research.md](../plans/eda-prompt-research.md),
416435
[eda-codegen-run1.md](../plans/eda-codegen-run1.md),
417436
[haiku-codegen-findings.md](../plans/haiku-codegen-findings.md),

docs/mcp-server.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,13 @@ The compile-and-persist family. The `code` argument is a self-contained Python
143143
script that binds a top-level `expr` to a xorq/ibis expression;
144144
`catalog_run`'s docstring is the canonical cookbook for writing it (namespaces,
145145
the datafusion-only backend, data sourcing, and the `xorq.ml` MODELING section).
146+
Data sourcing inside `code` uses four helpers from `tallyman_xorq.io`:
147+
`tracked_expr_from_alias` (a catalog entry, recorded as a lineage parent),
148+
`pinned_expr_from_alias` (a catalog entry by alias or hash, no following),
149+
`read_project_file` (a raw parquet under `data/`), and `tallyman_read_csv` (CSV
150+
ingest — injects an `original_row_order` column so the baked snapshot is
151+
byte-stable across builds; use it for all CSV reads instead of
152+
`xo.deferred_read_csv`, #137).
146153

147154
### `catalog_run(code, prompt="") -> dict`
148155
Execute an expression and persist it as an **unnamed (scratch)** entry. Claude's

0 commit comments

Comments
 (0)