Skip to content

A comparison edge can carry its own date - #439

Open
ccerv1 wants to merge 2 commits into
mainfrom
carl/comparison-edge-attestation
Open

A comparison edge can carry its own date#439
ccerv1 wants to merge 2 commits into
mainfrom
carl/comparison-edge-attestation

Conversation

@ccerv1

@ccerv1 ccerv1 commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

check_capability binds a dependent's whole-axis capability.last_verified to its root's
whole-axis capability.last_verified. Those are two claims about two different products, and the
coupling has a consequence #436 measured: as the corpus grows, every new product's natural root
was confirmed before the product existed, so a tranche can compare its members to each other and
to nothing else. It happened three times in one week — #431, #432 and #433 each dropped
arithmetically correct comparisons into prose rather than assert a re-derivation nobody performed.
181 of 553 products sit in the comparison graph today; 305 have no edge in or out.

The mechanism

An optional capability.comparison block dates the edge rather than either product, and
carries the read that dated it:

capability:
  relative_to: verl
  relation: one_below
  comparison:
    last_attested: 2026-08-31
    sources:
      - url: https://github.com/volcengine/verl
        shows: "the algorithm catalogue, still wider than this product's"
        accessed: 2026-08-31
        http_status: 200
        content_sha256: <64 hex>

Its own source list, not capability.sources, because these are citations about somebody else's
product: folding them in would let a root's page count as this product's own evidence and would
make the weak-root check read the wrong thing. The shape is #/definitions/source verbatim, so
build/fetch_source.py writes it unchanged and build/check_refetch.py re-checks it once
load_sources walks the extra list — which it now does.

Gate semantics

state behavior
no comparison block unchanged. The root's capability.last_verified must exist and be >= the dependent's.
comparison present, dependent's last_verified > last_attested FAIL. relative_to/relation are part of the axis's score, so a whole-axis confirmation cannot outrun one of its parts.
comparison present, last_attested > every attestation source's accessed FAIL. The date rests on nothing. Mirrors check_verification.invariant.
a covering source (accessed >= last_attested) lacking http_status or content_sha256 FAIL. A claimed confirmation needs a fetch to point at. Mirrors check_verification.digests.
comparison present, last_attested < the root's capability.last_verified REPORT, not fail. The root was re-read since the spacing was judged. The arithmetic check fires if the root's score moved; this is the case where its value moved and its score did not.
comparison present and consistent, root's axis date older than the dependent's PASS. This is the unlock.

check_capability's summary gains two lines: how many recorded comparisons are attested versus
resting on their root's axis date, and the re-judge queue from row five.

Why hash-only reaffirmation was rejected

Option B in the issue — let a comparison stand when the root's cited sources still return the same
content_sha256 — fails on two counts.

It is unsound. A spacing can go false with no byte changing anywhere, because the falsifier is a
third product that neither side cites. And it does not reach the roots that matter. Re-fetching
every digest-carrying capability source on all 54 comparison roots: 22 of 84 sources reproduce,
12 of 54 roots have every capability source reproduce, and of the 13 roots the three dropped
tranches needed, zero reproduce. The reason is structural — 26 of the 95 sources on those roots
are GitHub landing pages carrying a star count and 13 more are READMEs on active repos, while the
stable ones are arXiv abstracts and dataset cards. So the cheap path exists and does not go where
the expansion needs to go.

An attestation therefore costs a real read of the root. It is cheaper than a full anchor refresh in
what it claims, not in what it costs. docs/reference/evidence-and-freshness.md now states the
may-prove / may-never-prove rule so it does not drift back.

Migration

Mechanism only, no backfill. All 181 existing edges carry no comparison block and stay on the old
rule, which they satisfy today. Backfilling would mean attesting 181 spacings, which is the work
this issue exists to make affordable rather than a precondition for it.

Anchor-first stays mandatory in refresh-category — a refresh pass is where a re-read that moves a
score belongs, and it is the read that catches a root whose surface changed under the bands resting
on it (openhands). Attestation is what add-product uses so a batch is not blocked between
refresh cycles.

Deliberately out of v1

  • No age ceiling on last_attested. The 30-day freshness window is about axes; adding an edge
    dimension to it is its own decision, and the report line surfaces the same information without a
    new failure mode.
  • No cycle detection. check_capability would accept a reciprocal at pair today, which is
    arithmetically consistent and semantically circular. Separate issue.
  • Not exposed in the payload. details_payload's AXIS_KEYS trims comparison out, and it
    stays trimmed. Surfacing "spacing last judged on" in the product panel is a front-end question,
    and there is no payload size gate forcing the call now.

The attestation date rides inside axis_assessments' existing basis_detail string
(relative_to=X;relation=Y;attested=DATE) rather than a new column, because a column add on a
deployed static model fails on the platform.

Scores

No score changes. This PR adds a mechanism and writes no comparison edges, so no product's
score, no category's L, and no stage moves. build/notebook_data.json regenerates byte-identical.

Validation

uv run python -m build.validate            0 error(s)
uv run python -m build.check_capability    [OK]  181 recorded, 0 attested
uv run python -m build.check_verification  invariant / digests / producible-pairs / placeholder-shows all [OK]
uv run python -m build.check_recipe        0 failure(s)
uv run python -m build.check_components    [OK]
uv run python -m build.serialize           no diff against the committed payload
uv run python -m pytest tests/             1272 passed

Closes #436

ccerv1 added 2 commits August 31, 2026 21:44
A capability band placed against a peer is dated today by the peer's whole-axis
last_verified. Those are two claims about two different products, and as the corpus
grows every new product's natural peer was confirmed before the product existed, so
the comparison graph stops growing at the periphery.

capability.comparison dates the edge itself and carries the read of the peer that
dated it. An attested edge is freed from the peer's axis date and pays for it with
its own evidence requirement: a source read on or after last_attested, carrying
http_status and content_sha256. Where the peer has been re-read since the spacing
was judged, the edge is reported for re-judgment rather than failed.

An edge with no attestation is unchanged, so the 181 recorded today keep the old
rule and nothing is retroactively legitimized. check_refetch now walks the new
source list, and the attestation date rides in axis_assessments' existing
basis_detail string rather than a new column, which a deployed static model cannot
take.
evidence-and-freshness.md gains the normative split between the date on a product's
own capability evidence and the date on a spacing, plus the rule on what a digest
match may prove and what it may never prove. capability.md documents the field and
the gate. add-product.md gains the attest-the-spine step and lists check_capability,
which CI has always run and the workflow never named. refresh-category.md keeps
anchor-first mandatory and points at the alternative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[methodology] A structured comparison needs a fresh root, so the relative_to graph cannot grow as the corpus does

1 participant