Skip to content

reconcile: the card reads agreement, says could not compare, names the failed side (ACE-134) - #320

Open
sandeep-agami wants to merge 1 commit into
ACE-133-fan-trap-written-edgefrom
ACE-134-reconcile-card-agreement
Open

reconcile: the card reads agreement, says could not compare, names the failed side (ACE-134)#320
sandeep-agami wants to merge 1 commit into
ACE-133-fan-trap-written-edgefrom
ACE-134-reconcile-card-agreement

Conversation

@sandeep-agami

Copy link
Copy Markdown
Collaborator

Spec: ACE-134 (agami-sdlc projects/agami-core-extension/specs/reconcile-evidence/ACE-134-reconcile-card-agreement.md)

Stacked on #319 (ACE-133); reads the score fields and the outputs claim from #316 (ACE-131) and falls back without them. Round 9 of the reconcile stack, from the 51-row run.

The findings

  1. The values row over-reported. One differing cell made the card say "0% of the values match" with a missing-column sentence for a column that was there.
  2. Every error row said "agami's query failed" in the label, the sentence and the change text, although error means the row could not be scored: 18 error rows, agami's statement had run fine on 11. Phase 3 may rewrite sentence and change, never result.label.
  3. "Can the reconcile skill determine if the query is equivalent based on the join and filters?" The query fact was dropped whenever the data could not be compared: ("could_not_compare", *) always read "could not run".
  4. Phase 2e handed the statement as --golden-sql-file, so row order decided the comparison (ACE-131's finding A).

The change (plugins/agami/scripts/reconcile.py, the report template, SKILL.md 2e, part-ledger.md)

  • Values row: accuracy 1.0 reads "identical"; with pairs and a paired_row_share, 1.0 reads "identical on the N paired columns" and below 1.0 reads "K of N rows match" with the note "differs in total on 1 of 10 rows" per pair whose column_agreement is below 1. An older score file keeps today's branches. _values_agree_on_shared_columns reads the share, so "partly" needs the paired columns to agree on every row.
  • _error_cause, from the files: yours_failed (the runs or scope part not confirmed), agami_failed (no statement, or nothing recorded with an error), nothing_to_compare (the score unscored), no_ground_truth (a question-only row), else unknown. The label is "could not compare"; the sentence "This row could not be compared: : "; the fix ask_again only for agami_failed and unknown ("run this row again"), query or semantic_model for yours_failed, none with its own change text for the other two; the owner follows.
  • The outputs claim reads "selects" and is definitional. Labels for a row whose data could not be compared: "same query, answer not compared" (amber), "different query, answer not compared" (red), "could not compare" (gray). The keep gate is untouched: status == match from the run.
  • The near-miss sentence ("matches no rows; the data spells it …") only on a defect (ACE-130's other half).
  • SKILL.md 2e: --unordered replaces --golden-sql-file; the eight claims named. part-ledger.md: which claims are parts and which the page reads; ordering is the one place row order is judged.

Pins moved on purpose

"identical, row for row""identical" (two); the error sentence "agami's query failed: boom.""This row could not be compared: agami's query failed: boom."; the label "could not run""could not compare"; the 2e pin --golden-sql-file--unordered. The plan's note about the end-to-end test's compare helper does not apply: it diffs scalars.

Tests

New: one differing cell reads "9 of 10 rows match" with the differing column named; paired columns agreeing on every row beside an unpaired one read "partly", and not when they disagree too; an older score without the share keeps the grace; each of the six error causes yields its fix, owner, sentence and change text; an error row whose claims all agree reads "same query, answer not compared" and a differing outputs alone reads "different query, answer not compared" with a "selects" row, and is never keepable; a different projection alone is a different query. 259 passed across the reconcile suites, the promotion-skill pins and the end-to-end test.

The gallery artifact is republished after the remaining page changes in this round (ACE-135, ACE-136).

🤖 Generated with Claude Code

…e failed side; 2e unordered

One differing cell printed "0% of the values match" for a column that was there; the values row now
reads the comparator's share ("9 of 10 rows match", "differs in total on 1 of 10 rows"), and "same
rows, different columns" needs the paired columns to agree on every row. Every error row said
"agami's query failed" although the cause was often the person's statement, two empty results or a
bare question; the card reads the cause from the row's files, offers agami again only when agami
failed, and sends a question-only row to the grading page. The result pill reads "could not
compare", or "same query, answer not compared" / "different query, answer not compared" when the
claims could be read, the `outputs` claim ("selects") included, so query equivalence stands on its
own when the data cannot be compared. The near-miss sentence shows only on a defect. Phase 2e passes
`--unordered`: a different sort is a different query, never a different answer.

Pins moved on purpose: "identical, row for row" reads "identical"; the error sentence carries its
cause; "could not run" reads "could not compare"; the 2e pin names --unordered.

Spec: ACE-134

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant