Skip to content

HANDOFF-2026-06-30.md

Latest commit

 

History

History
592 lines (529 loc) · 38.1 KB

HANDOFF-2026-06-30.md

File metadata and controls

592 lines (529 loc) · 38.1 KB
Raw

Session Handoff — 2026-06-30 (last updated 2026-07-03)

Detailed handoff spanning several sessions. Newest work is in §10 (Haiku A/B + fleets P51–P52) — read that plus §9 (gate recovery, prompt/model, §9e pipeline) and §7 (resume) first; §8 is the caption/NDCC-JSON session, §5 the older verification state, §§1–4 the earlier star-page / P39–P47 / marker-coverage history. As of 2026-07-03 (later) the working tree is DIRTY at close, 10 commits ahead of origin/main (UNPUSHED), and no release has been cut — the DB corrections are local only, not yet on the live server (server ships from GitHub release zips). Standing rule: commit/push only when asked.

Authoritative-text bar applies to everything here (see MEMORY.md → project_authoritative_text_goal). All DB writes are logged to changelog and revertible per batch with python -m ndcourts_mcp.cleanup revert <batch>.

1. The headline change — star pages bracketed, emphasis preserved

Decision (resolves TODO-validation §1 "markup cohort"): an asterisk in the corpus was overloaded between two different things —

  • structural: reporter star-page markers (*1079 = "N.W. page 1079 begins here")
  • semantic: markdown emphasis (*State v. Smith*), which carries authorial meaning and must be preserved for fidelity.

Rather than strip emphasis (the original framing), we bracket the structural markers → [*1079], mirroring the existing [¶N] paragraph-marker convention. Now a bare *…* means emphasis; [*NNN] means a page break. All 106,029 emphasis pairs were left untouched.

Detection (precision-first — never bracket a citation/year/section number)

Implemented in ndcourts_mcp/star_page_reformat.py:

  • candidate = escaped \*NNN (anywhere) OR bare *NNN preceded by whitespace (emphasis-close markers are glued to the preceding text — a letter/comma — not whitespace, so they're excluded), and NOT closed by */[.,;:]* (which would be emphasis-on-a-number *1841.*).
  • accept = candidate on a run of ≥2 doc-ordered values stepping by ≤2 (_GAP). Real pages march consecutively; citation pincites/years/section numbers are isolated or jump. Applied to a fixpoint so a real run split by an interleaved citation is still recovered. Idempotent (bracketed markers aren't re-detected).
  • A separate singleton-rescue (rescue_singletons) brackets lone *NNN that the run-rule can't validate, by checking the value falls within the opinion's own N.W. page span (start page from the parallel cite … start + est. page count), with a <Reporter>. <page> citation-tail guard.

Applied

Batch Effect
star-page-reformat-2026-06-30 53,972 markers → [*NNN] across 10,561 opinions
star-page-singletons-2026-06-30 1,003 isolated *NNN bracketed across 980 opinions

608 off-track candidates remain bare and are correct non-star-pages (citation pincites *81 Pa., abuttals *3814, omission * * *23). Off-track audit CSV: triage/star-page-offtrack-2026-06-30.csv.

Downstream code updated in lockstep (5 parsers — these read [*NNN] now)

proofread.py (_STAR_PAGE, _BODY_BOUNDARY, star_page_before), merge_westlaw_text._extract_star_pages, ingest_westlaw (star_strip), strip_westlaw_headnotes (_STAR/_WH), memo.py. The Westlaw-incoming ones accept BOTH *NNN and [*NNN].

Wiring (so new opinions stay consistent)

  • ingest --incremental now runs, after ingest, BOTH the page-footer cleanup AND the star-page reformat over the scraped year (idempotent, year-scoped).
  • Every Westlaw body-text write path (receive_westlaw, merge_westlaw_text) calls bracket_star_pages() at store time. (ingest_westlaw writes metadata only, so it's untouched.)

New files

  • ndcourts_mcp/star_page_reformat.py — reformat_corpus, reformat_opinion, rescue_singletons, bracket_star_pages
  • ndcourts_mcp/page_footer_fix.py — clean_page_footer_leaks (refactored out of scripts/fix_2026_page_footers.py, which is now a thin wrapper)
  • scripts/reformat_star_pages.py — CLI (--singletons for the rescue pass)
  • tests/test_star_page_reformat.py (13 cases), tests/test_page_footer_fix.py (9)

Backups: opinions.db.bak-star-page-reformat-2026-06-30, opinions.db.bak-starpage-singletons-2026-06-30.

2. Proofing fleet — P39–P45 (date-DESC)

Coverage is now contiguous 2013 ND 203 → 2026 (P1–P45).

Batch Range Corrections Heading queue
P39 2016 ND 254→122 951 fully worked
P41 2016 ND 4→2015 ND 199 936 fully worked
P40 2016 ND 121→5 1,141 fully worked (2026-07-01)
P42 2015 ND 194→78 818+194 auto + 12 manual + 2 caption fully worked (2026-07-02)
P43 2015 ND 77→2014 ND 189 647+194 auto + 4 manual + 4 caption fully worked (2026-07-02)
P44 2014 ND 188→77 586+194 auto + 16 manual + 1 caption fully worked (2026-07-02)
P45 2014 ND 76→2013 ND 203 612+194 auto + 3 manual + 3 caption fully worked (2026-07-02)

P42–P45 all needed a retry pass first — running the 4 fleets concurrently hit heavy API rate-limiting (328/480 agents failed, not content issues). Fixed by re-running each fleet's specific failed opinions via gen_proofing_workflow.py --ids <comma-list>, one fleet at a time — all 328 retries succeeded cleanly. Lesson: don't run more than 1–2 fleets concurrently. Full detail (incl. the id16274 anchor-construction bug caught before applying, and the id16360/2014 ND 197 large-opinion repair) in CHANGELOG-data.md.

✅ P46 + P47 DONE 2026-07-02 (385+536 autosafe, 4 heading gap-fills, 1 caption; retries all succeeded — NB derive retry lists from the workflow INPUT id set, one P47 agent died without a failure log line). Manual queues + statute-digit cohort ✅ DONE 2026-07-02 (98 opinions of heading/ellipsis defers applied; corpus-wide (l)→(1) sweep, 1,781 fixes / 838 opinions, statutes.db-as-oracle + convention + fleet PDF flags — see CHANGELOG). Remaining from P46/P47: ~470 flags (caption cohort 46, markup contamination, etc.). Pending: re-run cite_extract over the 838 statute-digit opinions. Next fleet: P48, OFFSET 3550, starting after 2012 ND 234.

Per-batch detail in CHANGELOG-data.md. Rule 13 (star-page/asterisk handling) was added to the fleet prompt (scripts/gen_proofing_workflow.py + triage/corpus-proofing-prompt-v2.md) and is now baked into every generated workflow — agents flag bare star-pages for bracketing and never touch [*NNN].

The proofing pipeline (how to run a batch)

# 1. compute next slice (see §4), generate workflow:
.venv/bin/python scripts/gen_proofing_workflow.py OFFSET 120 triage/corpus-proofing-pNN.wf.js
# 2. run it via the Workflow tool: Workflow({scriptPath: ".../corpus-proofing-pNN.wf.js"})
#    (120 read-only Explore agents; ~40 min; returns a 120-item array under result.result)
# 3. extract the array to triage/proofing-pNN-results.json
# 4. verify -> consolidate -> apply:
.venv/bin/python scripts/verify_proofing_proposals.py triage/proofing-pNN-results.json
.venv/bin/python scripts/consolidate_proofing_trio.py pNN
.venv/bin/python scripts/apply_proofing_proposals.py triage/proofing-trio-apply.json --apply --batch corpus-proofing-pNN-2026-XX-XX
.venv/bin/python scripts/apply_proofing_proposals.py triage/proofing-trio-caption.json --apply --batch corpus-proofing-pNN-caption-...   # after eyeballing

Heading-queue sub-procedure (the careful part — see §3)

After the autosafe apply: the proofing-trio-heading.json bucket (paragraph_seq) is mostly garbled [¶N] marker repairs — verify each is a sequence gap-fill (target number missing, both neighbours present) then apply. Then run scripts/heading_move.py --ids ... --out triage/heading-moves-pNN.json for the merged/fused headings it can auto-construct, and apply. The remaining heading_seq flags are manual MOVE/reconstruction work (§3).

3. ✅ DONE (2026-07-01) — P40 heading queue

Resolved as corpus-proofing-p40-heading-2026-07-01 (24 edits / 14 opinions, triage/heading-manual-p40.json) — see CHANGELOG-data.md. The two held-back marker insertions (id16678 [¶9], id16653 [¶8]) were checked directly against the DB and confirmed NOT to duplicate any existing [¶N] (the earlier "~70 chars later" duplication worry did not hold up under inspection) — applied as originally proposed.

id16651 turned out to be a much bigger job than a heading fix — see below.

id16651 (2016 ND 19) — full reconstruction, ✅ DONE (2026-07-01)

This one was NOT a simple Ill→III fix. The opinion had an interleaved second, partial OCR pass: duplicate/mislabeled ¶3 and ¶7 sitting between real ¶2 and ¶4, a duplicate OCR-noised ¶13 mislabeled [¶18], the III heading stranded with no correctly-placed copy, and the entire ¶29 signature/concurrence block missing. Reconstructed by diffing the full DB text against the authoritative slip PDF paragraph-by-paragraph (all 30 ¶s, all 6 headings verified 1:1). Resolved as corpus-proofing-16651-reconstruct-2026-07-01 (49 fixes, triage/reconstruct-16651.json) — see CHANGELOG-data.md for the full defect inventory. footnote_structure() re-verified to still resolve footnote 1 after the reconstruction.

Judgment calls made that a reviewer may want to revisit (all applied as "leave as-is", consistent with how this corpus already treats them elsewhere, and consistent with citation-formatting being explicitly out of scope for the proofing fleet's own coverage notes throughout P39–P41):

  • (N.D.YYYY) no-space vs the slip PDF's (N.D. YYYY) — West-reporter-style convention, prevalent corpus-wide (421 no-space vs 157 spaced in this era alone).
  • Compact ... vs the slip PDF's spaced . . . mid-quote ellipses.
  • Hyphen vs the slip PDF's en-dash in compound terms (Code-Leases vs Code–Leases).
  • Minor citation-internal microspacing (ed.2010) vs ed. 2010), [U.C.C.§ vs [U.C.C. §) — scoped to a dedicated citation-rejoin pass elsewhere in this project.

id16360 (2014 ND 197, MKB Mgmt. Corp. v. Burdick) — reconstruction + whitespace normalizer, ✅ DONE (2026-07-02)

Heavily cited (the H.B. 1297 medication-abortion case), so this one got the same depth of treatment as id16651, done in three passes as the user reviewed the result each time:

  1. 11 paragraph-marker relocations (corpus-proofing-16360-manual-2026-07-02) from the P44 heading queue — a recurring "marker displaced one line/segment" defect, incl. a two-part ¶93 fix and a ¶170/¶171/¶172 reassembly (a whole sentence — "I would reverse the district court." — had been spliced into the wrong paragraph, leaving ¶171's marker orphaned with no text). Applied 2 more fixes than consolidate's automated bucket surfaced after independently confirming both against the live DB + PDF.
  2. User asked to open the file in an editor (open -a NvimGhostty <file> — there's a dedicated NvimGhostty.app on this machine) to review directly. That surfaced 92 PDF page-footer-number leaks in the running text (corpus-proofing-16360-pagefooter-2026-07-02) — e.g. "20↵↵2↵↵percent" where a page-footer split "20 percent" — plus one genuine content-transposition inside the quoted Appendix complaint. Visually confirmed via 200dpi PDF page renders (not just pdftotext) before removing anything. Known unfixed gap: this opinion's Appendix footnotes use a bare N text-on-one-line format proofread.py's footnote_structure() doesn't parse (returns 0 bodies) — pre-existing, unrelated, confirmed untouched by the leak removal.
  3. User asked for a general whitespace cleanup, having noticed the body text still carried the PDF's own line-wrap breaks as literal blank lines mid- sentence. New module ndcourts_mcp/paragraph_whitespace.py (normalize() / normalize_opinions(), CLI scripts/normalize_paragraph_whitespace.py, 18 tests) establishes the convention: exactly one blank line before every [¶N]/heading, paragraph/quote bodies unwrapped to flowing prose, blank-line-separated list items ((a), 1.) preserved. First pass had a real bug the user caught by reading the output: the list-item classifier was too permissive and mistook case-citation v., the abbreviation art., and multi-digit pinpoint page numbers (129.) for list openers, splitting citations at paragraphs 25, 28 (×2), and 32. Fixed with an explicit exclusion list + capping bare openers at 1 char + rejecting a bare-digit opener right after a citation-pinpoint signal like "...Johnson, at". Re- verified all 4 spots plus a full re-audit of all 255 remaining internal splits in the document for the same bug class — none found.

All three passes verified content-preserving via the same paragraph-level PDF diff used for id16651 (0 new discrepancies each time, all 185 paragraphs sequential). The normalizer is applied to id16360 only — not yet run against a broader corpus sample, so treat it as available-but-unproven at scale. See CHANGELOG-data.md for the full batch-by-batch detail (3 batches: ...-manual-, ...-pagefooter-, ...-wsnorm-/...-wsnorm-fix-2026-07-02).

4. ⏳ PENDING — other queues

  • Next fleet batch: P46, OFFSET 3310, starting 2013 ND 202 (see §2/§7).
  • Malformed *Case**,* double-asterisk emphasis (surfaced by rule 13, e.g. *Throlson**,*, *Dufner**,*): an emphasis-markup cleanup (NOT star-page; keep the emphasis, fix the doubled *). Scope corpus-wide before fixing.
  • P40–P45 general flag backlog (caption contamination, ocr_digit-in-citation, etc. beyond headings/autosafe/captions already applied). Lower priority.
  • Star-page residue that stays bare is intentional (citation pincites etc.); the fleet now flags any genuine bare star-page it sees (rule 13).
  • paragraph_whitespace normalizer validation. ✅ PARTLY RESOLVED 2026-07-02: a corpus dry-run found (and fixed) a content-DELETION bug — the first version dropped all body text between a bare heading and the next [¶N] marker (markerless-body opinions; id19495 lost ~10K chars in dry-run). Heading-trailing text is now preserved verbatim (not unwrapped — paragraph boundaries are unknowable without markers), and content preservation is enforced by a new shared gate (ndcourts_mcp/textfix.py, assert_whitespace_only_change: whitespace-stripped char sequences must be identical; a gate trip skips the opinion, never writes). Gate-checked dry-run over all 7,217 modern opinions: 0 trips; only 15 opinions carry ≥500 chars of the real line-wrap-blank-line defect. ✅ The 15-opinion targeted pass is DONE (ws-join-pdfverified-2026-07-02, 3,291 joins) — but NOT via blanket normalize(): junction-selective, each join verified against the court PDF (see CHANGELOG-data.md; script triage/ws_join_pdfverified_2026-07-02.py). Key learnings baked into that script: (a) id12692/id13049 prove blanket normalize() is unsafe — most of their blank lines are REAL boundaries; (b) headings with trailing periods (I.) are invisible to the structural tokenizer and the PDF centers them with no blank line → PDF-flow false positive, guarded; (c) colon-introduced block-quote blank lines kept (174) as the quote-boundary signal for the planned blockquote pass. Residue: ~800 NOMATCH + 604 FLAG junctions left as-is (correct), and normalize() remains applied only to id16360.
  • 2 harmless residual skips from the P42/P43 autosafe apply: id16461 and id16403 each have one stale whitespace anchor that no longer matches post-apply (safe no-op, not investigated further).
  • NEW (2026-07-02): period-suffixed headings (I. vs bare I). Section headings with a trailing period are invisible to paragraph_whitespace's structural tokenizer (_HEADING_PAT matches bare Romans/capitals only), to the fleet's heading checks, and the court PDF centers them with no blank line (a PDF-flow false positive — see the HEAD guard in triage/ws_join_pdfverified_2026-07-02.py, 12 kept in the 15-opinion cohort, e.g. id17255). Scope corpus-wide, then decide the convention (likely normalize to the bare form the tokenizer/fleet already recognize, or teach every heading consumer the \.? variant) — fold into the blockquote/heading markup work.
  • Paragraph-marker coverage gaps — ✅ MOSTLY REPAIRED 2026-07-02 (was: 48 modern opinions <60% marker coverage). Done: id19495/id19480 fully restored (81 markers, print-verified page-by-page — the PDFs' marker glyphs have no text-layer mapping, a defect class CL shares) + 21 tens-digit 3→8 marker misreads fixed across 20 opinions (batches marker-restore-2026-07-02, marker-digit-fix-2026-07-02; scripts + triage CSV in triage/). See CHANGELOG-data.md. THE CLASS IS NOW CLOSED (2026-07-02): dropped markers ✅ (marker-dropped-fix), version-divergence flags ✅ resolved as false alarms (egeland-ocr), and the wholly-markerless queue ✅ fully adjudicated (marker-bracket: 2 West-style ¶ N bracketings, 3 garble repairs, 2 glyph-drop insertions, 1 digit-flip + signature restore; 13 accepted residue = 12 genuinely-unnumbered 1997 orders + id18915 pending its stored-twice dedup). New invariant nd_marker_coverage guards the class (passes at 0); nd_modern_paragraph_markers baseline lowered 18→13. Invariants now 25 ok / 2 known / 0 regressed. All small follow-ups worked 2026-07-02 (small-fixes-2026-07-02): id18915 deduped + markers (baseline 13→12), id19495 ¶51 scramble reconstructed, id14118/id17070 spelling+heading fixes, 3 stubs adjudicated as legitimate COA table dispositions (captions normalized, Ernst's impossible date fixed 2003→2005). Only deliberate residue: id18692's handwritten-signature OCR junk (sole textual witness) and the 12 genuinely-unnumbered 1997 orders. v1.1.2 released 2026-07-02 (this class shipped); fleet P46 launched 2026-07-02 (OFFSET 3310, 2013 ND 202→105) — process its queues per §2.

5. Verification state (END OF 2026-07-02 LATER SESSION — v1.1.3)

  • Invariants (opinions): 25 ok, 2 at known baseline, 0 regressed (known: 12 genuinely-unnumbered 1997 short orders; restored-corrections). Re-confirmed as the v1.1.3 release gate.
  • Tests: 126 pass (uv run pytest -q; +6 in tests/test_ndcc_html.py pinning the NDCC html→subsection parser + pincite normalization).
  • Footnote-pincite detector: 176 PRESENT / 0 MISSING.
  • statutes.db: 29,107 provisions, JSON-canonical, quick_check ok; html on 29,104; 87,637 subsection rows. FTS integrity ok.
  • v1.1.3 released + pushed 2026-07-02 (5 DB assets: opinions, constitution, rules, statutes, admincode). Live server auto-self-updates the next morning (deploy/self-update.sh) — no manual deploy needed.
  • Working tree CLEAN — commits 9aada1e (opinions), c1ce0e9 (statutes JSON re-base), 2006b81 (release v1.1.3).
  • Full-corpus cite_extract re-extract still owed under jetcite 2.5.3 (the 838 re-run this session only sampled the gain; whole table predates 2.5.3).

6. Gotchas / non-obvious context for the next session

  • Proofing file naming: verify_proofing_proposals.py RESULTS.json writes RESULTS.{auto,review,reject}.json; consolidate_proofing_trio.py pNN reads triage/proofing-pNN-results.{auto,review}.json and writes the fixed-name triage/proofing-trio-{apply,caption,heading,defer}.json — these are OVERWRITTEN every consolidate run, so process one batch fully before the next.
  • proofing-trio-heading-apply.json / -defer.json are STALE (from a pre-P39 batch). consolidate does NOT regenerate them; heading_move.py writes its own triage/heading-moves-pNN.json. Don't trust the -apply/-defer heading files.
  • heading_move.py only constructs MOVES for merged/fused/dropped headings (heading present elsewhere, fused to a line/marker). It does NOT handle garbled-OCR Romans, mid-paragraph relocations, or fully-missing headings — those are manual.
  • Anchor breakage: apply the main autosafe bucket BEFORE the heading bucket, and re-verify heading anchors (text.count(old_exact)==1) — the autosafe edits can shift a heading proposal's byte-exact anchor to count 0 (it then skips).
  • apply_proofing_proposals.py is atomic per opinion — if one proposal for an opinion fails its anchor, ALL of that opinion's proposals skip.
  • Agents confabulate [¶ N]→[¶N] despacing even when the DB is already tight [¶N]; those land in the heading bucket as anchor-count-0 no-ops (harmless skip).
  • Star-pages are reporter (N.W.) pagination, not in the court PDF — the modern court PDF is paragraph-numbered. To validate a star-page, use the opinion's N.W. parallel-cite page range (what rescue_singletons does), not the PDF.
  • DB freshness / weekly scraper: the live MCP DB is current to 2026 ND 129 (filed 2026-06-25). Scraper runs Fri 11:00/13:00/15:00 via launchd (~/code/scraper/tools/weekly_scrape.sh); it needs the court VPN up. Stage 2 runs ingest --incremental → merge_nd_metadata → cite_extract → quality_scan → invariants. The page-footer + star-page cleanups now run inside ingest --incremental.
  • A verified anchor (text.count(old_exact)==1) is not the same as a verified RESULT. Two real bugs this session slipped past the anchor check and were only caught by re-reading the actual output: id16274's "move heading VI" proposal had a source_quote correctly describing the fix but old_exact/new_exact that didn't implement it (§2); the whitespace normalizer's first pass had a classifier that mistook citation fragments (v., art., pinpoint page numbers) for list items (§3). Always eyeball a sample of the actual before/after text, not just the dry-run's anchor count.
  • Opening a file in an editor for the user is a real, useful review step — both of id16360's post-reconstruction bugs were caught this way, not by automated diffing alone. open -a NvimGhostty <file> opens nvim in a new Ghostty window (there's a dedicated app for this on this machine); write the current DB text_content to a scratch .md file first since the on-disk source markdown under ~/refs/nd/opin/markdown/ is NOT kept in sync with DB corrections.
  • Don't commit unless asked. Working tree is large now (git status) — modified: CHANGELOG-data.md, TODO-validation.md, TOOLS.md, 6 parser/ ingest modules, scripts/{fix_2026_page_footers,gen_proofing_workflow}.py, tests/test_footnote_pinpoints.py, the fleet prompt + P42-caption-etc. trio files. New: ndcourts_mcp/{star_page_reformat,page_footer_fix,paragraph_whitespace}.py, scripts/{reformat_star_pages,normalize_paragraph_whitespace}.py, tests/test_{star_page_reformat,page_footer_fix,paragraph_whitespace}.py, this handoff doc, and a large number of triage/*.json/*.wf.js audit artifacts from every fleet round (commit or .gitignore per existing convention — these are working files, not something to prune blind).

7. One-line resume (END OF 2026-07-03 — see §9 for full detail)

Working tree CLEAN; 9 commits ahead of origin/main, UNPUSHED; NO release cut (this session's ~10K DB corrections are LOCAL only — not on the live server). Corpus 19,807, invariants 25 ok / 2 known / 0 regressed, 146 tests pass. Done this session (§9): fixed a verify-gate bug that had been silently rejecting ~27% of every proofing fleet's proposals on quote/dash/marker mismatches → recovered ~1,600 falsely-rejected fixes across P1–P47; hardened the gate (canon normalization + edit-region verification + guards G0 style-only / G0b footnote-call / G0c emphasis, 20 tests); refined the fleet prompt (ellipsis-widen caution + blank-line descope); pinned the fleet model (Sonnet 5 default, Opus 4.8 via --flagged); and fully worked fleets P48, P49, P50 (~2,300 fixes, all manual queues cleared, 0 residual holds). Adopted inline reconstruction verification (memory [[feedback_verify_reconstructions_inline]]) — P50 needed zero hold round-trip. Next: fleet P51 (OFFSET 3910, after 2011 ND 102), one at a time; process per §9 (verify → consolidate → apply autosafe with high-stakes PDF checks → inline-verify manual queues → retry dropped ids from the workflow INPUT set). Standing queues: the 62-item markup HOLD for the rule-11 markup pass (triage/proofing-p1p41-HOLD-for-review.json), the flag-review backlog (malformed *Case**,* emphasis + case-name/statute-pinpoint divergences the fleets flag, not fix), full-corpus cite_extract re-extract under jetcite 2.5.3, and a possible release cut to push all of this to the live server.

8. 2026-07-02 (later session) — caption cleanups + NDCC statutes JSON re-base

Opinions (committed 9aada1e):

  • cite_extract re-run over the 838 statute-subsec-digit opinions. Finding: jetcite normalizes N.D.C.C. cites to the base section (0/37,987 carry a subsection), so the (l)→(1) fix is invisible to the citation graph — the queued rationale was moot. Real value came from the jetcite 2.5.3 bump: 44 new cites. Did a clean delete+rescan of the 838 + global --cited-by-only rebuild. A full-corpus re-extract under 2.5.3 is still owed (whole text_citations predates it).
  • Smushed caption headers — 618 (not ~40). The # HEADER line == CL case_name_full, whose party role-labels were stripped leaving dangling , and around v./at the end. Tier A surgical strip (mechanical junk only; full party names kept), header + case_name_full in lockstep. Guarded: lowercase v.+space so uppercase party initials (and V.M.) are never hit.
  • Cross-appeal captions — 23 (Tier B). The ~3,290 role-designation captions (Plaintiff and Appellee, v. …) are authentic published caption text (PDF-confirmed) — LEFT INTACT. Only 23 were genuinely mangled (CL stripped the primary roles leaving an orphan and Cross-Appellee); reconstructed each from its court slip-opinion PDF (full role block restored).

Statutes (committed c1ce0e9, shipped v1.1.3): the whole [[project_ndcc_json_canonical]] re-base — audited statutes.db against the structured century_code.json and found the pdftotext mirror had a catchline-wrap artifact in ~15% of sections (truncated heading + leaked catchline fragment into body). Fixed in three steps: (1) catchline-split repair (4,227 sections); (2) new html/subsection-pinpoint layer (87,637 rows; lookup_authority now resolves § X(1)(a) + a numbered/lettered oracle); (3) JSON-canonical ingest (ingest_statutes --json) + full rebuild — validated content-equivalent to the authoritative source, reproduced all prior fixes + restored 37 residual PDF-drops. Predecessor at statutes.db.bak-pre-jsonrebuild-2026-07-02. Full detail in CHANGELOG-data-primarylaw.md.

9. 2026-07-02/03 — verify-gate recovery, gate/prompt hardening, fleets P48–P50

9a. The headline: the verify gate was silently discarding correct fixes

scripts/verify_proofing_proposals.py (the Layer-C gate) compared DB text against raw pdftotext -layout output, but the DB stores straight ASCII quotes/ apostrophes (policy), em/en dashes, and bracketed [¶N]/[*NNN] markers while pdftotext emits curly quotes, plain hyphens, and no markers. Correct OCR / hyphenation-join / statute-digit fixes were therefore rejected on characters the fix never touched. Across P42–P47, 88% of rejects fell in the two affected buckets; this had been happening since P1 (~27% of every fleet's proposals lost).

Gate fix (all in verify_proofing_proposals.py, verification-only — never alters applied text):

  • canon() — folds curly↔straight quotes, dash variants→hyphen, strips [¶N]/[*NNN]; used by pdf_text, core(), and guard G6.
  • edit_window(coe, cne) — verifies the changed span + context, not the 45-char anchor prefix. Closed a second blind spot: a bogus suffix edit (an appended footnote digit, a phrase swap) had passed because core(ne)[:45] only tested the unchanged prefix.
  • Guards: G0 style-only (a canon-folded quote/dash/marker-only change → reject, rule 9/13), G0b footnote-call (any [digit] bracket add/remove/change → reject, rule 9 — caught 9.9[1]→9.91), G0c emphasis (any * asterisk-count change → reject; *Id.* italics are intentional, pdftotext can't render them).
  • Tests: tests/test_proofing_gate.py (20 cases pinning canon/edit_window/guards).

9b. Recovery + audit (~1,600 fixes)

Re-ran verify→consolidate→apply on the preserved P1–P47 -results.json (anchor- idempotent — already-applied fixes bad_anchor out). Applied 1,611 recovered fixes, then audited every applied edit against the improved gate and reverted 12 (7 *Case*/*Id.* emphasis strips, 4 unverified reconstructions, 1 footnote-call merge — the last surfaced by a failing test_footnote_pinpoints regression). Net ~1,600 authoritative corrections. P1–P41 were riskier (modern- era markup), so 62 markup/citation/marker items were HELD → triage/proofing-p1p41-HOLD-for-review.json (belong to the rule-11 markup pass, NOT applied). Full detail: CHANGELOG-data.md (batches corpus-proofing-gate- recovery-* and gate-recovery-*-revert-*). Reusable resilient applier: triage/gate_recovery_apply_2026-07-02.py (dedups, longest-anchor-first, per-proposal skip on overlaps — use this instead of apply_proofing_proposals.py for any set with cross-file duplicates or intra-opinion overlaps).

9c. Prompt refinements + model pin (scripts/gen_proofing_workflow.py)

  • Ellipsis-widen caution: pdftotext RENDERS the court's compact .... as spaced . . . . (justification artifact); never propose widening. Killed the class (P48 had 23 ellipsis-widen rejects; P49/P50 had 0).
  • Blank-line descope: whitespace covers mid-sentence hard-break rejoin only; blank-line-count normalization is out of scope (owned by paragraph_whitespace.py).
  • Mirrored in triage/corpus-proofing-prompt-v2.md.
  • Model pin (NEW --flagged arg): Sonnet 5 default (sonnet), Opus 4.8 (opus) for ids passed via --flagged (known-hard: heavily-cited, reconstructions, large opinions). Sonnet 5 ran clean at fleet scale (P50: 120/120, 0 rate-limit failures; contrast P42–P45's 328/480 concurrent-run failures — run 1 fleet at a time). Confirm rate climbed with the fixes: P48 84% → P49 88% → P50 89%.

9d. Fleets P48–P50 (all fully worked, ~2,300 fixes, 0 residual holds)

Coverage now contiguous 2011 ND 234 → 2026.

Fleet Range Autosafe Manual Notes
P48 2012 ND 231→113 805 64 first on hardened gate + Sonnet 5; +guard G0c
P49 2012 ND 116→2011 ND 220 652 22+76 first on refined prompt; 24 holds all resolved (6-for-6 correct on re-verify)
P50 2011 ND 234→102 785 73 inline verification — ZERO holds; +1 retry

Inline reconstruction verification (memory [[feedback_verify_reconstructions_inline]]): from P50 on, PDF-verify heading/caption/paragraph reconstructions inline during processing and apply the confirmed in the same pass — hold only what the PDF can't confirm. Method: search the Roman/letter heading directly (\bIII\b, not phrase III); disambiguate repeated anchors by context; strip fused [*NNN]/[¶N] before substring search; simulate paragraph relocations and check PDF order. P50 → 73 manual items all verified, 0 held. Notable fidelity-to-source catches across these fleets: $8,957.26→$3,957.26, date July 1988→1983, 60(b)(6)→60(b)(vi) (court printed roman subsections), Adolph Rub Trust→Adolf, Boston & Maine→Boston & Main, suspended→suspend — each a court typo/actual text the DB had silently "corrected"; fidelity restores the printed form.

9e. Pipeline reference (how to run a fleet now)

# 1. compute next slice, generate (refined prompt + model pin baked in):
.venv/bin/python scripts/gen_proofing_workflow.py OFFSET 120 triage/corpus-proofing-pNN.wf.js [--flagged id,id]
# 2. verify boundary (no gap at OFFSET-1 vs OFFSET), then run ONE fleet:
Workflow({scriptPath: ".../corpus-proofing-pNN.wf.js"})
# 3. extract d['result'] -> triage/proofing-pNN-results.json ; identify dropped ids from the workflow INPUT set
# 4. retry dropped ids:  gen_proofing_workflow.py 0 0 .../corpus-proofing-pNN-retry.wf.js --ids a,b,c  -> Workflow
# 5. verify -> consolidate -> back up DB -> apply autosafe (resilient applier), PDF-verify high-stakes digit/$/citation/signature items first
.venv/bin/python scripts/verify_proofing_proposals.py triage/proofing-pNN-results.json
.venv/bin/python scripts/consolidate_proofing_trio.py pNN
.venv/bin/python triage/gate_recovery_apply_2026-07-02.py triage/proofing-trio-apply.json --batch corpus-proofing-pNN-2026-XX-XX --apply
# 6. manual queues INLINE: apply structure-preserving glyph fixes (Ill/TII->III, TV/rv->IV), verify marker gap-fills (target absent+neighbours present), PDF-verify heading moves/inserts + captions + paragraph reconstructions, apply confirmed
# 7. invariants + quick_check after every batch; snapshot trio files to proofing-pNN-{caption,heading,defer}.json BEFORE the retry consolidate overwrites them

Gotcha: consolidate overwrites triage/proofing-trio-{apply,heading,caption,defer}.json every run — snapshot per fleet before processing the retry. High-stakes digit/$/ citation/rule/date/signature changes pass guard_digit_in_cite only when inside the narrow _CITE pattern; dollar amounts, §/rule/(b)(vi) subsections, and dates slip through — always PDF-verify those individually (all have been correct fidelity fixes so far, but the bar requires the check).

9f. State at close (2026-07-03)

  • 9 commits ahead of origin/main, UNPUSHED. No release cut — corpus changes are LOCAL only; the live MCP server ships from GitHub release zips, so these ~10K corrections are NOT live until a release is built (regenerate opinions.db.zip, tag, push).
  • Working tree CLEAN. Corpus 19,807. Invariants 25 ok / 2 known / 0 regressed. 146 tests pass. quick_check ok. Backups: opinions.db.bak-{gate-recovery,p48,p49, p50}-2026-07-0X.
  • Next: P51 (OFFSET 3910, after 2011 ND 102).
  • Residual: 62-item markup HOLD (rule-11 pass), flag-review backlog, full-corpus cite_extract re-extract, and 3 named P49/earlier holds already resolved (§8/§9).

10. 2026-07-03 (later) — Haiku A/B + fleets P51, P52

10a. Haiku-vs-Sonnet A/B (verdict: KEEP SONNET)

Question: can the fleet run on Haiku 4.5 for cost? Ran a controlled 30-opinion head-to-head against the pre-P50 snapshot (opinions.db.bak-p50-2026-07-03), the same text Sonnet's P50 fleet saw — the Sonnet P50 results are the control arm. First run was confounded (Haiku proofed the already-P50-corrected live DB → recall unmeasurable); re-ran against the snapshot via a new gen_proofing_workflow.py --db that injects an absolute DB path into the agent's read. Result: Haiku recovered 47% (105/223) of Sonnet's gate-confirmed fixes (span-overlap metric, triage/ab_analyze2.py), reject rate 15% vs 10%, 7× guard_emphasis trips vs Sonnet's 0, and its 30 "extra" finds were mostly rule-10 citation-spacing scope violations, not genuine recall. The gate + PDF-reverify firewall means Haiku can't corrupt — it's just not thorough enough for the authoritative bar. Decision rule was ≥90% recall → adopt; Haiku failed decisively. Keep Sonnet 5 workhorse + Opus --flagged. New gen_proofing_workflow.py flags: --model <tier> (batch-wide override) and agent-DB-path injection (both kept).

10b. Fleets P51 + P52 (1,370 fixes / 160 opinions, contiguous to 2010 ND 115)

Both full Sonnet-5 fleets, 120/120, 0 drops, processed per §9e with inline reconstruction verification. Per-batch detail: CHANGELOG-data.md.

Fleet Range Applied Holds Notes
P51 2011 ND 101→2010 ND 237 663 / 82 0 584 autosafe + 68 headings + 11 statute/caption
P52 2010 ND 245→115 707 / 78 2 646 autosafe + 47 headings + 2 list-recon + 12 manual

High-stakes catches (all PDF-verified individually, all correct fidelity): $800,000 →$300,000 (8-for-3 OCR), 6 case names (Haney→Harvey etc.), statute #s 2A-201:3/:6, 06.2(1)(j), thirty (80)→(30) days, [¶4]→[App. P. 4] (DB stored an appendix cite as a paragraph marker), caption reorders, 4 marker gap-fills, AD.S.→A.D.S. (107× in PDF).

The l/1 enumerator rule (important, reusable): P51 id15564 (i)→(1) — court printed a digit 1 in an otherwise-alphabetic (k)(l)(m) sanctions list (500-dpi glyph confirmed top-flag + base-serif); P52 id15462 1.→l. — court printed lowercase l (best-interest factors §14-09-06.2(1)(a)-(m), confirmed by adjacent m.). Rule: trust the pdftotext text-layer glyph when it AGREES with the alphabetic context; image-verify at ≥500 dpi only when text-layer and context CONFLICT.

P52 dispositions worth noting: 2 holds = id15462 ellipsis-spacing (word. . . .→ word . . . ., owned by the deterministic whitespace normalizer; pdftotext ellipsis rendering unreliable). 1 correctly-rejected proposal = id15544 caption Bank→Bak: the court's PDF caption prints the scrivener typo "Bak" but the body spells "Bank" 3× → body-spelling method ([[feedback_westlaw_caption_authoritative]]) keeps "Bank".

10c. State at close (2026-07-03, later)

  • 10 commits ahead of origin/main, UNPUSHED. No release cut. Working tree DIRTY (not committed per standing rule): modified CHANGELOG-data.md, scripts/gen_proofing_workflow.py, triage/proofing-trio-*.json; untracked triage/proofing-p5{1,2}-*.json snapshots + Haiku A/B artifacts (ab_analyze*.py, corpus-proofing-p50-haiku-ab*.wf.js, proofing-p50-haiku-ab*-results*.json). DB edits applied locally (opinions.db doesn't surface in git status — LFS/pointer quirk; corrections live in the DB + changelog).
  • Corpus 19,807. Invariants 25 ok / 2 known / 0 regressed. 146 tests pass. Backups add opinions.db.bak-p5{1,2}-2026-07-03.
  • Next: P53 (OFFSET 4150, after 2010 ND 115).