Skip to content

HANDOFF-2026-07-15b.md

Latest commit

 

History

History
135 lines (125 loc) · 8.48 KB

HANDOFF-2026-07-15b.md

File metadata and controls

135 lines (125 loc) · 8.48 KB
Raw

HANDOFF — 2026-07-15 (b): evening session

Follows HANDOFF-2026-07-15.md (the morning/afternoon three-wave day). This session executed the ratified sequencing from that handoff's §5 top-to-bottom and closed four arcs. Released: v1.7.0 (all work committed; see §6).

0. Status at a glance

  • Corpus 19,825; invariants 93 OK / 0 regressed (5 new JEAC checks); 235 tests (8 new); validated_by_decade = 19,693/19,825 = 99.3%.
  • The pre-1997 text-fidelity arc is COMPLETE (seg A verified 07-15 am, seg B done 07-12, seg C certified this session).
  • New corpus: jeac_opinions.db (26 JEAC opinions, alias jeac, 4 tools).
  • Server deploy still HELD on v1.4.0 — JT's call (three releases pending: v1.5.0, v1.6.0, v1.7.0).

1. Markdown/LaTeX contamination class — CLOSED (batch fix-mdtable-contamination-2026-07-15)

  • id20180 (2025 ND 193) rebuilt vs its image-only second-corrected PDF (OCR + render adjudication of every span); date_filed → 2025-11-20 (C-Track stamp + neighbors; DB had the correction date).
  • The class generalized: 22 opinions (2024–2025 + LaTeX stragglers 2021 ND 42, 2023 ND 154), ~380 edits via a 20-agent render-verified fleet + central count/residue/marker-gated applier. Signature hidden damage: 17 dropped section headings, a missing [¶2] (2025 ND 26), a dropped sentence (2024 ND 162 ¶13), a dropped footnote citation (2024 ND 109), missed clerk-correction deltas ("multiplications"→"multiplicitous", nested-quote closers, 62(l)).
  • Root cause found & fixed in ~/code/scraper (pushed): marker-pdf+Claude OCR fallback emits markdown; nearly all 2024-25 court PDFs have broken ToUnicode text layers (mojibake passes a length gate). Fixes: legibility gate (_text_is_legible: common-word ratio + word density), ocrmypdf+ pdftotext -layout as default OCR engine, marker demoted to explicit opt-in.
  • 2002 ND 122 (Johnson Farms) C-Track-verified: 16 fixes incl. the missing caption/front-matter block, counsel P.O. boxes, verbatim signature lists, cite-spacing, injected-parallel strip. JT RULED: fn 2 is properly numbered 2 (the print's "3" is a publication glitch). HOLD: include the PDF's "Editor's Note" preamble? (policy).

2. NO_TARGET root fix (batch strip-archive-header-frontmatter-2026-07-15)

  • All 14 false edges traced to ONE cause: residual archive-HTML prepended headers (synthetic cite lines w/ placeholder N.W.2d pages) in 43 opinions the corpus-wide preamble strip missed. Stripped; those opinions' citations re-extracted → the false edges cannot resurrect from any rebuild.
  • Belt-and-suspenders: cite_extract.apply_print_anomaly_overrides now also processes print_anomalies rows with kind='false_edge' (printed_cite set, intended_opinion_id NULL) → durable suppression for future NO_TARGET verdicts whose text is genuine court print. No rows inserted yet.

3. Segment C — CERTIFIED (the 584 reporter-markdown pre-1997 opinions)

  • Fleets segC1–segC2 sampled 215 opinions (94% clean; ~10M subagent tokens), then JT ruled option 3: deterministic sweep instead of fleets 3–5.
  • scripts/segc_md_db_diff.py (seg-A method for markdown; fold heading MARKERS not heading lines — first run stripped source captions and read 0.5% clean): 577/584 DB==source; 7 divergent all KNOWN-OK (triage/segC-divergences-2026-07-15.KNOWN-OK.json).
  • One print-verified fix: Bauer v. Nestos (48 N.D. 894) — two attribution lines dropped at ingest restored from the NW/187/233.pdf p.236 scan.
  • Ledger: 579 → single_source_accepted (honest tier: the reporter markdown is the only on-disk authority). Seg C's quality ceiling is its SOURCE OCR — 418 fleet flags (triage/proofing-segC-flags.json) mostly document source-level artifacts; upgrade path = N.W. page-scan acquisition (only 47 NW + 40 NW2d scans on disk).
  • HELD for JT: id5118 caption proposal ("New Hampshire, and v." restore, source-confirmed) in triage/proofing-segC1-results.review.json.

4. Validation metric (plan 6.1) — LIVE

  • validation_status refreshed with EARNED promotions only (scripts/promote_validation_2026-07-15.py + segC/C-Track batches): P1–P80 → cross_checked/court_pdf (6,993); seg A → cross_checked/westlaw_doc (5,646); seg C → single_source_accepted (579); 5 C-Track verifies.
  • get_database_stats now returns validated_by_decade. 99.3% overall; the ONLY unvalidated remainder: the 117-opinion 1997 no-PDF C-Track cohort (awaiting JT pulls) + 3 ND-image pre-1953 items.

5. NEW CORPUS: jeac_opinions.db (feat commit dcc9ca5)

  • 26 opinions (1990-1 → 2026-1; the TODO's "low hundreds" guess was 26, and the first page-parse read 22 — four links use absolute URLs, one with <br> inside the anchor and a two-digit "Opinion 22-1.pdf" filename; the exact-count recheck caught 2022-1, 2025-2, 2025-3, 2026-1).
  • Full arc in one session: scrape (curl_cffi) → ~/refs/nd/jeac/ (6 image-only originals OCR'd) → schema+FTS (status column modeled; no withdrawn markers on the page as of 2026-07) → jetcite outbound + canon-level supplement for pre-2007 spelled-out Canon cites (rules.db keys the Code at canon grain; old-Code canon 5 stays unresolved by design) → backlinks (1 prose reference: Bauch v. Bauch 1997) → 4 MCP tools + lookup_authority construed-by → 5 invariants, 8 tests, release asset, docs (CORPUS-SCHEMA/TOOLS/NOTICE/CLAUDE/~refs).
  • Upstream jetcite TODO (plan 3.3): recognizer for the spelled-out Canon form (would also catch it in the ~56 court opinions citing the Code).

6. Commits / release

  • ndcourts-mcp: 005fb6b (data work), dcc9ca5 (JEAC), b8b0d61 (bump 1.7.0) + this handoff commit. v1.7.0 released with 8 DB assets (first release shipping jeac_opinions.db).
  • scraper: 9e82c52 (mojibake gate + ocrmypdf default) pushed.

7. What's left — pick up here

  1. Review queues (per-item sessions): sigblock reformat holds ~106; star-page off-track 710; segC flag classes (star-page residue / markup-contamination feeds for the existing passes); id5118 caption proposal + 2002 ND 122 Editor's-Note policy (JT).
  2. Phase 4 metadata: the 13 date_filed suspects (triage/date-filed-suspects-2026-07-15.json) feed 4.1 court-report validation (case_name 1,808 / date >31d ~130 / author ~25 / 94 cite collisions / Nature-of-Action→case_type); pre-1997 panel rebuild (4.2); empty dockets ~5,072 (4.3); DIVERGED scalars 295 (4.4); MISSING_OPINIONS refresh + ~14 phantom admin orders (4.6, JT ruled ingest-with-flag).
  3. Blocked on JT: server deploy (v1.4.0 → v1.7.0); Replacement Vol 13 scan (const crosswalk 5.1); C-Track 1997-cohort pulls (161+1998 ND 22 — the last 117 unvalidated + 72 sig holds unlock); the "2017/2018 list" clarification (probably the [[Image here]] figure list: 2017 ND 244/254/262/269 need better scans); italics-loss class ruling (deferred).
  4. Plan phases 5–6 remainder: const structural (5.2), statutes/NDAC regeneration + PL validators (5.3–5.4), rules full-build guards (5.5), AG residue (5.6), figures crops (5.7); audit checks #8–#26 (6.2–6.3); prevention hooks (6.4); SCHEMA.md reconciliation (6.5).
  5. Acquisitions worth deciding: N.W. page scans for seg C (its quality ceiling); bound-volume sources for the [[Image here]] no-PDF tail.
  6. Upstream jetcite: spelled-out Canon recognizer; attorney_general cite type (3.5).

8. Durable traps (new this session, carry forward)

  • Broken-ToUnicode text layers pass length checks: 2024-25 court PDFs return full-length mojibake from pdftotext. Gate on legibility (common-word ratio ≥8% AND ≥70 alpha-words/kchar — measured split: real ≥130, mojibake ≤40), never on length alone.
  • Markdown artifacts in opinion text = tripwire for deeper loss: every contaminated opinion also risks dropped headings/sentences/footnotes — always full-diff vs a render, never spot-fix the visible markup.
  • Committee pages mix relative and absolute hrefs (+ markup inside anchors, + two-digit filenames): parse with markup-tolerant anchors and verify the item COUNT against the visible page before trusting a scrape.
  • Fold markers, don't strip lines in source-vs-DB diffs: the reporter markdown stores captions as # headings; deleting heading LINES made every caption a false divergence (0.5% clean → 98.8% after the fix).
  • validation_status promotions must stay EARNED (design contract): cite the batch/authority per promotion; never silent-backfill 'cross_checked'.