Skip to content

PLAN-FULL-VALIDATION.md

Latest commit

 

History

History
170 lines (143 loc) · 16.4 KB

PLAN-FULL-VALIDATION.md

File metadata and controls

170 lines (143 loc) · 16.4 KB
Raw

PLAN — Full Validation (2026-07-15)

Goal (per project charter): a database the ND Supreme Court could adopt as authoritative — every opinion's text verbatim to what the court published, metadata validated, the citation graph correct, the companion corpora certified, and standing checks that keep it that way. This document is the complete open-item inventory (reconciled against the TODO-* files, whose stale entries are flagged at the bottom) and the phased plan to get there.

Definition of done ("fully validated"):

  1. Every opinion's text_content verified against its authoritative source (court PDF ≥1997; West .doc/bound reporter pre-1997), with a recorded verification status.
  2. All deterministic defect classes closed or reduced to documented, individually-adjudicated holds.
  3. Citation graph re-extracted under current jetcite with manual queues drained.
  4. Metadata (captions, dates, dockets, panels, citations) validated against independent authority.
  5. Companion corpora (const/statutes/rules/admin/AG/figures) pass their own validation suites, incl. second-engine extraction diffs.
  6. A % validated metric wired into get_database_stats and an invariant dashboard that regresses to zero on every mutation.

Phase 0 — Ship what's done (user action + hours)

# Item Notes
0.1 Release + server deploy DONE: v1.7.0 released 2026-07-15; server deployed by JT 2026-07-16 (verified live: 19,825 opinions, JEAC corpus, validated_by_decade).
0.2 Prune .bak-* snapshots 6 new backups from 07-14/15.
0.3 Retire TODO-distribution.md; refresh stale TODO items (list at bottom) Bookkeeping so the open-item surface is honest.

Phase 1 — Opinion text fidelity, modern era residuals (days)

The modern era (7,232 opinions, P1–P80) is fleet-proofed. Residual classes:

# Item Size Method
1.1 Signature-block verbatim reconstruction PDF-ERA COMPLETE 2026-07-17: (a) data-loss hunt DONE 07-15 (110 flagged/100 restored, batch restore-dropped-signature-paragraphs-2026-07-15); (b) full reformat DONE 07-15 (1,776 opinions) + hold queue DRAINED 07-17 (121 → 2 permanent KNOWN-OK court-printed run-ons; batch sigblock-holds-drain-2026-07-17 — surfaced + fixed a 6-opinion duplicated-rehearing-addendum class). (c) pre-1997 rode seg-A. done —
1.2 Parallel-cite Phase 3 survey needed Injected U.S.+S.Ct.-only pairs, S.Ct.+L.Ed.2d-without-U.S., first-series L.Ed., X U.S. at N, Y S.Ct. at M short-forms (id12428 class). Same hardened PDF gate. Plus re-check the 16 scotus window-only holds + 114 ndnw holds under phase-3 patterns.
1.3 [[Image here]] text restoration apply TEXT SIDE COMPLETE 2026-07-17: 1,141 ellipses applied 07-12, holds drained 07-15, and the 14 no-PDF 1997 opinions (19 placeholders) drained via archive evidence 07-17 (batch fix-imagehere-1997-archive-2026-07-17; incl. 2 KAT trademark figures → figures.db). Residue: 24 PDF-backed figure-side opinions → 5.7. done —
1.4 Footnote Phase 2c COMPLETE: the 19 losses were restored 2026-06-23 (detector PRESENT=178/MISSING=0; this row was stale); 2222/4906 locality redos done 07-17 (batch footnote-locality-redo-2026-07-17). Remaining: Phase 2d analyzer fix (scraper repo) + parser span-bounding for the 59 mid-text westlaw footnotes. done —
1.5 1997 no-PDF cohort COMPLETE 2026-07-16: C-Track PDFs LOST in migration (JT); archive-HTML gate + full queue drain + JT rulings 1–3 (West corrections govern ≤1997; Pavek = West; West omissions restored from archive). ~2,100 fixes, 101 sig blocks, 6 Dated/ATTEST blocks; all 125 promoted court_archive_html. Corpus 99.96% validated — remainder is the 3 Emmons 1900 bound-volume items (2.5) + 4 Westlaw shared-page flags (4.x). CHANGELOG-data.md 2026-07-16
1.6 C1 em-dash holds 109 instances Per-instance manual (line-wrap/ambiguous); or image-verify.
1.7 Deterministic small classes — Separator-row garbles (⅜ ⅜ ⅜→* * * * *, flagged P77/P80); 91 star-page word-splits (needs convention ruling); 62-item markup HOLD + *Case**,* emphasis (markup pass); blank-line collapse (cosmetic, LAST); underscore-for-ellipsis residue sweep (_→...., pattern now known).
1.8 Singleton holds 2 id12513 ¶8 concur (source ambiguous); id12761 ambiguous placeholder.

Phase 2 — Pre-1997 text fidelity (the big block, weeks)

Post-seg-B reality: 11,991 of 12,593 pre-1997 opinions are West-.doc-primary (95%) — the deterministic diff method covers nearly everything; only 584 markdown (seg C) need LLM+human.

# Item Size Method
2.1 Seg A close-out: ws-insensitive body-clean confirm (use gtimeout), front-matter strip pass (~237 diverged), 7 residual diverged, 8 promote-holds (merged-record class) 11,991 Deterministic diff (doc_db_diff.py), NOT fleets. Gate: >1 N.W. parallel / >1 .doc source ⇒ manual (2473 lesson).
2.2 Seg A front-matter policy — Ratify what the canonical body includes (syllabus, counsel lines, "Opinion of the Court by X") so the diff has a contract. Syllabus-cutoff question (§1): sample .docs at 1955/65/75/85/95.
2.3 Seg C DONE 2026-07-15 (evening): fleets segC1–2 sampled 215 (94% clean) → JT ruled deterministic for the rest (segc_md_db_diff.py): 577/584 DB==source, 7 KNOWN-OK, 1 print-verified fix (Bauer/Nestos); 579 promoted single_source_accepted. Quality ceiling = SOURCE OCR → N.W. page-scan acquisition is the upgrade path; 418 fleet flags feed the star-page/markup passes. See HANDOFF-2026-07-15b.md §3. 584 done
2.4 Cite-spacing pre-1997 ~4,820 opinions deferred Blocked on .doc space-drop (textutil). Options: RTF-native parser for spacing, or N.W.2d print spot-checks per (abbrev,year) class.
2.5 Bound-volume residuals — Hassen v. Salem N.D.-cite conflict (50 N.D. 825 vs 813) vs bound vol; 1890–1924 text-merge triage (455 rows).
2.6 Pre-1953 “Syllabus by the Court” completeness — Verify restored syllabi coverage vs Westlaw .docs (most restored in seg-A promotes; audit the remainder).

Phase 3 — Citation graph (days, mostly automated + queues)

# Item Size
3.1 Full-corpus cite_extract re-run under jetcite ≥2.5.5 (whole text_citations predates it) + ag_backlink rebuild corpus-wide
3.2 Manual queues: reconcile-manual 676, collisions 213, antecedent-witness 1,535, caption-lint 460 (bulk classes first) 2,884 rows
3.3 §14 residuals: antecedent_name coverage (95% NULL — jetcite span fixes), parallel-group for out-of-corpus cites, recognizer omissions (spelled-out const refs, statute ranges), garbage-volume tail —
3.4 §13 const-cite coverage: wire nd_const_phrases.match() into cite_extract; old-form recognizers; phrase-table buildout —
3.5 jetcite attorney_general cite type (≥2.6.0) → simplify ag_backlink upstream
3.6 Statutory cross-reference edges (provision_xrefs) + generic cross-DB citation_edges model design first

Phase 4 — Metadata validation (days + acquisitions)

# Item Size
4.1 Court-report (cTrack xlsx) field validation: case_name 1,808 CAPTION QUEUE CLOSED 2026-07-17, zero DB corrections (subcategorizer triage/casename_subcat_2026-07-17.py; all diffs = policy/report-side; incl. the "State v. Lopez" wrong-cite report row); 13 staged date suspects ADJUDICATED 2026-07-17 (2 fixed: Delzer 1997-01-16, Hart 1905-11-24 vs bound vol 14 image; rest KEEP with documented press-time-insertion findings). author residue / cite-mismatch / collisions CLOSED 2026-07-17 (10 author fixes incl. 4 surrogate D.J.s; 3 table-cite fixes; 2 dup merges → corpus 19,823; 1 HELD: Petition of Thompson author convention). date >31d DRAINED 2026-07-17 (65 rows: 34 report-side, 18 DB fixes — original-opinion-date convention for the As-Modified class, 13 keeps/known; future: sub-31d As-Modified sweep). docket cross-check DONE 2026-07-17 (12,069 rows; cTrack scheme = 18600000+docket; 1 fix: Traynor v. Leclerc caption/dockets; 58 consolidated-companion docket enrichments noted). Nature-of-Action→case_type DONE 2026-07-17 (4,976 fills → coverage 11,994/19,823; SCHEMA.md Contract 5). REMAINING: consolidated-docket enrichment (58 noted) —
4.2 Pre-1997 judges/panel rebuild from justices.py + disqualification lines (now that concur lines are restored) —
4.3 Empty dockets ~5,072 (all pre-1931) — bound N.D. Reports / historical index investigation; re-run spreadsheet backfill on next court export manual
4.4 DIVERGED scalar review COMPLETE 2026-07-17: 0/0 (295→107→0; 21 silent author overwrites RESTORED — watch item: identify the unlogged pre-guard writer; 86 adjudicated-current) done
4.5 Citation hygiene RULED 07-15 + VERIFIED COMPLETE DB-side 2026-07-17: citations carries ZERO LEXIS rows (the 15,253 pointers are source-file YAML frontmatter that ingest filters; never entered the DB); in-text LEXIS = 52 opinions, the court's own citations, KEPT per ruling (4 injected variants stripped 07-15); L.R.A./A.L.R. kept done
4.6 MISSING_OPINIONS refresh (report predates dedup): re-run audit; then chase the ~14 phantom admin-order cites (2xxx ND 28x class — likely administrative orders; policy: ingest or annotate?), 2 missing source files (1997ND5.md, 1998ND22.md — DB text exists), multi-cite conflicts, missing NW/NW2d volume list —
4.7 Truncation dashboard triage: 38 flags (31 CONTENT) PDF-verified pass 38
4.8 Audit follow-up queues: class3 parallel clusters ~25; OTHER parallel-pair 271 sample+classify; parallel_pair 556 / para_continuity 520 / xref 759 dashboard drains —

Phase 5 — Companion corpora certification (parallel track)

# Item
5.1 Const crosswalk (#1): acquire NDCC Replacement Vol 13 (1981) disposition table (user/library action) → lineage table → close the ~98 rewritten-provision matches.
5.2 Const structural (#2): apply the 4 structural amendments upstream in ingest_constitution_history (done on scratch); clean 54 ndconst.org extraction artifacts; amendment-event dedup (#3); external census cross-check (#4, Blue Book/SoS); snapshot-harness filed holes (#5: art VII pre-1982 heads, art IV stubs, art V §14, art X decimals; 1942 BB acquisition).
5.3 Statutes/NDAC: regenerate ~/refs markdown with fixed converters + commit scraper fixes (PL-1b/c, PL-2) — re-ingest safety; PL-3 table-split tail (~5 sections); NDAC freshness pass (§20-03-01-02 may be stale); request native XML/ODT source from Leg Council (PL-SOURCE-FILES — retires the PDF-extraction bug class).
5.4 PL-VALIDATE validators: #2 second-engine extraction diff (the capstone certifier), #3 section-inventory vs ndlegis, #4 enumeration continuity, #5 doubled-token, #6 heading/body sanity, #7 char histogram.
5.5 Rules: full-build H1-title/same-date-twin guards; mirror the c/d/e/l/m appendices.
5.6 AG: 12 ocr_derived cite review; declare httpx/bs4 in [ingest]; 6 broken-link gaps documented as unrecoverable.
5.7 figures.db: crop 14 table renders to bbox; 16 unmarked figures (marker-insert deferred); 14 no-PDF placeholders via bound volumes.

Phase 6 — Continuous assurance + the "validated" metric (finish line)

# Item
6.1 Verification-status provenance DONE 2026-07-15 (evening): earned promotions into validation_status (P1–P80/segA/segC/C-Track; scripts/promote_validation_2026-07-15.py) + validated_by_decade in get_database_stats → 99.3%; unvalidated remainder = the 117 no-PDF 1997 cohort + 3 ND-image.
6.2 Build remaining audit checks: #8 char-histogram (all DBs), #9 two-column interleave detector, #13 frontmatter-vs-citations sync, #15 enumeration continuity, #16 doubled-token standing scan, #17 heading/body split, #18 section sequence, #20 FTS sync (all DBs).
6.3 Tier-2 external checks: #21 ndlegis inventory, #22 second-engine diff, #23 upstream-correction watch (weekly pipeline), #24 CL re-diff by cluster_id, #25 per-year counts vs court statistics, #26 statute currency heartbeat.
6.4 Prevention: pilcrow-OCR detection at ingest; scraper freshness heartbeat (>3wk alert); letter-spaced header cleanup in scraper; optional re-ingest tombstone guard.
6.5 SCHEMA.md data dictionary + db.py create_schema reconciliation (§11).

Decisions — RULED by JT 2026-07-15

  1. Release v1.5.0 — SHIPPED 2026-07-15 (tag 355acb6, all 7 assets, marked Latest). Server deploy: HOLD on v1.4.0 — do NOT run update-db.sh yet; deploy with a later release at JT's call.
  2. Citation hygiene: strip LEXIS only. Remove the CL-injected proprietary LEXIS pointers (15,253 frontmatter). KEEP L.R.A. (real parallel reporter) and A.L.R. (finding aid) — both reflect real historical publication.
  3. Admin-order phantom cites: investigate then ingest. Fetch the ~14 documents from court records, ingest with a document-type flag (the precedential_status pattern) so they're excluded from opinion statistics.
  4. Seg-A front-matter contract: syllabus + counsel + body — the full official report as the bound reporter printed it, minus West editorial. Matches the existing pre-1953 syllabus policy.
  5. Star-page mid-word convention: AFTER the word (informing [*214]) — marker moves to the first word boundary after the true break; the split word attributes to the earlier page; no-mid-word invariant preserved.
  6. Replacement Vol 13: JT scans/photographs the disposition-table pages from a library copy → ~/Downloads; we extract and build the lineage table.
  7. C-Track PDFs: JT fetches manually. We generate the docket worklist (triage/ctrack-pdf-worklist.tsv); JT pulls from C-Track and drops PDFs in; we stage into ~/refs/nd/opin/pdfs/<year>/ and re-verify the no_pdf-deferred fleet items.

Suggested sequence

Phase 0 immediately (release). Then run Phase 2.1 (seg A) as the backbone — it's deterministic, covers 95% of pre-1997, and every other track can interleave: Phase 1 items during seg-A batch waits; Phase 3.1 (cite re-extract) after any big text wave lands; Phase 4/5 as evening/queue work; Phase 6.1 (the metric) EARLY, not last, so progress toward "fully validated" is visible from now on.

Cursor as of 2026-07-16 (v1.8.0): THE VALIDATION LEDGER IS COMPLETE — 19,825/19,825 opinions carry an earned status; validated_by_decade reads 100.0% in all 14 decades. Phases 0–2, 1.5, 2.5's Emmons tail, and 6.1 are DONE (modern P1–P80, segs A/B/C, the 1997 no-PDF cohort via the archive gate + JT rulings 1–3, the Emmons 1900 trio vs the vol 9 scan, and the last 4 westlaw-receive flags). v1.8.0 released (all 8 assets); server deploy of v1.8.0 pending (JT). REMAINING: Phase 1 residuals (1.2 parallel Phase 3, 1.3 [[Image here]] holds — the 1997 placeholders now have archive evidence, 1.4 footnote 2c), 2.4/2.6 residuals, Phase 3 (cite re-extract NOW STALE-DRIVEN: today's text wave touched ~130 opinions), Phase 4 metadata (+ execute the ruled LEXIS strip), Phase 5 companion certification, 6.2–6.5; JT style-slice decision (archive-style edits applied pre-ruling-1). See HANDOFF-2026-07-16.md §3.

Stale TODO entries to prune (verified done as of 2026-07-15)

  • TODO-validation §17 item 3 "next = P46" → fleets DONE through P80; modern era complete.
  • TODO-validation item 2 structural markers → marker-triage queue clean (07-13/14).
  • TODO-validation §6 ellipsis normalization → normalize-ellipsis-modern-2026-07-08 shipped.
  • HANDOFF-07-08 §5 unpublished-disposition decision → DECIDED 07-11 (keep+flag, precedential_status).
  • HANDOFF-07-08 seg-B "47 batches remain" → seg-B COMPLETE 07-12 (48/48).
  • HANDOFF-07-08 "CL parallel-cite enrichment pass" → Phases 1–2 shipped 07-14 (Phase 3 = item 1.2).
  • TODO-audit "invariants.py covers opinions.db only" → 90 checks / 7 DBs since 07-13.
  • TODO-ag "not yet pushed / no release" → AG shipped in v1.2.0/1.2.1.
  • TODO-reversion-recovery "MAKING" garble → fixed 07-15 (fix-maring-ocr-signature); other names roll into 1.1.
  • TODO-const #0 modern point-in-time → COMPLETE + LIVE (v1.0.0, snapshot-diff run 07-13); #5 holes remain (5.2).
  • MISSING_OPINIONS.md → predates dedup (corpus 20,382 vs 19,825); re-run before acting (4.6).