Skip to content

HANDOFF-2026-07-13.md

Latest commit

 

History

History
178 lines (168 loc) · 11.6 KB

HANDOFF-2026-07-13.md

File metadata and controls

178 lines (168 loc) · 11.6 KB
Raw

HANDOFF — 2026-07-13 (end of session)

Supersedes HANDOFF-2026-07-12.md. Validation/verification session: invariants extended to all 7 DBs (+ const FTS repair), rules currency pipeline built + first live sync (July 2026 amendment cycle in; Admin Order 21 repeal reconciled), citation graph rebuilt under jetcite 2.5.5 (West spaced-abbreviation fix recovered 40k cites), §4 name reconciliation shipped (934 corrections), three new standing audits, and the sigscan-38 triage worked to zero genuine defects (9 PDF-verified restorations). Read top-to-bottom; NEXT SESSION STARTS AT §7 (const snapshot-diff harness).

0. Status at a glance

  • Corpus 19,825; invariants 86 ok / 4 known / 0 regressed (dashboard now covers opinions + const/rules/statutes/admin + AG + figures — 90 checks); 205 tests (43 new this session).
  • Citation graph: 359,106 text_citations / 126,387 cited_by (was 334,603/114,647), antecedent coverage 78.6%; 959 print_anomalies overrides active (25 prior + 934 name-reconciled).
  • rules.db: July 1, 2026 cycle applied (21 amendments, 22 silent corrections, 4 new rules); Admin Order 21 repealed eff. 2026-07-01; mirror marker at 46235a9; weekly Stage 3 automation live in ~/code/scraper/tools/weekly_scrape.sh.
  • Git: committed through HEAD of main (this session: 98c8242 invariants, 9066528 rules pipeline, ae1a9aa Order 21, 885577f citegraph/§4, sigscan/handoff commit). 6+ commits ahead of origin — NOT PUSHED (JT's call). Scraper repo: Stage 3 committed (e9a8404).
  • jetcite 2.5.5 MERGED TO main (ad0a37d, tagged v2.5.5): the multistate branch (Arizona/Iowa support + the spaced-abbreviation fix) was merge-reviewed end of session — differential extraction over 120 opinions showed 207 gains / 0 losses; the AZ-neutral removal matches nothing corpus- wide; 510 jetcite + 205 ndcourts tests green post-merge. main now reproduces the extraction that built today's citation graph. multistate is fully merged (deletable).
  • NOT RELEASED / NOT DEPLOYED: all of today's DB changes (rules July cycle, Order 21, const FTS backfill, citegraph rebuild + reconciliations, sigscan restores) are local. Next release ships opinions.db + rules.db + constitution.db at minimum. Live server (ndconst:/srv/ndcourts) still on v1.3.2. Note: deploy/self-update.sh notifications renamed ndcourts→ndlaw + FQDN (commit 898f372).

1. Invariants — all corpora (DONE)

invariants.py 29→90 checks: 14 structural per primary-law corpus (CORPUS-SCHEMA Contracts 1–3: current-version integrity, window overlap/seam, two-way FTS parity, orphans, status domain, Reserved-exempt non-empty text) + 3 figures.db checks (cite→opinions resolution, fig_index uniqueness, image sha256). Missing DBs skip silently. First catch fixed: 145 const provision_versions unindexed by the modern-splice batches (30 current provisions invisible to search_authority) — FTS backfilled, parity 774/774. Baselines documented in BASELINES (const seam gaps 278 = historical-layer modeling; const active-no-current 219 = 1889 layer awaits 'superseded' status). Tests: tests/test_invariants_corpora.py.

2. Rules currency pipeline (DONE) + Order 21 (RESOLVED)

  • ingest_rules: full builds REFUSE over a populated rules.db (#14b footgun closed; --force backs up first); --baseline records mirror_head; --update [--apply] incrementally applies mirror commits since the marker. Divergence guard: DB standing text must equal the mirror's marker-time state or the file HOLDS (protects the 2026-06-10 surgery + hand-added appendices). Foreign-page H1 guard is kind-aware (admin-order vs admin-rule share bare numbers). Already-applied check makes reruns after partial applies clean. Exit 0/1/2 = clean/holds/fatal; marker advances only hold-free.
  • Weekly Stage 3 (weekly_scrape.sh): rules-scraper --update --all → ingest_rules --update --apply → invariants; macOS banners on holds/changes.
  • Admin Order 21: the court's page serves Admin RULE 21's document (site publication defect — root cause of the 2026-06-10 surgery). Order 21 (Minority Justice Implementation Committee) repealed eff. 2026-07-01 (JT confirmed; Order 35 consolidation No. 20250147 has no MJIC). Reconciled via triage/admin_order_21_reconcile_2026-07-13.py: v1978 closed, 4 foreign events deleted, 2021 event re-pointed, seam artifact fixed, 3 Admin R. twin refs re-pointed. Both rules invariant baselines retired to 0. The mirror file is in SITE_DEFECT_SKIPS (revisit if the court fixes the page).
  • Durable trap: mirror commit dates (author AND committer) are backdated to effective dates — never date-anchor; use the sha marker.
  • OPEN: when the scraper eventually captures N.D.R.Ct. appendices c/d/e/l/m, expect "file new to mirror but provision exists" holds — reconcile the NULL-start manual rows then.

3. Citation graph — jetcite 2.5.5 rebuild + §4 reconciliation (DONE)

  • Root discovery: West's bound-volume house style spaces two-letter reporter abbreviations (49 N. D. 915, 194 N. W. 663, 210 N. W. 2d 82, 347 U. S. 483). 4,351 opinions (22% of corpus) — every such cite missed by ALL prior extractions. Fixed in jetcite 2.5.5 (c312c44; normalized stays compact; NDCC/NDAC lookahead still blocks spaced statute forms).
  • Full re-extract (clear tc/cb/progress → cite_extract, ~80s): +40,215 case cites, +13,463 edges. 1,723 removed edges verified as phantoms (old bodies' bare star-page runs misread as cites — e.g. oid 8000 278 N.W.2d 120–123). process_all now applies print_anomalies overrides itself (was --cited-by-only only).
  • cite_reconcile (§4) shipped + applied: unique antecedent-name match + two-of-three cite-component anchor + temporal guard → 934 corrections (86 false-edge re-points, 848 recovered edges) written to print_anomalies (kind='citation' — durable across rebuilds; used instead of a match_kind column). Guards learned live: confusable names (Jacobson/Jacobsen — error direction ambiguous) → manual; generic procedural tokens (Interest/Matter/Estate) non-distinctive; fictitious-citation opinions excluded (City of Dickinson v. Helgeson prints a table of a litigant's AI-hallucinated cites — 20340 must never be "reconciled").
  • New audits (audit_corpus): antecedent_witness (960 residual flags), sig_truncation (see §4), case_name_lint (460 flags).

4. Sigscan-38 triage (DONE — zero genuine defects remain)

PDF-arbitrated all 38 flags (triage/sigscan-triage-classified-2026-07-13.tsv; restore script triage/sigscan_restore_2026-07-13.py, batch sigscan-restore-2026-07-13):

  • 9 restored (11 edits): 7 dropped signature/surrogate blocks (1997 ND 246 incl. garbled [II17]; 2001 ND 87; 2001 ND 130; 2003 ND 139; 2007 ND 41; 2009 ND 106; 2017 ND 174); 2021 ND 144 missing substantive ¶12 (50 U.S.C. § 3932(b) analysis; tail renumbered ¶13–15 — the 2026-06-21 audit had wrongly called this a numbering FP); 2023 ND 51 rehearing ¶¶10–12 (court updated the published PDF post-filing — the audit-#23 upstream-correction class caught in the wild; DB ¶4 verified identical to PDF ¶4).
  • Residual 30 flags all adjudicated non-defects: 17 source-tree contamination (DB correct, ~/refs bad), 10 no-PDF 1997 cohort (Priority B/C backfill), 13291 (¶478 PDF-text-layer artifact), 12747 (1998 ND 171 scrambled text layer, deferred), 16985 (source contamination after restore).

5. Open review queues (accumulated, prioritized)

Queue Size Where Note
Reconcile manual 463 triage/cite-reconcile-manual-2026-07-13.tsv confusable names / no-anchor; judgment + spot PDF checks
Witness residue 960 triage/audit-antecedent_witness-2026-07-13.tsv mostly benign abbreviation mismatches + §14(d) bad spans
Caption lint 460 triage/audit-case_name_lint-2026-07-13.tsv 342 double-space (bulk-safe), leaked dates/terms, Westlaw a1 markers — good fleet-style batch
Shared-page collisions ~397 triage/cited-by-unresolved-collisions.csv grew with the bigger graph
~90 opinions.db.bak-* — repo root prune (housekeeping, carried from 07-12)

6. Recommended next-session priorities

  1. §7 below — const snapshot-diff harness (agreed next; tracked task #7).
  2. Release + deploy: cut a release carrying all of today's DB changes; deploy to ndconst (self-update.sh runs automatically on release detection).
  3. Caption-lint bulk classes (double-space collapse, a1 strip) — quick wins.
  4. Tracked task #8 (reversion-recovery Phase 2+4), then #9 (heartbeats).
  5. Routine: proofing fleets P67 → ~P78 (OFFSET 5830, after 2003 ND 34).

7. NEXT UP: Constitution snapshot-diff harness (task #7 — the capstone)

Goal: certify constitution.db end-to-end. Reconstruct the FULL constitution as of date T from the DB (lookup_authority semantics: version in force at T for every provision), render to comparable text, diff against the official compilation nearest T. Green diffs at 1925 / 1954 / 1973 / current ⇒ the const corpus is demonstrated correct, not asserted.

  • Prototype exists: triage/const-pilot-2026-06-14/snapshot_diff.py — scored 97–99% at 1925/1973 on the scratch DB; re-run against the LIVE constitution.db (modern layer shipped v1.0.0, so the whole timeline is now meaningful).
  • 14 official compilations on disk 1889–1981 (see TODO-const-history-validation.md §5 for the inventory/paths) + current text (ndconst.org already mirrored). Historical-layer citations are the 1889 flat §§ 1–217; modern layer is art./§ — pick the layer by T ≷ 1981-07-01 (the reorganization seam; crosswalk PL-CONST-CROSSWALK is NOT built, so cross-seam identity mapping is out of scope — diff each era against its own numbering).
  • Expect noise classes from the pilot: OCR of the old compilations, punctuation/whitespace normalization, section-ordering. Normalize like refs_diff does (whitespace/case/punct-insensitive banding) and report per-provision similarity bands, not a single pass/fail.
  • Deliverable shape: scripts/const_snapshot_diff.py (or promote the pilot), TSV per snapshot date to triage/, summary bands, and a CHANGELOG-data-primarylaw.md entry for any DB errors it surfaces (each correction changelog'd per feedback_log_changes).
  • Related open const items (same file): amendment-census external cross-check; the 3 structural splices' upstream application (TODO-const-history §2 — check whether the scratch-DB work was folded before trusting; memory says 10/13 done, 3 structural remained as of 06-08 but the modern promotion may supersede — VERIFY against the live DB before re-doing anything).
  • Tracked as task #7; #8 (reversion Phase 2+4) and #9 (heartbeats) queued behind it.

8. Durable traps (carry forward)

  • Mirror commit dates backdated (rules) — sha-anchor only (§2).
  • cite_extract full rebuild = clear text_citations+cited_by+ cite_extract_progress first (INSERT OR IGNORE never refreshes rows).
  • print_anomalies is the ONLY durable edge-override layer — never hand-edit cited_by (rebuilds wipe it).
  • Fictitious-citation opinions (20340 + future sanctions opinions): excluded from reconciliation by text markers in cite_reconcile._FICTITIOUS_MARKERS.
  • Court PDFs can change post-filing (2023 ND 51 rehearing append) — audit #23 (upstream-correction watch, task #9) is the systematic answer.
  • Receiver recurses into subdirs; macOS has no timeout; promote-revert must fix both opinion_sources and denormalized opinions.source_* (07-12).