Skip to content

HANDOFF-2026-07-11.md

Latest commit

 

History

History
92 lines (85 loc) · 7.69 KB

HANDOFF-2026-07-11.md

File metadata and controls

92 lines (85 loc) · 7.69 KB
Raw

HANDOFF — 2026-07-11 (end of session)

Supersedes HANDOFF-2026-07-08.md. This session: Westlaw seg-B batches 2–7, the unpublished-disposition ruling, finished segment A by promoting the West .doc over the diverged Lexis/archive text (~232 opinions), recovered P59 gate-rejects, uncovered a corpus-wide leaked-preamble class (5,516 opinions), and shipped release v1.3.0. One-line live state: memory project_ongoing_work.md (2026-07-11 section). Read top-to-bottom before resuming.

0. Status at a glance

  • Corpus 19,822; invariants 29 ok / 2 known / 0 regressed; quick_check ok; FTS 19,822/19,822.
  • RELEASE v1.3.0 SHIPPED (main @ d78fb4a; https://github.com/jet52/ndlaw/releases/tag/v1.3.0). All this session's work + the 2026-07-08 P59 work is committed on main and live in the release (all 6 DB assets incl. the 740 MB opinions.db.zip). Working tree clean.
  • NOT deployed to the live MCP server yet (ndconst:/srv/ndcourts via update-db.sh — separate step, [[project_server_deploy_ops]]). The running server still serves the pre-v1.3.0 DB.

1. ⭐ BIGGEST OPEN ITEM — corpus-wide leaked front-matter preamble (5,516 opinions, 28% of corpus)

Full deep-dive done; NOT applied — needs a user decision + a pipeline forward-fix. See TODO-leaked-frontmatter.md + memory project_leaked_frontmatter_preamble + CHANGELOG-data.md.

  • 5,516 opinions (68% of modern 1997+) begin text_content with a synthesized header: # <Case Name>\n\nNorth Dakota Supreme Court\n<cites>\nDecided <date>\n\n[## Opinion].
  • Confirmed a fabricated ingest header — not in ANY current source file (verified across 1997–2022; all ND-markdown-primary, sources start at the cite/caption). Redundant with the case_name/court/ citations/date_filed columns. Even 2022 opinions have it → the live scraper pipeline still adds it (forward-fix needed, generator not yet pinned — it's in the scraper's text_content construction).
  • Safe to strip, no text loss: dry-run over all 5,516 = 5,494 clean strips + 22 held (all just very long multi-party captions, also correct); 99.7% keep their [¶] markers; the 14 that don't are legit unnumbered 1997 orders / 2005 COA. Use the regex-strip (scripts/strip_segA_frontmatter.py, PREAMBLE regex, id list frozen triage/leaked-preamble-all-ids.txt) — NOT re-ingest (would discard the proofing corrections).
  • Open policy calls before applying: (a) for ~3,900 "straight-to-author" variants the leaked # heading is the ONLY full-party caption in text_content (stays in case_name/case_name_full cols + sources) — strip it or preserve a caption? (b) also strip the follow-on ## Opinion marker (795) / dup court header (677) / dup cite (124)? (c) FTS surface changes. Raise the strip guard to ~12K for the 22 long-caption cases.

2. Segment-B Westlaw conversion — batches 2–7 DONE (of 48)

Archive-HTML text → authoritative West bound .doc, via the tested receive_westlaw pipeline. 7/48 done, 41 remain (~4,071 opinions). All: 100/100 promoted, 0 rejects, jaccard mean ~0.87 (big multi-party captions pull overall jaccard to ~0.45; bodies match — verify the low outliers each batch). Batches share the date-derived changelog batch westlaw-receive-2026-07-11. Westlaw limits daily downloads — resume tomorrow. NEXT: batch-8 (triage/westlaw-batches/batch-008.tsv). The loop:

  1. cut -f4 triage/westlaw-batches/batch-0NN.tsv | tail -n +2 | paste -sd';' - | pbcopy → hand cites.
  2. User runs Westlaw Find&Print → drops the zip in ~/Downloads (newest *.zip).
  3. Cross-check the batch: cut -f3 batch-0NN.tsv first names vs unzip -l ZIP.
  4. Stage: INC=~/refs/nd/opin/westlaw-incoming/<date>-bNN; mkdir -p "$INC"; unzip -q -o "$ZIP" -d "$INC"; find "$INC" -name '*.zip' -delete (TRAP: a .zip left in the incoming dir gets auto-extracted → doubles the doc count; confirm subdirs: 0, only loose .docs).
  5. Dry-run receive_westlaw --incoming "$INC"; verify any jaccard<~0.60 by opening the .doc (cite+docket+panel).
  6. --apply → invariants (want 29/2/0) + quick_check + FTS parity.

3. Done this session (context; no action)

  • Unpublished dispositions — DECIDED (keep+flag). opinions.precedential_status column (NULL=presumed published, not backfilled; SCHEMA Contract 4). id20421 Richmond (Rule 35.1(a)(6),(7)), id20439 Kudrna (35.1(a)(2)) flagged Unpublished. Policy: a Westlaw reject that's a Rule 35.1 table disposition → keep+flag, don't delete/"correct". scripts/flag_unpublished_dispositions.py. Pre-1997 YYYY ND N cites are SYNTHETIC ([[project_pre1997_neutral_cites_synthetic]]).
  • Segment A diverged set RESOLVED. scripts/segA_ws_confirm.py (substring-tile; never SequenceMatcher on long prose — autojunk fabricates false runs) found many diverged had non-.doc text mislabeled westlaw. scripts/promote_segA_westdoc.py promoted ~232 to West .doc (53 syllabi recovered); post-promotion doc_db_diff 97.4% byte-clean (7336/7532). 8 holds verified: promoted id7180/4863/4829; kept id1031 (bound-PDF-confirmed complete; leaked-# now STRIPPED), id2473+id2125 (merged records), id20479 Larson + id9768 Schirado (different order at shared cite → task below). LESSON: merged-record risk isn't captured by shrink% alone (id2473 lost a companion at 8% shrink) — guard on >1 N.W. parallel / >1 westlaw .doc. Parser note: strip_synopsis_editorial rarely misclassifies a large facts-under-"Synopsis" block (id1031-type); shrink guard catches it; harden before pre-1950 promotes.
  • P59 gate-reject recovery. Recovered 16 genuine fixes the gate rejected on formatting: OCR justice names (DALE Y.→V. SANDSTROM, EAPSNER→KAPSNER, MAKING→MARING, ROBERT 0.→O. WEFALD), statute em-dash→hyphen, AG-cite 0→O glyphs ×7, marker [1112]→[¶12], rendered-page-verified 65-05-83→33, statute 25-03.1-18.1(1)(a).

4. Other open items / queues

  • Fleet P60 (OFFSET 4990, after 2006 ND 210) — modern proofing, PDF method. ~18 fleets to P78.
  • P59 residual reject queue (triage/p59-current.results.reject.json): ~49 genuinely-unverifiable (need manual page reading) + correct-rejects + id14702 parallel-cite strip (belongs to the CL-parallel-cite pass). The recoverable ones are DONE.
  • Larson/Schirado order identity (task): id20479 (460 N.W.2d 716) & id9768 (396 N.W.2d 729) — the West .doc is a different order than the DB; no on-disk N.W.2d bound page to arbitrate. Keep DB; low priority.
  • [[Image here]] (636, TODO-image-reconstruction.md); CL parallel-cite enrichment/strip pass; 62-item markup HOLD (triage/proofing-p1p41-HOLD-for-review.json); full-corpus cite_extract re-run under jetcite 2.5.3; statutes #4 minor; reversion-recovery opinion-BODY re-validation.
  • Deploy v1.3.0 to the live server if/when desired.

5. Durable traps

  • macOS has no timeout — use gtimeout or cap iterations.
  • Never sqlite3.connect() a backup path that may not exist — it silently creates a 0-byte file (clobbered the 194-promote backup). Check os.path.exists first.
  • receive_westlaw auto-extracts any .zip left in the incoming dir → remove it (only loose .docs).
  • Run everything via .venv/bin/python; full ingest re-scan resurrects §6-merged dups (use --incremental).
  • FTS is trigger-maintained (opinions_au) — text_content updates auto-sync; no manual rebuild.
  • Release = scripts/make_release.sh --publish from main (ff-merge the data branch first, bump pyproject version, commit). 740M opinions.db.zip 422-flakes on one-shot upload → create with the 5 small DBs + tag, then gh release upload opinions.db.zip separately (done for v1.3.0).