Skip to content

HANDOFF-2026-07-12.md

Latest commit

 

History

History
95 lines (87 loc) · 7.57 KB

HANDOFF-2026-07-12.md

File metadata and controls

95 lines (87 loc) · 7.57 KB
Raw

HANDOFF — 2026-07-12 (end of session)

Supersedes HANDOFF-2026-07-11.md. Big session: Westlaw segment-B COMPLETE (48/48), fleets P60–P66, the corpus-wide leaked-preamble strip (5,516), the 445 N.W.2d 25 table ingest, and the [[Image here]] figure pipeline built end-to-end (Phase 1+2 — figures.db + MCP tools). Live state also in memory ndcourts-mcp-ongoing-work (2026-07-12 EOD section). Read top-to-bottom.

0. Status at a glance

  • Corpus 19,825; invariants 29 ok / 2 known / 0 regressed; FTS 19,825/19,825; 162 tests; footnote 178/0. NEW: figures.db — 34 figures / 17 opinions (own DB, gitignored, shipped asset).
  • Git: committed + pushed through 8ca60e8 on main (P60–P66, leaked-preamble strip, Westlaw 8–48, 445-table, figure pipeline). Working tree clean. The DB is gitignored (changelog lives in it).
  • RELEASES: v1.3.1 shipped mid-session (P60–P64 + strip + Westlaw 8–17, 6 DB assets). v1.3.2 is being cut at end-of-session to ship figures.db + the two new MCP figure tools (the first release with figure-serving). Release = scripts/make_release.sh --publish from main (bump pyproject, commit first — working tree must be clean; 740M opinions.db.zip may 422-flake, then gh release upload it separately). NOT deployed to the live MCP server (ndconst:/srv/ndcourts via update-db.sh).

1. Westlaw segment-B — COMPLETE ✅

All 48/48 batches converted (archive-HTML → authoritative West bound .doc). ~4,063 promoted today. Reordered by descending DB citation count so the most-cited converted first. Handling of the unpublished tail (durable policy):

  • COA (Table) summary affirmances: DB already has clean archive text → do NOT promote the West (Table)/Unpublished Disposition furniture over it; KEEP + flag precedential_status='Unpublished'. 6 this session (P.C., Schultz, Eisenzimmer, Anderson, City of Minot v. Geiger, Bishop). Exclude the doc by moving it OUT of the incoming tree (receiver recurses into subdirs).
  • 445 N.W.2d 25 is a 4-case shared table page; ingested all 4 per JT (McAdam promoted + Shane/ Selleys/Powell created, ids 20550–52). CREATE path is messy (leaked YAML, furniture in case_name, wrong court, spurious WL cites) — cleaned via fix-445table-create-2026-07-12.
  • Low-sim holds: genuine same-case matches the receiver declined on furniture-driven low overlap (disciplinary orders, short memos); DB kept its valid text — fine.
  • TRAP: Westlaw returns docs in a DIFFERENT order than submitted; the positional name cross-check is advisory only — the receiver's cite-based match is authoritative.
  • Next pre-1997 work = segment C (markdown-primary, ~5%, LLM+human). Segment A (West .doc) is essentially done; segment B now done.

2. Proofing fleets P60–P66 — DONE

~9,000+ transcription fixes / ~1,050 opinions, all PDF-gated. Fleet cursor → P67 (OFFSET 5830, after 2003 ND 34). ~12 fleets to ~P78 finishes the 1997+ era. Notes:

  • Dropped-agent recovery: gen_proofing_workflow.py --ids <id,...>; a hung single agent → TaskStop + fresh single-id run. Pause/resume mid-fleet works via Workflow({scriptPath, resumeFromRunId}).
  • Always PULL [[Image here]] proposals OUT of the apply set (filter '[[Image here]]' in old_exact) → they feed the figure fleet, not per-fleet surgery. 27 collected so far in triage/image-here-deferred-ALL.json.
  • Hand/image-verify high-risk items before apply: statute/rule numbers, dollar amounts, citation years, signature-block reconstructions, person's-middle-initial captions (Jay V. Thomas).

3. [[Image here]] figure pipeline — BUILT (Phase 1+2 DONE); tail items remain

Full detail: TODO-image-reconstruction.md, memory project_image_here_figures. DONE this session:

  • Phase 1 (vision classify, 621 PDF-backed opinions): 1,206 placeholders = 1,182 TEXT / 23 FIGURE / 1 AMBIGUOUS → triage/image-here-classified.jsonl. Applied 1,141 .... ellipsis restorations (scripts/apply_image_here_text.py). Scripts: gen_image_classify_workflow.py.
  • PDF-scan false-negative recovery (scripts/scan_pdf_images.py): the [[Image here]] approach MISSED 7 figure opinions with no placeholder (incl. Boger 2021 ND 152 body-cam stills, 2026 ND 99 aerial/GLO-survey, 2022 ND 211 oil/gas plats) — ~half the figures. Vision-verified via gen_figure_miss_workflow.py. RUN THIS SCAN PERIODICALLY — the recent-scraper era silently drops figure placeholders, so new figures will keep slipping past the [[Image here]] net.
  • Phase 2 (extract + serve): figures.db (34 figures, scripts/build_figures_db.py, ships via make_release floor 20) + MCP tools get_opinion_figures / get_opinion_figure_image (ndcourts_mcp/figures_corpus.py, ATTACH alias fig). 23 [Figure N: caption] markers applied.
  • STILL OPEN (figure tail): 41 held-verbatim placeholders (asterisk omissions * * * * *, roman-numeral headings, ~2 caseload tables mis-called TEXT — apply_image_here_text.py holds them); 16 miss-set figures lack an in-text marker (served by cite anyway); 14 no-PDF [[Image here]] opinions; the 14 vector "table_image" figures are coarse full-page renders (crop to bbox later).

4. Other open items / queues

  • Leaked-preamble strip DONE (5,516 opinions, batch strip-leaked-preamble-2026-07-11). project_leaked_frontmatter_preamble RESOLVED.
  • Cite-graph §4 (NEW) — name-assisted reconciliation for court publication errors (wrong volume/page/year in a cite): resolve the graph edge to the intended case when name+near-cite is unique. Name-clarity is the guard, wide digit variation allowed (TODO-citation-graph.md §4, per JT). Pairs with the full-corpus cite_extract re-run under jetcite ≥2.5.3.
  • HELD items (small, documented): triage/proofing-p60-64-defer-HOLD.json (5), triage/proofing-p65p66-defer-HOLD.json (1: YII→VI) — heading renumbering / l-1 / VI-vs-VII.
  • Per-fleet rejects (correct-rejects) + ~1,400 flags/fleet (CL-parallel-cite + markup) → dedicated passes.
  • 62-item markup HOLD, star-page residue, reversion-recovery BODY re-validation, statutes #4 (all pre-existing).
  • ~90 opinions.db.bak-* backups accumulating — prune old ones.

5. Recommended next-session priorities (pick up here)

  1. If v1.3.2 didn't finish: verify the release (gh release view v1.3.2) has all 7 DB assets (opinions + 5 corpora + figures) and deploy to the live server if desired.
  2. Modern proofing fleets P67 → ~P78 (OFFSET 5830): the biggest remaining routine work; ~12 fleets finish the 1997+ era. Run pairs in background, same verify→consolidate→apply chain.
  3. Figure tail (§3): the 41 held-verbatim placeholders + 16 miss-set markers are quick wins.
  4. Cite-graph §4 name-assisted reconciliation (§4 below) + the full-corpus cite_extract re-run.
  5. Segment C (pre-1997 markdown, ~5%) — the last pre-1997 conversion slice (LLM+human).
  6. Housekeeping: prune the ~95 opinions.db.bak-* backups.

6. Durable traps (this session's additions)

  • receive_westlaw RECURSES into incoming subdirs — to exclude a doc, move it OUT of the tree.
  • A manual promote-revert must fix BOTH opinion_sources AND the denormalized opinions.source_reporter/source_path (else source_*_matches_primary regress).
  • The CREATE path leaves malformed records (YAML frontmatter, furniture in case_name, wrong court, spurious WL cites) — clean before trusting. WL is not in the reporter taxonomy; don't keep them.
  • macOS: no timeout (use gtimeout); never sqlite3.connect() a bak path that may not exist.