Skip to content

TODO-primarylaw.md

Latest commit

 

History

History
116 lines (72 loc) · 23.8 KB

TODO-primarylaw.md

File metadata and controls

116 lines (72 loc) · 23.8 KB
Raw

TODO — Primary-Law Corpora Validation

Validation backlog for the primary-law databases (constitution.db, constitution_history.db, statutes.db, rules.db, admincode.db). Opinions-corpus work lives in TODO-validation.md; const-history ingest notes live in data/const_history_*.md.

Open

PL-1 · Investigate dropped-line extraction bug in ingest_statutes.py and re-validate all of statutes.db

✅ ROOT-CAUSED + FIXED + APPLIED 2026-06-08. Root cause: scrape_nd_code.py _text_to_markdown TOC filter (skips NN-NN-NN + non-period lines) also ate wrapped body lines starting with a section cross-reference. pdfplumber extracted correctly; the converter dropped the line. Fixed by gating the TOC filter to the pre-first-section region (seen_section flag) — ~/code/code-mirror/scrape_nd_code.py (working-tree edit, NOT yet committed). Isolated the fix by diffing old-vs-new re-extraction of all 2,526 chapter PDFs; applied 627 strictly-additive, unique-header, DB==old sections to statutes.db (batch ndcc-dropline-fix-2026-06-08). Seam-detector 600→202 (residual = PL-2 holds + source/FP, verified not TOC drops). Tools: triage/apply_dropline_fix_2026-06-08.py, triage/detect_dropped_xrefs_2026-06-08.py, ~/code/code-mirror/rebuild_ndcc_md.py (parallel re-extractor). DB backup statutes.db.bak-pre-dropline-2026-06-08.

Remaining under PL-1: (a) 16 sections held — they have the dropped-line bug too but are entangled with PL-2 phantom-duplicate headers, so fixing them depends on PL-2: 10-33-68, 21-10-06, 26.1-06.1-34, 30.1-05-05, 39-06.1-10, 39-16.1-11, 40-58-20.1, 41-04-31, 44-04-18, 50-01.1-04, 54-44.4-13, 54-46.1-05, 54-52.1-03.2, 57-15-06.7, 65-01-02, 65-06.2-08. (b) regenerate the ~/refs/statute/NDCC markdown with the fixed converter (the served DB is fixed, but the markdown source still has the bug — a future re-ingest would reintroduce it; the tree also has unrelated uncommitted changes the user should review). (c) commit the scrape_nd_code.py fix in the code-mirror repo.

Original trigger / analysis (kept for the record):

Trigger: N.D.C.C. § 29-15-21(3) was found with a full clause missing — 17 words dropped between "pursuant to section" and "considered a proceeding separate," removing both cross-references (§ 14-05-24, § 14-05-22) and the verb "must be." Fixed under batch ndcc-fix-29-15-21-subsec3-2026-06-08 (see CHANGELOG-data-primarylaw.md); the official text was verified against the ndlegis.gov cencode PDF.

Why it's systematic, not one-off: the missing words — "14-05-24 or an order for child custody pursuant to section 14-05-22 must be" — are exactly one wrapped physical line in the source PDF (https://ndlegis.gov/cencode/t29c15.pdf). The drop sits at a line boundary (section⏎considered), which points at a line-join / wrap-handling defect in PDF text extraction rather than a content-specific fluke. If the extractor drops a line under some repeatable condition (page break, column artifact, specific wrap pattern), there are likely other affected sections among the 29,107 ingested (batch statutes-ingest-2026-06-07, source ~/refs/statute/NDCC, via ingest_statutes.py).

Two tasks:

  1. Root-cause the extraction path in ingest_statutes.py (and whatever PDF-text layer it calls). Reproduce on § 29-15-21, find the exact condition under which the line was dropped, and determine its blast radius. Fix the extractor so a re-ingest is clean.

  2. Re-validate the corpus for similar drops. Candidate detectors (cheap → thorough):

    • Cheap heuristics over current text_content: dangling cross-references ("pursuant to section" / "under section" / "in section" immediately followed by a non-numeric token or newline); sentences with no terminal punctuation before a subsection marker; abnormally short lines mid-sentence; lowercase-after-line-break where a clause looks severed.
    • Authoritative diff: re-extract all sections from the source PDFs with a corrected/independent extractor and diff against DB text_content section-by-section (the same diff-QA method used for the const amendments). This is the definitive check; the heuristics just triage where to look first.

    Apply any corrections as their own changelog batches + CHANGELOG-data-primarylaw.md entries, each verified against the official ndlegis.gov text. Note that this bug is not idempotent-safe to ignore — if the extractor is at fault, the same drops will recur on the next statutes re-ingest until §1 is fixed.

Confidence the bug is systematic: moderate → HIGH. Confirmed systematic by heuristic sweep 2026-06-08.

Heuristic sweep results (2026-06-08): scanned all 29,107 current text_content rows for the tight bug signature — a bare "section"/"subsection" preceded by a cross-reference preposition (of/under/in/to/by/with/or/and/pursuant) and not followed by a digit or spelled number. After stripping the land-description false-positive class (section, township, range): 607 hits across 555 distinct sections. A ~55-context eyeball sample read as >90% true positives — only residual FPs were "section line" (road term) and "part of section" (property-tax land refs).

Refined signature (sharper than the original line-drop guess): the drop almost always begins at a section-number cross-reference and frequently bleeds the referenced section's heading/catchline in where the number should be — e.g. § 12.1-06-01 "under section guilty of committing…", § 11-28-06 "as provided in section construction, equipping, and maintaining…", § 45-20-01 "of section partnership business…", § 54-52.1-03.2 "pursuant to section education who elect…". § 29-15-21(3) is the same bug with a longer span (it swallowed a full line). Working hypothesis: the ndlegis.gov cencode PDFs render section cross-references as hyperlinks/named destinations (#nameddest=…); the extractor drops the link's anchor text (the number) and sometimes adjacent runs, occasionally substituting the destination's heading. Root-cause should focus on hyperlink/annotation handling in the PDF text layer, or switch to a cross-reference-clean source.

Magnitude: ~600 corrupted cross-references across ~555 sections (≈1.9% of the corpus) — each silently drops a statutory cross-reference, the highest-stakes kind of error for an authoritative text. This is NOT a hand-fixable tail; it needs the extractor fix + full re-ingest + re-validation, not per-section patches. (29-15-21 was patched individually only because it was the surfacing instance.)

Detector: the tight-pattern query is the triage tool; save it as triage/ script when §1 begins. The full authoritative diff (re-extract vs DB) remains the definitive check — the heuristic only finds drops that leave an ungrammatical seam, not clean substitutions.

PL-2 · Phantom-duplicate section headers — body cross-references mis-parsed as ### § headers

✅ ROOT-CAUSED + FIXED + APPLIED 2026-06-08. Fix in scrape_nd_code.py _real_header_indices: keep only the longest strictly-increasing subsequence (LIS) of candidate headers after dropping cross-chapter candidates (numeric sort-key prefix, zero-padding-insensitive). Phantoms demote to body text. 0 phantom-duplicate headers corpus-wide (was 168). Applied 258 section corrections (72 headings) via triage/apply_phantom_fix_2026-06-08.py (batch ndcc-phantom-header-fix-2026-06-08); DB now == clean re-extraction for all 29,107 ndcc sections. Subsumed the 16 PL-1 held sections. DB backup statutes.db.bak-pre-phantom-2026-06-08. Validation: integrity ok, FTS ok, 0 dup citations, seam-detector 202→41 (residual all source/FP), near-empty sections all legit dispositions (Superseded/Omitted/Reserved). Remaining: (a) regenerate ~/refs/statute/NDCC markdown with the fixed converter — shared with PL-1(b); until then a re-ingest re-corrupts the DB; (b) commit the scrape_nd_code.py fix (both PL-1 + PL-2 edits) in code-mirror.

Original analysis (kept for the record):

Discovered while isolating PL-1. Same trigger as PL-1 (a section cross-reference wrapping to the start of a line), opposite symptom. scrape_nd_code.py's section detector _NDCC_SECTION_RE = ^([\d.]+(?:-[\d.]+){2,})\.\s+(.+) matches any line beginning NN-NN-NN. Word. A body line that wraps to start with a cross-reference ending a sentence — e.g. …as provided in section 10-19.1-30. Unless reserved by the articles… wrapping so a line begins 10-19.1-30. Unless reserved by the articles… — is mis-parsed as a new section header. (PL-1 is the no-period version of the same wrap; PL-2 is the period version.)

Effect — already corrupts statutes.db (worse than PL-1): the spurious header fragments a section and creates a phantom duplicate of section number NN-NN-NN. The DB ingest's INSERT OR IGNORE keeps whichever occurrence the parse emits first, so for some sections the DB silently holds a garbage fragment instead of the real text:

  • § 65-01-02 (workers-comp "Definitions") — DB has a 208-char fragment (a mis-parsed body line as its heading); the real section is ~42,997 chars.
  • § 14-15-16 — DB text is literally "14-15-16." (9 chars); real section ~14,930 chars.
  • § 29-26-22 — DB 16 chars; real ~3,066. Conversely some sections are correct in the DB but the phantom would win on re-ingest (order-dependent), so this is also why a naive full re-ingest is unsafe.

Scope: 168 sections carry >1 ### § header corpus-wide — 55 cross-chapter (the phantom number doesn't belong to the file it's in — unambiguously a mis-parse) + 113 intra-chapter. This is the footprint of the bug; the count of DB-corrupted sections is a subset (where the phantom sorted first) and must be enumerated against the official text.

Fix approach (harder than PL-1 — mis-parses occur mid-body, so position-gating won't work): make the header detector reject a candidate unless its number (a) belongs to the current chapter (prefix match) AND (b) is greater than the last accepted section in this chapter (monotonic). That rejects both cross-chapter phantoms and backward mid-body refs. Mind sub-section suffixes (.1, .2) and the first real header. Then regenerate + re-validate, and per-section verify each DB-corrupted section against the official ndlegis.gov text before overwriting (some sections the DB is right and the re-extraction is wrong — do not bulk-trust either source). Re-validation detectors: section-length outliers vs neighbors, headings that are sentence fragments / lowercase-start bodies, and the cross-reference resolver (PL-VALIDATE below). The 16 PL-1 held sections get fixed as part of this pass.

PL-3 · Admin code (NDAC) — same two extraction bugs

✅ SCANNED + FIXED + APPLIED 2026-06-08. The admin code was extracted by the same converter, so it carried both PL-1 (dropped lines) and PL-2 (phantom-duplicate headers) bugs: scan found 117 phantom-dup section numbers + 287 dropped-cross-ref seams; admincode.db had 0 dup citations (first-wins hid the corruption). The committed converter fix transfers to NDAC unchanged (confirmed incl. the 1,169-section hazardous-waste chapter 33.1-24-05 → 0 phantom dups). Re-extracted all 2,131 NDAC PDFs (~/code/code-mirror/rebuild_ndac_md.py, parallel), compared vs admincode.db, applied 646 corrections (59 headings; batch ndac-extraction-fix-2026-06-08, triage/apply_ndac_fix_2026-06-08.py). DB == clean re-extraction for every non-held section; integrity ok, FTS ok, 0 dup citations. DB backup admincode.db.bak-pre-ndacfix-2026-06-08.

✅ Manual remainder DONE 2026-06-08:

  • 5 held sections reconstructed from the official chapter PDFs (real header → next header, converter line-filters) and applied — batch ndac-held-tablesections-2026-06-08 (33.1-24-02-42 "Wording of the instruments", 33.1-24-05-235, 33.1-24-05-280, 33.1-24-06-14, 33.1-24-08-64). Each ends at its official History/Law-Implemented footer.
  • 1 spurious section 33.1-24-06-235 deleted — batch ndac-delete-spurious-2026-06-08. Admin provisions now 13,837.
  • Refs ~/refs/reg/NDAC markdown regenerated + re-indexed (13,837 sections); rebuild_ndac_md.py committed (code-mirror 675db14).

Still open — the third extraction weakness (low priority): the fixed converter still mis-splits sections inside giant multi-page embedded tables (rows look like section bodies / cross-refs become phantom headers). It affects ~5 sections, all in the hazardous-waste chapters (33.1-24-*), now hand-corrected in the DB — but the served ~/refs/reg/NDAC markdown for those 5 stays wrong until the converter learns to treat a detected "Treatment Standards"/instrument-wording table as opaque section body. Fix only if table-heavy NDAC chapters become important; otherwise the DB is authoritative and the divergence is documented.

DOCUMENTED LIMITATION — multi-column table reading-order scramble (decided 2026-06-13: leave as-is, do not fix): Beyond the header mis-split above, the RCRA comparison tables in ch 33.1-24 are laid out as 3–4 narrow columns in the official ndlegis.gov PDF. Linearizing them interleaves the columns line-by-line, so a cell's text is split and re-joined across the gutter — e.g. § 33.1-24-05-234/-235 render ...subsection 2 of section provisions of sections / 33.1-24-06-235. 33.1-24-05-250 through 33.1-24-05-299. This is inherent to PDF text extraction and identical in both poppler pdftotext and our pdfplumber (verified 2026-06-13), so it is NOT a pipeline defect. Marker OCR (marker_single, layout model, no --use_llm) recovers correct cell reading order (the cross-references come out clean and correctly associated) but flattens the row→column grid to paragraphs and is not worth running for a cosmetic, documented issue. The cross-references in these cells were independently verified faithful to the official source (PL-VALIDATE #1, 2026-06-13). Confirmed-affected sections (multi-column tables; text reads scrambled though all content is present): 33.1-24-05-234, -235, -280, -290, -400, -427, -713, -733, plus the earlier hand-corrected 33.1-24-02-42, 33.1-24-06-14, 33.1-24-08-64. Not exhaustive — any multi-column table in 33.1-24-* is subject to the same scramble; full enumeration would need a table-detection pass. The real fix is upstream source files, not better PDF heuristics — see PL-SOURCE-FILES.

PL-SOURCE-FILES · Request native source files (XML / ODT / word-processor) from Legislative Council

Open (proposed 2026-06-13). Every extraction bug in this corpus — PL-1 dropped lines, PL-2 phantom headers, PL-3 table scramble — is an artifact of reverse-engineering structure out of typeset PDFs. The N.D. Legislative Council Legal Division composes the Century Code and Administrative Code in a structured upstream format (the published PDFs are generated from it). Requesting those native files (legislative XML, ODT, or the word-processor source) would let us ingest structure directly — sections, headings, table cells, cross-reference anchors — and retire the PDF-extraction heuristics and their entire bug class. Action: contact the Legislative Council (Legal Division / Code Revisor) to ask what machine-readable source formats they can share for the N.D.C.C. and N.D.A.C. (and whether a bulk/periodic feed is available for the weekly refresh). This also bears on the NDAC-staleness finding (PL-VALIDATE #1: § 20-03-01-02 ingested 2026-06-07 is already one amendment behind the live PDF) — a structured feed would make freshness tractable. Pairs with the code-revisor error memo drafted 2026-06-13 (memos/2026-06-13-code-revisor-crossref-errors.md).

PL-VALIDATE · Broader text-faithfulness validators (proposed 2026-06-08)

Beyond the two known bugs, to certify the corpus as authoritative (and reusable for the admin-code + constitution corpora). Ranked by which blind spot they close — the seam-detector only catches drops that leave an ungrammatical seam; clean substitutions and whole-section loss need these:

  1. ✅ Cross-reference resolver — BUILT + RUN 2026-06-13. triage/xref_resolver_2026-06-13.py (read-only). Parses every internal section/chapter/article/title NN-NN-NN reference (+ list continuations "X, Y, and Z" / "X through Y"), normalizes zero-padding (01-03-19→1-3-19), scopes by corpus, checks the sibling corpus before flagging, guards USC/UCC shapes, ranks unresolved by suspicion. Parsed ~22,785 NDCC + ~17,847 NDAC references; 99%+ resolve. NDAC's 4,256 CROSS_CORPUS are rules citing enabling NDCC statutes (correct). The 19 highest-suspicion hits (UNRESOLVED_CHAPTER_EXISTS: real chapter, missing section) were verified against official ndlegis.gov PDFs via poppler pdftotext (a second engine — doubles as a spot-check of PL-VALIDATE #2):
    • ALL 19 are faithful to the official source — ZERO extraction garbles introduced by our pipeline. This independently confirms our ingest did not corrupt cross-references (the complementary check to the seam-detector).
    • 18 are official-source danglers: drafting errors / typos / refs to non-adopted uniform-act sections, present verbatim in the official code (e.g. NDCC 28-22-03.1 officially prints "39-03-14.2" but the real section is "39-03.1-14.2", confirmed in ch 39-03.1; NDCC 32-29.3-04 & 54-44.8-04 cite sections past their chapter's end; the seven NDAC 33.1-24-05-* hits are RCRA/40-CFR-adoption table artifacts — the "subsection 2 of section provisions of sections 33.1-24-06-235" scramble appears identically in the official PDF, ch -04 stops at -08 / ch -06 at -100). Per the preserve-source-typos policy, leave verbatim.
    • 1 is a version-currency lag, not a garble: NDAC 20-03-01-02 (our 2026-06-07 ingest) cites "20-02-01-05.1" in a subsection 12 that the official current PDF has since amended to drop the cross-reference. → flags that the NDAC corpus may be one amendment stale; worth a refresh pass / freshness check on the whole NDAC ingest.
    • The lower-suspicion buckets (~370 NO_PARENT/CONTAINER) are dominated by the legislature's own preserved references to former/repealed law ("former chapter 15-39", "as they existed on July 31, 1997"). Output TSVs: triage/xref-unresolved-{ndcc,ndac}-2026-06-13.tsv.
    • Optional follow-up: a "known official danglers" registry (à la print_anomalies) so future resolver runs don't re-surface the 18 verified-official danglers.
  2. Second-engine extraction diff — re-extract with poppler pdftotext -layout / PyMuPDF / mutool and diff per section vs pdfplumber. Only check that certifies faithfulness to the PDF (PL-1's old-vs-new was within-engine). Capstone.
  3. Section-inventory cross-check — diff the DB section list + headings against the PDF bookmarks/named-destinations and the ndlegis.gov index pages. Catches whole missing sections (no body heuristic can) and corrupted headings. Directly relevant to PL-2.
  4. Enumeration-continuity — flag gaps in subsection 1,2,3… / subdivision a,b,c… sequences (a dropped enumerated block reads grammatically). Cheap, no external source.
  5. Doubled-token detector — flag "of section section …", "under section under section …" (a distinct duplication artifact already observed). Near-zero false positives.
  6. Heading/body split sanity — the converter splits heading at the first period, so a catchline with "U.S."/"Inc."/"No." splits wrong; flag headings ending mid-phrase or bodies starting lowercase.
  7. Character/format histogram — mojibake, failed ligatures, stray form-feeds, anomalous whitespace.

PL-CONST-MERGE-B · DONE 2026-06-08 — historical layer merged into served constitution.db

scripts/merge_const_history.py --apply folded constitution_history.db (265 provisions / 421 versions, 1889 numbering, eff. 1889–1980) into the served constitution.db (now 466 provisions / 622 versions). Cite-disjoint from the modern art/§ provisions (0 collisions). lookup_authority("N.D. Const. § 82", as_of_date="1945-01-01") now returns the historical point-in-time text; modern queries unchanged; FTS indexes the historical text. This is Approach B (additive, original-numbering). The items below are the remaining integration work.

⚠️ BUILD-ORDER NOTE (important): constitution.db is a regenerable artifact and the merge is NOT part of ingest_constitution.py. The correct build sequence on the build machine is:

  1. python -m ndcourts_mcp.ingest_constitution --apply (rebuild modern, from ndconst.org)
  2. python -m ndcourts_mcp.ingest_constitution_history --apply (rebuild historical layer; rm -f constitution_history.db first — not idempotent)
  3. python scripts/merge_const_history.py --apply (fold history into constitution.db)

then scripts/make_release.sh. If you rebuild constitution.db (step 1) without re-running step 3, the served DB loses the historical layer. The merge refuses to run twice (guards on the 1889 scheme), so re-running after a fresh step 1 is safe. Corrections to historical text go upstream into constitution_amendments.json → re-run steps 2+3, not edited directly in constitution.db.

PL-CONST-CROSSWALK · 1889↔modern renumbering crosswalk (Approach A — the real integration)

After PL-CONST-MERGE-B, the two citation spaces are served but NOT connected: a modern cite for a pre-1981 date returns nothing, a 1889 cite for a post-1981 date returns nothing, and historical versions end at the 1980-12-31 reorg cap. To give continuous point-in-time history across the 1981 reorganization from either citation, build a crosswalk mapping each original §1–§217 + Schedule provision → its modern article/§ (≈217 mappings; some split/merge/repealed).

  • Source the mapping from the 1981 reorganization "disposition tables" / the 1972 Constitutional Convention + 1979–80 revision materials (some Blue Books print old↔new comparison tables). This is the load-bearing artifact.
  • Then either (A1) carry both citations on one provision's version timeline (old cite pre-1981, modern cite 1981+), or (A2) keep separate provisions linked by a successor/lineage table (old §82 → modern art. V §2). A2 is the smaller schema change.
  • Wire the crosswalk into merge_const_history.py (or a successor step) so the merged historical versions attach to / link from their modern counterparts.

PL-CONST-AMEND-RECONCILE · dedup amendment events across the two sources

constitution.db already carried amendment events from ndconst.org annotations (keyed to modern provisions); the merged historical layer carries the session-law text of each pre-1981 version plus its own amendments rows (keyed to 1889 provisions). The same amendment (e.g. the 1940 §82 / Art. 57 change) is now represented from both sources. Once the crosswalk exists, reconcile/dedup the amendments rows so get_authority_history shows one coherent chronology per provision rather than two parallel ones.

PL-CONST-STRUCTURAL · 4 structural pending amendments (still unserved)

constitution_history.db still has 3 status:"pending" amendments + 1 deferred change that the whole-provision ingest can't apply (so they're absent from the merged served DB too): LXXVIII (amends only subdivision (d) of subsection 6 of amend. art. LIV), XCVI (subsections 2 & 4 of art. LIV), XC's 2nd change (subsection 1 of art. LIV), and LXXXI (partial repeal of only the 10th paragraph of §25). amend. art. LIV is stored as ONE provision; these need either the full enclosing-provision text spliced per amendment or art. LIV subdivided into subsection provisions, plus partial-repeal modeling for LXXXI (note §25 was wholly repealed by CV in 1979, so LXXXI governs only the 1964–1979 window). Verbatim text already captured in data/const_amend_session_law_extraction_2026-06-08.md. Apply upstream (ingest_constitution_history), then re-run steps 2+3 of the build sequence above.