Skip to content

Latest commit

 

History

History
358 lines (304 loc) · 40.9 KB

File metadata and controls

358 lines (304 loc) · 40.9 KB

P02 · Module map — Document and contract intelligence for regulated operations

Every module is a lab a student does: a notebook they run and fill in, or a C exercise they compile. Every module declares a compute tier and is held to it by tools/execute.py.

Environment constraint, stated once. The whole programme runs on Python 3.12 with numpy, matplotlib, tokenizers and pytest, plus clang and make. There is no PDF library, no OCR engine and no model API on any required path, so every corpus is generated deterministically inside its lesson from a fixed seed. The labs below are designed within that, not around it.

11 of 11 modules built. A module is built when its lesson directory exists and has passed all 14 gates in QUALITY.md and an independent review; everything else is specified here and has no directory. Seconds and peak memory are measured by tools/execute.py; rubric size is read from each lesson's tests/test_lesson.py. Regenerate with python tools/status.py.

  1. Field extraction you can measure · built · ~55 minutes · Python
    Open Field extraction you can measure in Colab Open Field extraction you can measure in Kaggle Open Field extraction you can measure in Binder
    measured 0.2 s · 30 MiB · rubric 90 points in 24 cases
  2. Ingestion and layout without a PDF library · built · ~60 minutes · Python
    Open Ingestion and layout without a PDF library in Colab Open Ingestion and layout without a PDF library in Kaggle Open Ingestion and layout without a PDF library in Binder
    measured 0.2 s · 31 MiB · rubric 94 points in 23 cases
  3. From rules to a tagger · built · ~60 minutes · Python
    Open From rules to a tagger in Colab Open From rules to a tagger in Kaggle Open From rules to a tagger in Binder
    measured 0.8 s · 35 MiB · rubric 90 points in 20 cases
  4. Table extraction: structure and content are two different scores · built · ~70 minutes · Python
    Open Table extraction: structure and content are two different scores in Colab Open Table extraction: structure and content are two different scores in Kaggle Open Table extraction: structure and content are two different scores in Binder
    measured 7.2 s · 30 MiB · rubric 94 points in 21 cases
  5. Contract clause classification and calibrated abstention · built · ~80 minutes · Python
    Open Contract clause classification and calibrated abstention in Colab Open Contract clause classification and calibrated abstention in Kaggle Open Contract clause classification and calibrated abstention in Binder
    measured 2.1 s · 196 MiB · rubric 100 points in 29 cases
  6. Human-in-the-loop review queues that survive contact with humans · built · ~70 minutes · Python
    Open Human-in-the-loop review queues that survive contact with humans in Colab Open Human-in-the-loop review queues that survive contact with humans in Kaggle Open Human-in-the-loop review queues that survive contact with humans in Binder
    measured 2.9 s · 31 MiB · rubric 90 points in 26 cases
  7. Error analysis and slice discovery · built · ~75 minutes · Python
    Open Error analysis and slice discovery in Colab Open Error analysis and slice discovery in Kaggle Open Error analysis and slice discovery in Binder
    measured 10.4 s · 211 MiB · rubric 92 points in 26 cases
  8. Drift monitoring for a live document pipeline · built · ~70 minutes · Python
    Open Drift monitoring for a live document pipeline in Colab Open Drift monitoring for a live document pipeline in Kaggle Open Drift monitoring for a live document pipeline in Binder
    measured 8.8 s · 128 MiB · rubric 96 points in 27 cases
  9. Cost per document, and the routing budget that sets it · built · ~75 minutes · Python
    Open Cost per document, and the routing budget that sets it in Colab Open Cost per document, and the routing budget that sets it in Kaggle Open Cost per document, and the routing budget that sets it in Binder
    measured 0.8 s · 83 MiB · rubric 100 points in 30 cases
  10. The scanner in C · built · ~90 minutes · C
    Open The scanner in C in Colab Open The scanner in C in Kaggle Open The scanner in C in Binder
    measured 4.5 s · 65 MiB · rubric 39 points in 12 cases
  11. Capstone: the conformity pack · built · ~100 minutes · Python
    Open Capstone: the conformity pack in Colab Open Capstone: the conformity pack in Kaggle Open Capstone: the conformity pack in Binder
    measured 5.7 s · 29 MiB · rubric 149 points in 37 cases

Module 1 — Field extraction you can measure: build the evaluation harness before the extractor

lessons/P02-L01-extraction-evaluation/ · tier cpu8 · prerequisites T00-L01-the-8gb-track

The lab. The student is handed 180 short synthetic remittance advices with gold labels, generated inside the lesson from a fixed seed, and a deliberately mediocre stand-in extractor with a seeded, documented error model — it reformats dates and amounts into its own house style, drops fields, invents payment terms that were never printed, smudges a supplier name the way a scan does, and gets a digit wrong in an amount, an invoice number or a payment term. They then implement, in pure numpy and stdlib:

  1. normalise_value — a per-field-type normalisation policy (money, date, id, integer, text), including the trap that a comma is a thousands separator in one convention and a decimal point in another: a lone comma is decimal only when one or two digits follow it and end the number.
  2. match_value — exact, normalised and fuzzy matching, with fuzzy confined to text fields and its threshold inclusive. A later cell counts how many genuinely wrong amounts a fuzzy money matcher would wave through on this corpus.
  3. score_field and macro_f1 — per-field precision, recall and F1, where a wrong value counts as both a false positive and a false negative, and an unweighted macro average that refuses to let the field on every page bury the rare one.
  4. classify_cell and confusion_by_field_type — five mutually exclusive labels per cell (correct, miss, spurious, wrong value, true negative), aggregated by field type under the matching mode it is given, so one F1 number becomes a work order.
  5. review_queue and apply_reviews — a confidence-ordered routing policy under a fixed budget in which every cell is a candidate, a confident silence included, ties break on (doc_id, field), and corrections land in new records rather than the ones being scored.

They finish by sweeping the budget and reading a quality/cost table: macro F1 against a random routing control, cost per document from placeholder rates they are told to replace, and the marginal F1 bought per unit of cost. On this corpus the yield rises then falls, because the lowest-confidence cells are all misses and the next band brings in wrong values — and fixing a wrong value is worth strictly more than fixing a miss. That is measured in the notebook, not asserted.

Why this is module 1 rather than module 3. A May 2026 study of structured contract extraction found its domain-trained model weakest on currency fields requiring normalisation, and scored its models with no currency parsing at all (both are sourced in the lesson's claims.yaml). A student who has not written the normaliser cannot tell a formatting difference from a defect, and will spend the rest of the programme filing bugs against models that were already right.


Module 2 — Ingestion and layout without a PDF library

lessons/P02-L02-layout-reading-order/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: reading order from geometry. The student is given synthetic pages, laid out inside the lesson from a fixed seed, as token boxes (text, x, y, w, h, page) covering single-column, two-column and table-bearing layouts. They implement recursive XY-cut segmentation (find_gaps, xy_cut), a lines-first sort within each block (order_within_block) and a header/footer detector that masks digits (detect_header_footer); then score their reading order against the gold sequence with Kendall's tau and a block-level boundary F1 (kendall_tau, boundary_f1), and must show a setting where the two disagree: a page shredded into blocks keeps a perfect tau. They finish by measuring, with module 1's scorer reproduced inside the lesson, how much field-extraction F1 moves when reading order is wrong — naive raster order scores 0.745 against 1.000 for gold order — which is the argument for caring about layout at all.

Module 3 — From rules to a tagger: sequence labelling over document tokens

lessons/P02-L03-sequence-labelling/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation, P02-L02-layout-reading-order

Lab: build an averaged structured perceptron in numpy over BIO tags on a document token stream by implementing its parts: the features (word_shape, token_features — shape, affix, neighbouring token, line head, line position and one head|shape conjunction), decode_spans, sequence_scores, viterbi_decode and perceptron_update, whose transition half is graded because one wrong tag breaks two bigrams. The training loop and the weight averaging are given. Train it, then score it on held-out suppliers with module 1's harness — same scorer, different extractor, which is the whole point — and compare it field by field with a given rule-based baseline (improvement_table, fields_improved) to identify which field types the tagger actually improved; here it still loses to the rule on counterparty. Two deliberate departures: the token stream is regenerated inside the lesson from a fixed seed rather than imported from module 2, since a student bundle is one lesson directory, and the weights are averaged once per training document rather than once per update. One unplanned result is kept in the open: greedy decoding beats Viterbi on held-out macro F1, because the learned transitions cut the longer held-out supplier names short.

Module 4 — Table extraction: structure and content are two different scores

lessons/P02-L04-table-extraction/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation, P02-L02-layout-reading-order, P02-L03-sequence-labelling

Lab: on synthetic two-page tables laid out inside the lesson from a fixed seed rather than imported from module 2, reconstruct rows and columns with one projection profile (ink_runs) on both axes — row_bands on y, and columns on x read from body ink only, because a spanning header's continuous ink hides the gaps beneath it. cells_in_band must cluster a band into cells before assigning columns, and merge_continuations stitches a row split across a page break, matching cells on (col, colspan). Then two scorers, content_f1 over the multiset of cell texts and teds_like (built on row_distance, with a text-blind TEDS-Struct variant), and a case where content is perfect while structure is wrong — every body row cut in two scores content F1 1.000, TEDS-Struct 0.367 — and a case where the reverse is true. Ends with a rule: name the consumer, and gate on the smallest set of scores that catches the defects that would break it. teds_like is deliberately not TEDS — a two-level alignment over rows and cells, no rowspan, clamped at zero — and the lesson says where the resemblance ends.

Module 5 — Contract clause classification and calibrated abstention

lessons/P02-L05-clause-classification-abstention/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: 6,000 synthetic clauses of six types (governing law, limitation of liability, termination for convenience, assignment, indemnity, change of control), some of them hybrid, smudged or truncated, deduplicated as one pool before being split into training, calibration, test and a fourth "next quarter" split, so no clause appears twice. The byte-pair-encoding vocabulary is trained with tokenizers on the training split by given code, since filling in a BpeTrainer call is API lookup rather than a skill this module teaches, and the lesson trains it twice to show that it is byte-identical. The student implements the featuriser (bag_of_subwords), a multinomial logistic regression in numpy trained by full-batch gradient descent to a stated stopping rule (softmax, loss_and_gradient, train_softmax_regression), per-clause precision, recall and F1 with abstentions counted as false negatives (score_classes, macro_f1), temperature scaling and expected calibration error (fit_temperature, expected_calibration_error), the coverage/accuracy curve and an abstention threshold (coverage_accuracy_curve, choose_threshold, predict_or_abstain), and the policy with its hand-off (fit_policy, to_review_records). The temperature and the threshold may come only from the calibration split, and that ordering is graded directly: replacing the training and test splits must leave the fitted policy unchanged. Abstained clauses go to module 1's review queue, carried into the lesson under the same names and record format, with their real calibrated confidence.

Two results are printed rather than assumed. The fitted temperature comes out below 1, and the student is asked what that means instead of being handed the familiar line that models are overconfident. And temperature scaling leaves every clause's predicted type unchanged but does reorder confidences between clauses; the lesson counts the reversed pairs. The module warns that even an honestly fitted threshold is a point estimate chosen to just meet its accuracy target: over 100 reshuffles of the held-out clauses it meets the target on unseen clauses only about as often as a coin flip. It names selective classification with a guaranteed risk as the fix this lesson does not implement.

Module 6 — Human-in-the-loop review queues that survive contact with humans

lessons/P02-L06-review-queues-real-reviewers/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: module 1's reviewer was perfect; these are not. Module 1's 180 documents and 1,080 (document, field) cells are re-created record for record from its own seeds, and three simulated reviewers, two at tier 1 and one senior, label them with disagreement and with fatigue that grows through a shift. The student implements raw agreement and Cohen's kappa over the union of two reviewers' labels, including a label one of them never uses (agreement_table, raw_agreement, chance_agreement, cohens_kappa), an SLA-aware queue that serves arrived work earliest deadline first and reports its breaches (sla_schedule, breach_report), a two-tier escalation rule (escalate), and the quality ceiling the harness can see (harness_ceiling). On the day's 240 routed cells, earliest deadline first breaches no SLA, against 26 for first-in-first-out and 59 for module 1's confidence order. Averaged over 20 simulated days, the rota chosen against a 0.05 breach-rate target is 60-item shifts: 35.6 items an hour, no breaches, 7.5 routed cells still wrong a day, and macro F1 0.962 against the truth where module 1's perfect reviewer gave 0.967. The rota is chosen on cells still wrong, not on corpus macro F1, which moves by 0.007 across the sweep while cells still wrong roughly triple.

One departure from the original spec: the ceiling is not the reviewers' agreement itself. Their kappa is 0.561; a perfect system scored against gold one of them wrote reaches about sqrt(kappa), 0.749, against a measured 0.737 between the simulation's truth and reviewer A, and a perfect extractor scored against A's gold reaches macro F1 0.976, not 1.000. The lesson grades that form, states its assumption of independent mistakes, and shows the ceiling rising to 0.935 when the second reviewer shares A's mistakes. Exercise 3's hint warns about the readiness test that makes the scheduler loop forever, since the grader has no per-test timeout.

Module 7 — Error analysis and slice discovery: is that regression real?

lessons/P02-L07-slice-discovery/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: 1,600 synthetic documents carrying module 1's six fields and five attributes (vendor, layout family, language, page count, scan quality), with two given extractor versions scored by module 1's harness, carried into the lesson. The student implements slice discovery over single attributes and two-attribute crosses above a minimum size (find_slices), module 1's macro F1 computed from per-document counts for every replicate at once (macro_f1_from_counts), a paired bootstrap over documents for the change in a slice's macro F1, with a percentile interval and a one-sided p-value that is never zero (paired_bootstrap), Holm's step-down and Benjamini-Hochberg's step-up adjustments (holm_adjust, bh_adjust), the sample size a slice needs before it may block a release (required_slice_size), and the release decision (release_decision), in which a slice below that size is "too small to say" whatever its p-value. On this release only the fax slice blocks; the size gate is what stops a 13-document vendor slice (change −0.120, Holm-adjusted p 0.033) from blocking as well.

The regression is planted in one place, digit fields on faxed documents, and every slice inherits a diluted share of it. The lesson re-runs the whole pipeline on 20 null releases, one retrain of v1 against another, and counts how often each rule blocks when nothing changed: any drop 20 times, a drop beyond the margin 15, an uncorrected p-value 14, and Holm, with or without the size gate, never. It also discloses that the seeds were picked so that this one release shows every failure the lesson teaches, the Holm-flagged small slice among them. Depth-2 crosses are implemented and graded but not used for the gate: 253 slices raise the required size from 162 to 214 documents and put the smallest Holm-adjusted p-value that 2,000 bootstrap replicates can produce at 0.1264, above 0.05, so a search that deep could not block anything.

Module 8 — Drift monitoring for a live document pipeline

lessons/P02-L08-pipeline-drift-monitor/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: a synthetic replay year of 364 days, 200 documents a day and a random audit of 25, carrying a vendor template change, a seasonal mix shift and one real model regression. The student implements the population stability index and KL divergence over field-value and confidence distributions, on edges cut once on a baseline quarter and floored on both sides so that KL(a‖e) + KL(e‖a) = PSI exactly (population_stability_index, kl_divergence, fit_edges, drift_scores); the audited defect rate as a daily quality proxy, from module 1's miss, spurious and wrong-value labels over reviewed cells only (daily_defect_rate); an upper one-sided CUSUM with reset (cusum); the most sensitive threshold whose false-page rate on 200 simulated no-change years stays within a budget of one false page a quarter (false_pages_per_quarter, choose_threshold); and a rule that names a change by which distribution moved and whether the audited quality moved (triage). The chosen threshold is printed beside the handbook's rules of thumb, h = 4 and 5, with the false pages each costs and the documents each lets through before it pages.

Whether the monitor paged for an event is decided by counterfactual, not by calendar: the year is generated so that a copy with one event switched off differs from the log only by that event. On this replay the regression pages the day after it goes live, neither harmless change causes a page, and there are 0 unexplained pages in 105 event-free monitored days where the budget allowed about 1.2; paging on PSI above 0.25 does the reverse, paging for both harmless changes and never for the regression. Two warnings close the lab. The no-change years are resampled from the baseline quarter, because a real pipeline has no generator to draw from, so their tail is thinner than reality's. And a rolling baseline erases the template change within four weekly reports (confidence PSI from 0.3769 to 0.0053), which makes it invisible, not resolved.

Module 9 — Cost per document, and the routing budget that sets it

lessons/P02-L09-cost-routing-budget/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation, P02-L03-sequence-labelling, P02-L06-review-queues-real-reviewers, P02-L07-slice-discovery

Lab: a three-tier router (cheap extractor, expensive extractor, human review) priced from named placeholders: module 1's machine cost and reviewer rate, module 6's reviewing pace, and a per-page price for the expensive tier. It routes an eight-segment traffic fixture whose per-tier accuracies are shaped on what modules 1–4 and 6–8 measured, each segment labelled with the module it mirrors. That is the departure from the spec: a notebook runs alone and cannot import another lesson's results, so these are fixtures, not the student's own measurements. The student implements cost per document and field accuracy of a routing plan as volume-weighted means over segments (evaluate_plan), an exhaustive sweep and the most accurate plan within a cap (sweep_plans, best_within_budget), the Pareto frontier as a staircase rather than a convex hull (pareto_frontier), a greedy marginal-yield allocator that records its path (greedy_allocation), a segment mix on which the sweep beats greedy, graded against reference implementations so that it must be a genuine counterexample (greedy_counterexample), and the figures of a recommendation paragraph whose wording is given and adds no number of its own (recommend).

Of 6,561 plans, 736 fit the placeholder cap of 0.30 per document. The sweep's plan costs 0.2920 per document; greedy, held to the same cap, leaves 2,436 more fields wrong a month; and the next step up the 48-plan frontier costs 1.74 per wrong field avoided. Greedy matches the sweep at 213 of 241 caps, at all 15 of its own breakpoints and on 130 of 150 random six-segment mixes, so the module does not call it unreliable: it measures where greedy is exact and has the student construct a case where it is not. Every price is a placeholder the student is told to replace before anyone signs, and swapping module 6's reviewing pace for module 1's placeholder alone moves one of the eight segments to another tier.

Module 10 — The scanner in C: fixed memory over an unbounded export

lessons/P02-L10-streaming-scanner-in-c/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation

Lab: implement, in lesson.c, a streaming field scanner in constant memory over a synthetic export the program generates from a fixed seed: field_push (a fixed buffer, overflow counted per field, not per byte), field_complete, scan_chunk (quoted fields, escaped quotes and embedded newlines, with counters that must not depend on where the stream was split — graded down to one-byte chunks) and scan_finish (a truncated final record versus a legal one with no trailing newline). It is built with make and graded against a Python oracle on identical bytes, and on peak RSS staying flat as the export grows. They measure throughput and peak RSS against that Python scanner and learn where each factor comes from: C buys the throughput, the chunk loop the memory. Two departures: the Python scanner is the one given in this lesson, since module 1 has no scanner, and the export stops short of multiple gigabytes, because the flat memory line is already unambiguous. Its common-mistakes section tells students to run make clean after moving or rebuilding their checkout, as QUALITY.md requires.

Module 11 — Capstone: the conformity pack

lessons/P02-L11-capstone-conformity-pack/ · tier cpu8 · prerequisites T00-L01-the-8gb-track, P02-L01-extraction-evaluation, P02-L02-layout-reading-order, P02-L03-sequence-labelling, P02-L04-table-extraction, P02-L05-clause-classification-abstention, P02-L06-review-queues-real-reviewers, P02-L07-slice-discovery, P02-L08-pipeline-drift-monitor, P02-L09-cost-routing-budget, P02-L10-streaming-scanner-in-c

Lab: the harness, the residual-by-class analysis, the drift alarm, the escalation policy and the cost model from the earlier modules are carried in, given, and assembled into one evidence pack for a synthetic pipeline of 240 documents; the seven graded exercises are the validator that checks it. The student implements a reproduction gate that extracts every figure from the pack's text and requires each one to reproduce, within the precision the text itself claims, from a computed evidence bundle (extract_figures, computed_values, trace_gate); three provenance checks: the residual-by-class table recomputed from the raw records on the split the pack claims (check_residual_provenance), every cost figure presented as measured cited to a matching entry in a measurement registry (check_cost_provenance), and an alarm threshold calibrated strictly before the day it is credited with catching (check_threshold_timing); and a validator that refuses the pack unless all four pass, graded as an AND for every check rather than a majority (validate_pack). Module 10's C scanner is a prerequisite by programme order but is not carried into the pack.

Four defects are planted, each built to be caught by exactly one check: a residual table computed on the training split and labelled as the evaluation split, a derived cost figure labelled measured with no source, a threshold calibrated through day 258 for an alarm on day 253, and a typed commentary figure that nothing computed. The lesson plants each alone in a clean pack and prints which check catches it. The reproduction gate alone catches one of the four, because the other three figures are correct for the data the preparer actually used and trace against the preparer's own evidence bundle. The module warns that repairs which only make a validator quieter (widening a tolerance, loosening the timing comparison by one step, dropping a check from the combination) leave the pack no more honest, and the rubric fails each of them.


Synapsa Commons · © 2026 RealAI · licensed under CC BY-NC-SA 4.0. You may use this material to learn and to teach, and copy, fork, share and adapt it. You must credit "Synapsa Commons by RealAI" with a link to https://github.com/TarrySingh/Artificial-Intelligence-Deep-Learning-Machine-Learning-Tutorials, say what you changed, and share adaptations under the same licence. You may not use it, or anything adapted from it, for commercial advantage or payment; for a commercial licence, contact RealAI. The full summary is in NOTICE; the licence's legal code governs.