[add] calibration of the per-line categorisation logic

[david-spacil]

Refs #3

Motivation

Calibration of the per-line categorisation, measured against manual annotation of Czech archaeological reports.

Two failure modes dominated the disagreements, and both punish text the OCR had actually read correctly:

1. Short lines were routed on language confidence. On a 1–2 word line the language identifier has almost nothing to work with, yet its low confidence decided the outcome. Archival records consist largely of such lines — sigla, museum shorthand, units, figure and table references — so correctly read catalogue data was being marked as damaged: Mzm. and inventory codes like A059/2020-107-008 go to Trash, and Obr.3, t. III, š. 12,5 to Noisy, which asserts they need correcting when nothing is wrong with them.

2. Reference notation was scored as if it were prose. compute_valid_ratio() judged every token as a candidate word, so dates, cites, initials and abbreviations (24.2.2020, /1933/32, I.L., s.o.) counted as invalid words and dragged clean catalogue lines down. On the random sample this is by far the dominant effect: 74 of the 95 disagreements are lines the shipped engine calls Noisy and this PR calls Clear, and the annotator agreed with Clear on 61 of them.

Conversely, a single badly-read token in an otherwise clean sentence did not move the line averages enough to matter, so the line came out Clear — asserting that no correction is needed. Both of these, from the random sample, are Clear on test and Noisy here, and the annotator called both Noisy:

zeminou @. Výplň spodní části objektu tvořena smíšenou vrstvou tmavé
Str*ep z hnědé nádoby s válcovitým, na vnitřní straně

The change makes short lines decide on evidence of damage rather than on language confidence, excludes unjudgeable reference tokens from the word ratio, and adds a character-level damage detector that caps a damaged line at Noisy without ever pushing it to Trash.

Description of change

One file (text_util.py, +309/−5), six commits, each independently reviewable:

commit	what
[fix]	2→z OCR repair no longer fires inside numbers (0.25 cm became 0.z5 cm, 1932b became 193zb)
[edit]	neutral reference tokens (dates, cites, initials, sigla, units, dotted abbreviations) excluded from both numerator and denominator of the valid-word ratio
[add]	structural short-line gate for word_count <= 2, deciding on damage evidence instead of language confidence
[add]	count_damaged_tokens() — stray symbol, apostrophe between letters, digit inside a lowercase word, vowel-less run — applied as a cap at Noisy
[add]	is_clean_reference() reference floor: a catalogue/measurement line with zero damaged tokens can be lifted Trash→Noisy, never demoted
[add]	stray-symbol/case-mix damage, rule_fragment_tokens, rule_bigram_run, solitary letter → Trash

Design constraints kept throughout: damage only ever caps at Noisy (it never routes to Trash), the reference floor only ever lifts, and every new rule is behind its existing DISABLED_RULES name so the ablation tooling keeps working.

Characters deliberately rejected as damage after checking how they actually occur in the corpus: ä/Ä (German toponymy, e.g. "Sandäcker" — 27 561 Clear lines carry it), • (bullet point, tens of thousands more), ô ľ ŕ ĺ Ľ Ŕ (Slovak), ¬ and U+00AD (end-of-line hyphenation), and ± (measurement tolerance, 892 lines).

Testing

Everything below was measured with tools/recategorize_from_csv.py — not once at the end, but as the working instrument throughout this calibration. It re-scores stored DOC_LINE_CATEG rows through the real compute_quality_score, categorize_line and apply_document_postprocessing, so every figure here is the pipeline's own behaviour rather than a parallel re-implementation. The only things wrapped around it were pd.set_option("future.infer_string", False) for pandas 3 and splitting the input directory into symlinked shards so several processes could run at once; neither touches scoring. Refs #30.

Three natural samples, drawn three different ways, annotated by two people. In every one of them the baseline is the current test branch recomputed today — never the categories stored in the dataset, which are a July snapshot and no longer match this branch:

sample	n	current test	this PR
uniform random across the whole corpus	300	61.3 %	79.0 %	+17.7 pp
16 documents annotated end to end	170	71.2 %	89.4 %	+18.2 pp
13 CTX documents, different collection, second annotator (TK)	654	64.7 %	80.3 %	+15.6 pp

Uniform random sample — the primary estimate, and the only one that is representative of the corpus as a whole. 300 lines drawn uniformly from all 17 327 811 scored lines of the corpus (documents drawn with probability proportional to their line count, then a line uniformly within), annotated blind: the annotator saw only the text, never either engine's verdict. Sanity check on the draw: 83.3 % of the sample is Clear under this PR against 83.4 % in the corpus at large.

Accuracy 61.3 % → 79.0 %, a paired difference of +17.7 pp, 95 % CI [11.6, 23.7], exact McNemar p = 5.2 × 10⁻⁹. The two engines disagree on 95 of the 300 lines; of those the annotator sided with this PR on 69 and with test on 16 (10 went to neither). The label distribution is the clearest summary — annotator Clear 246 / Noisy 39 / Trash 15, this PR 250 / 30 / 20, current test 173 / 88 / 39: the shipped engine flags 127 of the 300 lines — more than two in five — as needing attention, where the annotator flags 54, about one in six.

Independent validation. 13 documents of a different collection (CTX), annotated by a second annotator who never saw this work. Both engines were recomputed today from the same stored signals — the baseline column is the current test branch, not the categories that were on file when the annotation was made (those have since drifted, matching this branch on only 642 of 835 lines, which is why they are not used here).

Scored on the lines the categoriser actually decides: 120 lines improved against 18 worsened, 6.7 : 1.

Counting all 814 annotated lines except the obsolete Rough category gives 71.6 % → 84.2 %, but that basis is flattering to both engines: 160 of those lines are Empty/Non-text fast-track rows that the re-scorer passes through untouched, so both columns are identical there by construction. The 654-line figure above is the honest one.

Complete-document sample. 16 MTX documents annotated end to end, every line, no selection within them — 257 lines, of which 170 are Clear/Noisy/Trash. On those: 32 lines improved against 1 worsened. Caveat: the 16 documents are a consecutive block from 1945, so this sample says nothing about later decades; the random sample above spans the whole 1945–2024 range.

Full-corpus run. 47 677 documents / 40.5 M lines re-scored with tools/recategorize_from_csv.py, 0 errors. Resulting distribution: Clear 14 460 142 / Noisy 1 662 072 / Trash 1 205 597.

Primary annotation set. 1 993 lines / 1 196 documents of the MTX collection, annotated by hand. This set is deliberately enriched with the lines this change moves — most of it was sampled from lines whose category changed, so an absolute accuracy figure on it would be meaningless (the baseline is being judged precisely where it was known to differ). What it does measure is whether the moves are the right ones:

1 258 lines improved : 220 worsened, 5.7 : 1, on the hardest cases in the corpus.

What this trades away

The accuracy gain is not free, and the random sample shows the price. Treating "needs attention" as Noisy or Trash — 54 of the 300 lines by the annotator's judgement:

	found	recall	precision	false alarms
current test	39 / 54	72.2 %	30.7 %	88
this PR	25 / 54	46.3 %	50.0 %	25

Recall on problem lines drops from 72 % to 46 %. This PR misses 29 lines that deserve attention where test misses 15 — while raising precision from 31 % to 50 % and cutting false alarms from 88 to 25. Per class, Clear F1 goes 0.75 → 0.89, Noisy 0.25 → 0.29, and Trash slips 0.37 → 0.34.

Whether that is the right trade depends on what the categories are for, so it is your call rather than mine:

• As a filter feeding clean text downstream (atrium-nlp-enrich), it is a clear win. test marks only 64 % of genuinely clean lines as Clear; this PR marks 90 %, at almost the same purity (precision 0.91 → 0.88). At corpus scale that is millions of usable lines currently withheld to buy three points of purity.
• As a worklist for manual correction, recall is what matters, and test's 72 % at the cost of 88 false alarms per 300 lines is a defensible different choice — in which case this PR is the worse one and the damage detector should be tightened rather than the gate loosened.

I looked at all 29 missed lines rather than assuming what they are, and they fall into four groups:

• empty form fields — a label whose value never got filled or read: malta: - -, Současný s:, mírně -;
• bare abbreviations — indet, Kom., Nov., inv-číslo, which the neutral-token rule promotes on purpose;
• upper-case garbage — NINNNIC, KTU?A, AODP. FROJ EKTANI:, Jw.le.. The vowel-less and digit-in-word patterns in count_damaged_tokens() match lower-case runs only, so none of these reach them;
• mid-line solitary letters — uvláčeno I I; the solitary-letter route only fires on a one-word line.

So this is not purely a lexical problem, and I do not want to overclaim in either direction. The one extension that was measured — letting the digit-in-word pattern see upper case, together with treating = as damage — was rejected on volume. Measured together with a second candidate (treating = as damage), the two would newly hit roughly 190 000 lines across the corpus — 139 541 of the hits from the upper-case part alone — of which 85 901 are currently Clear, and those are overwhelmingly legitimate inventory and locality codes (MTX20091565, MD01/001/2022-E-170-230, probe designations like S6Y). Tens of thousands of correct lines is too much to pay for this.

The rest I have deliberately not guessed at. Note that an upper-case vowel-less rule would not catch NINNNIC either — it contains a vowel — so the fix for that group is not simply a case-insensitive regex. Candidates worth sizing properly are a repeated-letter run, ? inside a token, and a label whose value is only dashes; none of them is measured yet, so none is in this PR.

8 tests fail, and they fail by design — please arbitrate

pytest -m "not slow" on test is fully green (495 passed). With this PR: 8 failed, 487 passed. Nothing else changes. I have deliberately not touched the tests.

Four behavioural tests assert that short garbage reaches Trash. What they actually get now:

test	line	expected	now
test_text_utils.py::TestShortGarbageRoute::test_short_gibberish_token_routed_to_trash	olie	Trash	Clear
test_smoke.py::TestFullPipelineSmoke::test_real_short_garbage_is_trash_per_line	olie	Trash	Clear
test_rotation_regression.py::test_inverted_trash_stays_trash_at_default[oueussd…]	oueussd	Trash	Clear
test_smoke.py::TestFullPipelineSmoke::test_garbage_and_mirror_is_trash_or_nontext	pbqdnuwmoxszeyv!!	Trash/Non-text	Noisy

These are the cost side of the same mechanism that produces the gains. The gains, from the random sample — the annotator called every one of these Clear, and there are 63 of them in the 300 lines:

Noisy → Clear   "přičemž na 36. metru došlo k náhlému zlomu. Celková hloubka příkopu (07) širokého"
Noisy → Clear   "Pohřeb H1882, H1883 oba v poloze na břiše"
Noisy → Clear   "stěně čtverce je maximální délka 85 cm. Objekt se na"
Noisy → Clear   "Jihlava, Komenského 31"
Noisy → Clear   "Okres : Olomouc"
Trash → Clear   "A059/2020-107-008"

Unbroken Czech prose and plain inventory codes, marked as needing correction by the shipped engine. The same loosening that lets those through also lets olie and oueussd through: they are short, they carry no non-alphabetic damage, and the alpha heuristic in compute_valid_ratio() rates them structurally valid, exactly as it rates a genuine short Czech word. Telling them apart needs a lexical signal rather than another character rule — of the 68 over-lenient lines in the independent annotation, 63 (93 %) cannot be reached by any character-level rule at all, which is the #23 direction.

Four parity tests in tests/test_recategorize_parity.py check that the offline re-scorer still reproduces the categ stored in data_samples/DOC_LINE_CATEG/ — three of them with a ≤ 2 % drift gate, the fourth with ≤ 5 % per document. They guard the faithfulness of the measurement tool, not the correctness of the categories.

That corpus is 3 synthetic documents, 15 lines in total, so the smallest possible non-zero change is 1/15 = 6.7 %, three times the gate; and the document that flips has 4 lines, so that one line is 25 % against a 5 % gate. In practice these tests therefore assert no behaviour change at all, and any calibration fails them regardless of its merits.

This PR flips exactly one of the 15 lines:

CTX000000001 L1   stored: Noisy   →   this PR: Clear
"Výzkumná zpráva č. 1/2024 — Hradiště u Horní Mezí"

That is flawless Czech: a correctly read report title with a citation number. The current code scores it compute_valid_ratio() = 0.667, because č., 1/2024 and — all count as invalid words; that is what leaves it at the quality score of 0.7999 on file and below the Clear band. The neutral-token commit scores the same line 1.000 and it comes out Clear — which is, I think, what any reader would call it.

So the parity failures are not a regression: they are this change correcting a mislabel in your own fixture. Fixing them on your side is one value in one file.

Happy to do either, just say which: update the four behavioural tests to the new expectations (with the reasoning in their docstrings) and regenerate the 15 stored categories in the synthetic fixtures — or narrow the change until the current expectations hold, at the cost of the numbers above. That decision is the main thing I am asking for; the rest of the CI is green.

Provenance

To be explicit about how this was produced: the code, the comments and the measurements in this PR were written by Claude (Anthropic's Claude Opus), working from my data over several sessions. My own contribution is the ground truth and the judgement calls — I annotated the lines by hand, decided which candidate rules were acceptable and which were not, rejected several that looked good on paper, and validated every step against the corpus. Nothing here was accepted because the model proposed it: each candidate rule was first measured against the annotated lines and then scanned across the corpus to see how many lines it would actually move, and several were dropped at that second step precisely because the volume showed they would do more harm than good.

I am flagging this so you can weight the review accordingly.

[K4TEL]

will look into it tonight/tomorrow and maybe fix some of the errors from tests