Algorithm change for 2010+ years documents

[K4TEL]

Proposed at the meeting - to be checked with tools/recategorize

https://github.com/ufal/atrium-alto-postprocess/blob/master/tools/recategorize_from_csv.py - make sure the new algo is aligned with the pipeline

[K4TEL]

#32 - recent changes to refine - mostly text_util.py script

[K4TEL]

Executive summary by GPT Sol 5.6

As of July 26, 2026, PR #32 has been merged into the test branch as commit f6adc23. The squash commit changes only text_util.py—294 additions and 5 deletions—and intentionally leaves the existing tests and stored sample categories unchanged. Your eight failing tests are the exact failures documented before the merge. (github.com)

---

What PR #32 changed

The merged calibration introduces six principal behavioral changes:

1. Reference tokens no longer lower the valid-word ratio.
   Dates, citations, initials, sigla, units, dotted abbreviations, figure references, and pure punctuation are excluded from both the numerator and denominator of compute_valid_ratio(). For example, 24.2.2020, /1933/32, I.L., Mzm., t. III, and Obr.3 are treated as neutral rather than invalid words. (github.com)

2. One- and two-token lines now use a structural gate.
   The new rule_short_line runs before the previous short-garbage route. FastText confidence is intentionally ignored for these lines because the language identifier is considered unreliable on very short input. Structurally clean short tokens can therefore become Clear even when a gibberish detector fires. (raw.githubusercontent.com)

3. Character-level damage caps output at Noisy.
   The new detector catches internal OCR damage such as stray symbols, apostrophes between letters, lowercase letter/digit mixtures, vowel-less lowercase runs, and suspicious mixed case. Its design constraint is that damage prevents Clear but does not itself force Trash. (github.com)

4. Clean references get a floor.
   Catalogue, inventory, and measurement lines with positive reference evidence and no detected character damage can be lifted from Trash to Noisy. The rule is explicitly a floor, not a demotion rule. (github.com)

5. Several narrowly measured patterns now force Trash.
   These include solitary letters, repeated alternating bigrams such as IDIDIDID..., and heavily fragmented lines with very short mean token length. (github.com)

6. The pre-filter no longer corrupts numbers.
   The 2 → z OCR repair is prevented from firing inside numeric measurements and codes, avoiding transformations such as 0.25 → 0.z5 and 1932b → 193zb. (github.com)

Reported evaluation results

The PR reports three manually annotated validation samples:

Sample	Lines evaluated	Previous test	PR #32	Change
Uniform random corpus sample	300	61.3%	79.0%	+17.7 pp
16 complete documents	170 categorized lines	71.2%	89.4%	+18.2 pp
13 independent CTX documents	654 categorized lines	64.7%	80.3%	+15.6 pp

For the representative 300-line sample, the paired difference had a reported 95% confidence interval of 11.6–23.7 percentage points and an exact McNemar p-value of 5.2 × 10⁻⁹. Among the 95 disagreements, the annotator preferred the new algorithm 69 times and the prior algorithm 16 times. A full-corpus re-score reportedly processed 47,677 documents and 40.5 million lines without runtime errors. (github.com)

However, the improvement has a substantial and clearly documented cost:

“Needs attention” metric	Previous test	PR #32
Recall	72.2%	46.3%
Precision	30.7%	50.0%
False alarms per 300 lines	88	25
Clear F1	0.75	0.89
Trash F1	0.37	0.34

Therefore:

• For clean-text delivery, the new version is probably preferable: it sharply reduces false alarms and releases many more correctly recognized lines.
• For a manual-correction worklist, it is less suitable without further work because it misses more actual problems. (github.com)

One review caveat is that the contributor disclosed that the implementation, comments, and measurements were produced with Claude Opus, while the human contribution consisted of manual annotation, acceptance decisions, and corpus-level review. That does not invalidate the results, but it strengthens the case for independently reproducing the evaluation and reviewing the shared scoring path. (github.com)

---

Recommended next change

Priority 1: extract one shared per-line scoring function

The strongest fix is to stop reconstructing production behavior in three places. Create one pure function, conceptually:

def score_line_from_frozen_signals(
    *,
    text: str,
    original_text: str,
    original_lang: str,
    orig_lang_score: float,
    perplexity: float,
    expected_langs: list[str],
    known_lang_bases: frozenset[str],
) -> dict:
    ...

It should own:

1. short-line perplexity capping;
2. remapped display language and display score;
3. trust-tier scoring confidence;
4. original-text density and vowel metrics;
5. all structural detectors;
6. compute_quality_score;
7. categorize_line;
8. reason flags and aligned quality score.

Then use that same function from:

• classify_TEXT.py;
• tools/recategorize_from_csv.py;
• test_smoke.py;
• test_rotation_regression.py;
• calibration tests.

That would satisfy the parity test’s stated principle in reality, rather than merely sharing the final two functions.

Priority 2: separate parity tests from accuracy tests

These are different contracts.

Engine parity fixture

Purpose: Given frozen model outputs and the same algorithm/config version, offline re-scoring must exactly reproduce production.

Recommended assertion:

assert predicted[DERIVED_COLUMNS].equals(stored[DERIVED_COLUMNS])

The fixture should be regenerated whenever the intended algorithm changes, with its generating commit/config recorded.

Manually labeled calibration fixture

Purpose: The algorithm must retain acceptable quality against human ground truth.

Track at least:

• overall accuracy;
• Clear precision and recall;
• combined Noisy|Trash recall;
• Trash precision and recall;
• false alarms per fixed sample size;
• results by line length, especially one- and two-token lines;
• results by document, collection, and language.

This prevents an overall accuracy improvement from hiding the substantial problem-recall reduction documented in PR #32.

Priority 3: handle the behavioral tests honestly

I recommend the following immediate policy:

Case	Immediate test treatment
Clean CTX title	Regenerate fixture; expect Clear
pbqdnuwmoxszeyv!!	Expect Noisy, or at minimum assert cat != "Clear"
olie	Keep as known desired behavior, but mark strict xfail linked to #23
oueussd	Mark only this parameter as strict xfail linked to #23

For example:

@pytest.mark.xfail(
    strict=True,
    reason="Known lexical ambiguity for structurally clean short alpha tokens; tracked in #23",
)
def test_short_gibberish_token_routed_to_trash():
    ...

Using strict=True is useful: once a lexical improvement makes the test pass, pytest reports an unexpected pass and forces removal of the marker.

I would not rewrite the olie and oueussd tests to claim that Clear is semantically correct. They are known garbage examples. Treat them as accepted technical debt, not newly correct labels.

Longer-term algorithm direction

For the specific failures here, a full replacement model may not be necessary initially. A narrower lexical component could be evaluated only in this ambiguous branch:

one or two tokens
+ alphabetic
+ structurally clean
+ no strong Czech/reference evidence
+ gibberish/fused/rotation suspicion

Possible signals include:

• Czech word-frequency or morphological-dictionary lookup;
• character n-gram language probability;
• compact learned word-likeness classifier;
• comparison of original and rotation/mirror transforms;
• calibrated fallback to Noisy, rather than immediately to Trash.

Any such rule should be measured against:

• short Czech words;
• surnames and place names;
• German and Slovak toponyms;
• inventory codes;
• abbreviations and sigla.

The PR already demonstrates why a broad regex is risky: proposed uppercase/damage expansions were rejected after corpus scans showed that they would affect very large numbers of legitimate codes and locality identifiers. (github.com)

---

Final recommendation

The best next PR would:

1. extract a single shared production scoring function;
2. make the offline re-scorer use production-identical inputs;
3. replace the duplicated smoke/rotation implementations;
4. regenerate the parity fixture, including every derived column;
5. change the pbq... expectation to Noisy or “not Clear”;
6. preserve olie and oueussd as strict known failures linked to Consider finetuning NLP model for quality score assignement #23;
7. add a human-labeled metric gate that tracks both clean-text precision and problem-line recall.

That produces green CI without pretending the remaining short-token false negatives are correct, while retaining the empirically valuable parts of the calibration.

TODO by me