fix(baseline): space block boundaries in the article tier (#896)

[taro0915]

Fixes #896 (the fusion half; see Scope below).

Problem

The <article> tier called text_content() on the whole subtree, so text ran together across block boundaries. <h1>Notice Title</h1><p>This ... came out as Notice TitleThis ..., corrupting the words themselves rather than merely dropping formatting.

The same failure mode is already handled a few lines down in the same file: html2txt() pads block boundaries via _BLOCK_ELEMS before reading text, with a comment naming this exact case. The other baseline tiers avoid it structurally — the paragraph tier calls text_content() per element and joins with "\n", the default tier uses itertext(). Only the <article> tier calls it on a whole subtree, so every block boundary inside the article fuses: h2 / p / li / td alike, not just the h1 in the report.

Change

Extract the spacing pass into _space_block_boundaries() and call it from both sites, so the loop body exists once rather than being duplicated.

The article tier goes through _spaced_text_content(), which applies it to a deepcopy: the pass writes .text/.tail, and baseline() shares one tree across all its strategies. html2txt()'s tree is freshly parsed or already copied, so it spaces the body in place and pays no copy cost.

Per your guidance in #896 this reuses the existing pass rather than restructuring the tier to build the body per block element.

Scope

The fusion only. The missing Markdown heading also reported in #896 originates earlier, in the stage 2 comparison as @ebarkhordar showed, and is not addressed here.

Threshold and benchmark impact

You asked for the evaluation data on the cutoff question. Measured on the bundled corpus (990 documents), comparing 2ba8f62 against the same tree with this change applied. Python 3.12, Windows, pip install -e ".[all]".

baseline() output length

	count	share
unchanged	678 / 990	68.5%
increased	311 / 990	31.4%
decreased	1 / 990	0.1%

Deltas over the 312 changed documents: min −1, median +8, max +908.

The single decrease is anglerboard.de-rute.html (4312 → 4311): the article now goes through remove_control_characters, which drops one U+200B that raw text_content() kept. That is the existing guard doing its job, not lost content.

Strategy selection — no document changed tier. Identical before and after: article 527, paragraph 369, json 77, default 17.

Threshold crossings

threshold	location	crossings
_MIN_CONTENT_LENGTH = 100	baseline.py:52	4 admitted, 0 rejected
cutoff = max/5	baseline.py:206	0 documents
MIN_EXTRACTED_SIZE = 250	settings.cfg:26	0 in either direction

The four newly admitted candidates (100→102, 98→101, 100→101, 100→101) are small related-content blocks in pages carrying a much larger dominant article, so the max/5 cutoff drops them again. The selected article set is unchanged in all 990 documents.

End-to-end extract(), comparing SHA-1 of the output: 4 / 990 documents changed, the same four under both runners.

document	extract()
archive.org.swap-stop.org.shuji.html	506 → 517
caymancompass.com-prison.html	1164 → 1180
knowledge-on-air.de.koa039.html	447 → 452
sprechblase.wordpress.com.zapfsaeulen.html	481 → 492

All four are short pages where the main extractor falls under MIN_EXTRACTED_SIZE and stage 3 hands over to baseline() — the path #896 reports. They gain only the separators that were missing at block boundaries. The remaining 986 are byte-identical.

Quality gate (tests/eval_gate.py)

	fast	fallback
before (2ba8f62)	0.9184	0.9243
after (this PR)	0.9184	0.9243
pinned floor	0.9180	0.9245

F1 is unchanged to four decimal places on both runners, and the gate exits 0 in both states. The before/after runs were taken by stashing only trafilatura/baseline.py; re-running after restoring reproduced the "after" numbers exactly.

One caveat rather than leave it implicit: fallback measures 0.9243 against a pinned floor of 0.9245 — 0.0002 under, inside the gate's EPSILON = 0.0005 band, so it passes. That is equally true before the change, so it looks like a property of this measurement environment rather than something this PR introduces.

Tests

Three tests in tests/baseline_tests.py. Full suite: 339 passed, 0 skipped.

Stashing the implementation and re-running turns 5 of them red, which is how the "h2/p/li/td alike" claim above was checked rather than assumed:

• test_baseline_article_block_boundaries — h1/p fusion, and that inline runs inside a block stay joined
• test_baseline_article_spacing_covers_all_block_elements[h2_p|li_li|td_td] — all three reproduce the fusion without the fix
• test_spaced_text_content_does_not_mutate_input — removing the deepcopy alone is enough to turn this one red

A note on that last one. An earlier version asserted non-mutation through baseline() instead, and it passed with the deepcopy removed — the later tiers run trim() per element, which absorbs the injected spaces. It was guarding nothing. Calling _spaced_text_content directly and comparing tostring() is what makes it fail on the mutation it is meant to catch.

ruff check, ruff format --check and mypy -p trafilatura show nothing new; the pre-existing zstandard note in utils.py:38 is unrelated.

Disclosure

Prepared with AI assistance. The design decisions, the scope boundary and every number above are mine — the measurements were run locally on the bundled corpus, and the before/after comparisons were taken by stashing only trafilatura/baseline.py.

[adbar]

Thanks, I may have a better fix for this after all, I'll need to check.