You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Short documents (< MIN_EXTRACTED_SIZE) lose all Markdown structure in 2.2.0: baseline rescue replaces a valid formatted extraction with fused text_content #896
Since 2.2.0, extracting a short but perfectly valid static page with output_format="markdown" and include_formatting=True returns a single
unformatted blob: heading markers and paragraph breaks are gone, and the
heading text is fused with the body text without any separator
("Notice TitleThis municipal notice…"). The same input on 2.1.0 kept the
Markdown structure.
To Reproduce
importtrafilaturabody= ("This municipal notice is short but perfectly valid static content ""that a reader would expect to keep its structure. ") *2html= (
"<html><head><title>Notice</title></head><body><article>"f"<h1>Notice Title</h1><p>{body}</p>""</article></body></html>"
)
print(trafilatura.extract(html, output_format="markdown", include_formatting=True))
2.1.0 → # Notice Title\n\nThis municipal notice … (structured; 2.1.0 also
duplicated the paragraph via the recovery path, which some extraction duplicated in xml #634 fixed)
2.2.0 → Notice TitleThis municipal notice … structure. (no #, no
newlines, heading fused into the first word of the body)
Analysis
The main extractor produces a correct, structured result for this page, but
its text length (~240 chars) is below MIN_EXTRACTED_SIZE (250), so stage 3
of the cascade in core.py (if len_text < options.min_extracted_size … baseline(tree)) replaces it with baseline()'s output. For a page with
an <article> tag, baseline() returns trim(article.text_content()) in a
single <p>, which:
discards headings/paragraph boundaries even though the caller asked for
formatted Markdown, and
joins text across element boundaries with no whitespace
(<h1>Notice Title</h1><p>This… → Notice TitleThis…), corrupting the
words themselves.
On 2.1.x this path rarely fired for such pages only by accident: the
recovery-path paragraph duplication (fixed in #634) inflated len_text
above the threshold. The dedup fix is correct — it just exposed the rescue
behavior.
Expected behavior
When the main extractor already produced a non-empty structured result, the
baseline rescue should not replace it with an unformatted dump of the same
content — or at minimum, baseline()'s article branch should join
block-level children with separators instead of raw text_content() so
word boundaries survive.
Workaround we use: when the first pass returns fewer than MIN_EXTRACTED_SIZE chars, re-extract with a config that lowers MIN_EXTRACTED_SIZE and keep the structured result if it is not shorter.
Environment
trafilatura 2.2.0 (regression vs 2.1.0), Python 3.13, lxml from PyPI
Describe the bug
Since 2.2.0, extracting a short but perfectly valid static page with
output_format="markdown"andinclude_formatting=Truereturns a singleunformatted blob: heading markers and paragraph breaks are gone, and the
heading text is fused with the body text without any separator
("Notice TitleThis municipal notice…"). The same input on 2.1.0 kept the
Markdown structure.
To Reproduce
# Notice Title\n\nThis municipal notice …(structured; 2.1.0 alsoduplicated the paragraph via the recovery path, which some extraction duplicated in xml #634 fixed)
Notice TitleThis municipal notice … structure.(no#, nonewlines, heading fused into the first word of the body)
Analysis
The main extractor produces a correct, structured result for this page, but
its text length (~240 chars) is below
MIN_EXTRACTED_SIZE(250), so stage 3of the cascade in
core.py(if len_text < options.min_extracted_size … baseline(tree)) replaces it withbaseline()'s output. For a page withan
<article>tag,baseline()returnstrim(article.text_content())in asingle
<p>, which:formatted Markdown, and
(
<h1>Notice Title</h1><p>This…→Notice TitleThis…), corrupting thewords themselves.
On 2.1.x this path rarely fired for such pages only by accident: the
recovery-path paragraph duplication (fixed in #634) inflated
len_textabove the threshold. The dedup fix is correct — it just exposed the rescue
behavior.
Expected behavior
When the main extractor already produced a non-empty structured result, the
baseline rescue should not replace it with an unformatted dump of the same
content — or at minimum,
baseline()'s article branch should joinblock-level children with separators instead of raw
text_content()soword boundaries survive.
Workaround we use: when the first pass returns fewer than
MIN_EXTRACTED_SIZEchars, re-extract with a config that lowersMIN_EXTRACTED_SIZEand keep the structured result if it is not shorter.Environment