Full-page OCR is blocked by UnicodeDecodeError in native PDF text parsing

[dobi02]

Bug

A PDF page renders successfully as an image, but native text parsing raises a
UnicodeDecodeError before full-page OCR can run.

The conversion is configured with OcrMode.FULL_PAGE. Since the page image is
valid and full-page OCR does not rely on native PDF text for its final text
cells, I expected OCR to proceed or the native text layer to degrade
gracefully.

Instead, the entire page conversion fails during page preprocessing.

This may be related to #1111, but this case uses the current
docling_parse.pdf_parser._to_cells_from_decoder() path and specifically
prevents full-page OCR from running.

Steps to reproduce

from pathlib import Path

from docling.backend.docling_parse_backend import DoclingParseDocumentBackend
from docling.datamodel.accelerator_options import (
    AcceleratorDevice,
    AcceleratorOptions,
)
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
    OcrMode,
    PdfPipelineOptions,
    RapidOcrOptions,
)
from docling.document_converter import DocumentConverter, PdfFormatOption

source = Path("202400733.pdf")

options = PdfPipelineOptions()
options.do_ocr = True
options.do_table_structure = False  # isolate the preprocessing/OCR failure
options.ocr_options = RapidOcrOptions(
    mode=OcrMode.FULL_PAGE,
    backend="torch",
)
options.accelerator_options = AcceleratorOptions(
    device=AcceleratorDevice.CPU,
)

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_options=options,
            backend=DoclingParseDocumentBackend,
        )
    }
)

result = converter.convert(source, page_range=(1, 1))
print(result.document.export_to_markdown())

The same underlying failure can be reproduced directly with docling-parse:

from pathlib import Path
from docling_parse.pdf_parser import DoclingPdfParser

source = Path("202400733.pdf")

document = DoclingPdfParser().load(source)
document.get_page(1)

Actual behavior

Page 1 fails before OCR with:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf
in position 0: invalid continuation byte

Relevant stack:

docling/models/stages/page_preprocessing/page_preprocessing_model.py
page._backend.get_segmented_page()

docling/backend/docling_parse_backend.py
self._ensure_parsed()
self._dp_doc.get_page(...)

docling_parse/pdf_parser.py
segmented_page_from_decoder(...)
_to_cells_from_decoder(...)
text=cell.text

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf
in position 0: invalid continuation byte

The page image itself can be rendered successfully. Running OCR after replacing
only the failed native text layer with an empty text layer produces OCR text,
so the rasterized page is not damaged.

Expected behavior

When full-page OCR is explicitly requested and page rendering succeeds, a
native PDF text decoding failure should not prevent OCR from running.

Possible behavior could be:

1. Attempt native text parsing normally.
2. If native text decoding fails in OcrMode.FULL_PAGE, emit a warning.
3. Continue with an empty native text layer and run full-page OCR.
4. Preserve the existing exception behavior for other OCR modes.

I am not sure whether invalid text cells should instead be handled inside
docling-parse, so guidance on the appropriate ownership would be appreciated.

Additional observation

After locally bypassing the preprocessing failure, table structure processing
can call page._backend.get_segmented_page() again and trigger the same
decoding error.

It may be necessary for downstream stages to use the OCR-updated
page.parsed_page rather than reparsing the native backend page.

Environment

Reproduced with:

Docling: 2.115.0
Docling Core: 2.87.1
Docling Parse: 7.8.1
Python: 3.11
Platform: Linux container

Also reproduced directly with docling-parse using:

Docling main base commit: 873f990
Docling Slim: 2.115.0
Docling Core: 2.86.0
Docling Parse: 7.7.0
Python: 3.13.2
Platform: Windows

Sample file

202400733.pdf

Failure occurs on page 1 of 202400733.pdf.

SHA-256:

5DA5D671FFD685F835D67FE0C39C0D8D6BC36F94C79CAC83C11796242CEEE896

The original PDF is attached without rewriting or extracting pages, since
rewriting the PDF may normalize the malformed text data and hide the issue.

[dosubot]

Hi @dobi02! I'm Dosu and I'm helping the IBM Docling team.

This is a legitimate gap in the preprocessing pipeline's error handling. Here's what I found after investigating the code:

Root cause: _parse_page_cells() unconditionally calls page._backend.get_segmented_page() with no try/except, and _ensure_parsed() in the docling-parse backend similarly has no error handling around self._dp_doc.get_page(). When _to_cells_from_decoder() raises a UnicodeDecodeError accessing cell.text, it propagates up and kills the entire page conversion.

No OCR-mode awareness: _parse_page_cells() does not check OcrMode at all — it unconditionally attempts native text extraction regardless of whether FULL_PAGE OCR is configured [1]. So even though full-page OCR would completely replace the native text layer, the preprocessing stage never gets that far.

Precedent in other backends: Other Docling backends already handle UnicodeDecodeError gracefully — e.g., the LaTeX backend tries multiple encodings and falls back to replacement mode [2], and the BoxNote backend catches the error and raises a structured DocumentLoadError [3]. The PDF preprocessing path lacks similar handling.

Relation to #1111: That issue reports a similar class of failure (RuntimeError: Invalid code point) from docling-parse during page parsing, and users have worked around it by converting problematic pages to images and using full-page OCR [4] — which is exactly what should happen automatically in your case.

Your proposed fix approach is sound. Wrapping the get_segmented_page() call in _parse_page_cells() with error handling that falls back to an empty text layer when OcrMode.FULL_PAGE is active would be the minimal change on the Docling side. For other OCR modes, preserving the existing exception behavior (or emitting a more structured error) makes sense since those modes rely on native text cells.

Regarding ownership: fixing it in _parse_page_cells() (Docling side) is the most pragmatic approach since it can be OCR-mode-aware. A complementary fix inside docling-parse's _to_cells_from_decoder() to skip or replace individual malformed cells rather than raising would also be valuable but is a separate concern.

Your observation about downstream stages (e.g., table structure) re-calling get_segmented_page() is also valid — they would hit the same error. The cached page.parsed_page from the preprocessing stage should be reused; if the fallback produces an empty text layer there, the same empty result would be cached and served to downstream stages.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.