Reading-order hard-hyphen continuations retain an inserted space

[Success6666]

Bug

ReadingOrderModel._merge_elements removes a trailing soft hyphen (\u00ad) before joining a cross-element continuation, but a hard hyphen (-) always follows the regular space-joining path. A split word such as algo- + rithms is therefore emitted as algo- rithms.

Steps to reproduce

The affected code path is deterministic with two TextElement instances carrying the same label:

1. Create a list item whose text is algo-.
2. Merge a following LIST_ITEM element whose text is rithms.
3. ReadingOrderModel._merge_elements(...) produces algo- rithms instead of algorithms.

Soft hyphens already produce algorithms, which makes the hard-hyphen behavior inconsistent.

Expected behavior

When the first element ends in - and the first non-whitespace character of the continuation is lowercase, treat the pair as a hyphenated line wrap: remove the hyphen and join without a space. Preserve the existing behavior for uppercase continuations, where a hard hyphen may be meaningful punctuation.

Docling version

main at 873f990 (package version 2.115.0).

Python version

Python 3.13.12.

Relationship to #3881

This is the downstream half of #3881. A companion change in docling-ibm-models makes eligible LIST_ITEM pairs reach the merge mapping; this change ensures the final document text is correctly dehyphenated once that mapping is present.

[dosubot]

Your analysis is on point. Here's what I found in the current code:

The _merge_elements method [1] has an explicit if for soft hyphens that strips \u00ad and joins without a space, but the else branch unconditionally joins with f" {merged_elem.text}" — so any trailing hard hyphen (-) gets the space treatment, producing "algo- rithms".

The fix site is clear: between the existing soft-hyphen check and the else branch, you'd add an elif that checks whether new_item.text ends with - and merged_elem.text starts with a lowercase letter, then strips the hyphen and joins without a space. Something like:

if new_item.text.endswith("\u00ad"):
    new_item.text = new_item.text[:-1] + merged_elem.text
    new_item.orig = new_item.orig[:-1] + merged_elem.text
elif (
    new_item.text.endswith("-")
    and merged_elem.text[:1].islower()
):
    new_item.text = new_item.text[:-1] + merged_elem.text
    new_item.orig = new_item.orig[:-1] + merged_elem.text
else:
    new_item.text += f" {merged_elem.text}"
    new_item.orig += f" {merged_elem.text}"

This is consistent with how sanitize_text in page_assemble_model.py already handles intra-cluster hard-hyphen dehyphenation [2], where it checks if a line ends with - and the next line's first word is alphanumeric before joining without a space.

As noted in #3881, the companion change in docling-ibm-models to get LIST_ITEM elements into the merge mapping is the other half — once those elements reach _merge_elements, this fix ensures the text is correctly dehyphenated [3].

No existing PR was found that addresses hard-hyphen handling in _merge_elements [4].

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.