Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/remove unnessary re for table ele in pdf #3984

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

JIAQIA
Copy link
Contributor

@JIAQIA JIAQIA commented Apr 9, 2025

RE_MULTISPACE_INCLUDING_NEWLINES is applied to all elements of the Text category after partitioning PDF files. The relevant code is shown below:

out_elements = []
for el in elements:
    if isinstance(el, PageBreak) and not include_page_breaks:
        continue

    if isinstance(el, Image):
        out_elements.append(cast(Element, el))
    # NOTE(crag): this is probably always a Text object, but check for the sake of typing
    elif isinstance(el, Text):
        el.text = re.sub(
            RE_MULTISPACE_INCLUDING_NEWLINES,
            " ",
            el.text or "",
        ).strip()
        if el.text or isinstance(el, PageBreak):
            out_elements.append(cast(Element, el))

Newlines will not be removed from Table or TableChunk elements now.

JIAQIA added 3 commits April 9, 2025 19:19
…table-ele-in-pdf' into feature/remove-unnessary-re-for-table-ele-in-pdf
@JIAQIA
Copy link
Contributor Author

JIAQIA commented Apr 9, 2025

This issue has a significant impact on our datasets because many of our image-based PDF files contain tabular data. Our OCR system extracts this information in Markdown format, where newline characters ("\n") are essential for preserving the table structure.

JIAQIA added 2 commits April 9, 2025 19:29
…table-ele-in-pdf' into feature/remove-unnessary-re-for-table-ele-in-pdf
…table-ele-in-pdf' into feature/remove-unnessary-re-for-table-ele-in-pdf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant