Skip to content

Feature/remove unnessary re for table ele in pdf #3984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
Two executions of the same code, on the same file, produce different results. The order of the elements is random.
This makes it impossible to write stable unit tests, for example, or to obtain reproducible results.
- **Do not use NLP to determine element types for extracted elements with hi_res.** This avoids extraneous Title elements in hi_res outputs. This only applies to *extracted* elements, meaning text objects that are found outside of Object Detection objects which get mapped to *inferred* elements. (*extracted* and *inferred* elements get merged together to form the list of `Element`s returned by `pdf_partition()`)
- **RE_MULTISPACE_INCLUDING_NEWLINES was incorrectly used for Table or TableChunk** Newlines should not be removed from Table or TableChunk elements. The re.sub is not needed for Table or TableChunk elements now.

## 0.17.5

Expand Down
7 changes: 5 additions & 2 deletions unstructured/partition/pdf.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
Link,
ListItem,
PageBreak,
Table,
TableChunk,
Text,
Title,
process_metadata,
Expand Down Expand Up @@ -823,8 +825,9 @@ def _partition_pdf_or_image_local(

if isinstance(el, Image):
out_elements.append(cast(Element, el))
# NOTE(crag): this is probably always a Text object, but check for the sake of typing
elif isinstance(el, Text):
# NOTE(crag): this is probably always a Text object, but check for the sake of typing.
# NOTE(JQQ): Escape RE for Table and TableChunk
elif isinstance(el, Text) and not isinstance(el, (Table, TableChunk)):
el.text = re.sub(
RE_MULTISPACE_INCLUDING_NEWLINES,
" ",
Expand Down
Loading