Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit ec59abf

Browse files
authoredJul 19, 2024··
enhancement: improve text clearing process in email partitioning (#3422)
### Summary Currently, the email partitioner removes only `=\n` characters during the clearing process. However, email content sometimes contains `=\r\n` characters, especially when read from file-like objects such as `SpooledTemporaryFile` (the file type used in our API). This PR updates the email partitioner to remove both `=\n` and `=\r\n` characters during the clearing process. ### Testing ``` filename = "example-docs/eml/family-day.eml" elements = partition_email( filename=filename, ) print(f"From filename: {elements[3].text}") with open(filename, "rb") as test_file: spooled_temp_file = tempfile.SpooledTemporaryFile() spooled_temp_file.write(test_file.read()) spooled_temp_file.seek(0) elements = partition_email(file=spooled_temp_file) print(f"From spooled_temp_file: {elements[3].text}") ``` **Results:** - on `main` ``` From filename: Make sure to RSVP! From spooled_temp_file: Make sure to = RSVP! ``` - on `PR` ``` From filename: Make sure to RSVP! From spooled_temp_file: Make sure to RSVP! ```
1 parent 1df7908 commit ec59abf

File tree

5 files changed

+56
-4
lines changed

5 files changed

+56
-4
lines changed
 

Diff for: ‎CHANGELOG.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
## 0.15.0-dev16
1+
## 0.15.0
22

33
### Enhancements
44

5+
* **Improve text clearing process in email partitioning.** Updated the email partitioner to remove both `=\n` and `=\r\n` characters during the clearing process. Previously, only `=\n` characters were removed.
56
* **Bump unstructured.paddleocr to 2.8.0.1.**
67
* **Refine HTML parser to accommodate block element nested in phrasing.** HTML parser no longer raises on a block element (e.g. `<p>`, `<div>`) nested inside a phrasing element (e.g. `<strong>` or `<cite>`). Instead it breaks the phrasing run (and therefore element) at the block-item start and begins a new phrasing run after the block-item. This is consistent with how the browser determines element boundaries in this situation.
78
* **Install rewritten HTML parser to fix 12 existing bugs and provide headroom for refinement and growth.** A rewritten HTML parser resolves a collection of outstanding bugs with HTML partitioning and provides a firm foundation for further elaborating that important partitioner.

Diff for: ‎example-docs/eml/family-day.eml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
MIME-Version: 1.0
2+
Date: Wed, 21 Dec 2022 10:28:53 -0600
3+
Message-ID: <CAPgNNXQKR=o6AsOTr74VMrsDNhUJW0Keou9n3vLa2UO_Nv+tZw@mail.gmail.com>
4+
Subject: Family Day
5+
From: Mallori Harrell <mallori@unstructured.io>
6+
To: Mallori Harrell <mallori@unstructured.io>
7+
Content-Type: multipart/alternative; boundary="0000000000005c115405f0590ce4"
8+
9+
--0000000000005c115405f0590ce4
10+
Content-Type: text/plain; charset="UTF-8"
11+
12+
Hi All,
13+
14+
Get excited for our first annual family day!
15+
16+
There will be face painting, a petting zoo, funnel cake and more.
17+
18+
Make sure to RSVP!
19+
20+
Best.
21+
22+
--
23+
Mallori Harrell
24+
Unstructured Technologies
25+
Data Scientist
26+
27+
--0000000000005c115405f0590ce4
28+
Content-Type: text/html; charset="UTF-8"
29+
Content-Transfer-Encoding: quoted-printable
30+
31+
<div dir=3D"ltr">Hi All,<div><br></div><div>Get excited for our first annua=
32+
l family day!=C2=A0</div><div><br></div><div>There will be face painting, =
33+
a petting zoo, funnel cake and more.</div><div><br></div><div>Make sure to =
34+
RSVP!</div><div><br></div><div>Best.<br clear=3D"all"><div><br></div>-- <br=
35+
><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gmail_signatu=
36+
re"><div dir=3D"ltr">Mallori Harrell<div>Unstructured Technologies<br><div>=
37+
Data Scientist</div><div><br></div></div></div></div></div></div>
38+
39+
--0000000000005c115405f0590ce4--

Diff for: ‎test_unstructured/partition/test_email.py

+12
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
import email
33
import os
44
import pathlib
5+
import tempfile
56

67
import pytest
78

@@ -230,6 +231,17 @@ def test_partition_email_from_file_rb_default_encoding(filename, expected_output
230231
assert element.metadata.filename is None
231232

232233

234+
def test_partition_email_from_spooled_temp_file():
235+
filename = example_doc_path("eml/family-day.eml")
236+
with open(filename, "rb") as test_file:
237+
spooled_temp_file = tempfile.SpooledTemporaryFile()
238+
spooled_temp_file.write(test_file.read())
239+
spooled_temp_file.seek(0)
240+
elements = partition_email(file=spooled_temp_file)
241+
assert len(elements) == 9
242+
assert elements[3].text == "Make sure to RSVP!"
243+
244+
233245
def test_partition_email_from_text_file():
234246
filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "fake-email.txt")
235247
with open(filename) as f:

Diff for: ‎unstructured/__version__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "0.15.0-dev16" # pragma: no cover
1+
__version__ = "0.15.0" # pragma: no cover

Diff for: ‎unstructured/partition/email.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -416,8 +416,8 @@ def partition_email(
416416
# <li>Item 1</li>=
417417
# <li>Item 2<li>=
418418
# </ul>
419-
list_content = content.split("=\n")
420-
content = "".join(list_content)
419+
420+
content = content.replace("=\n", "").replace("=\r\n", "")
421421
elements = partition_html(
422422
text=content,
423423
include_metadata=False,

0 commit comments

Comments
 (0)
Please sign in to comment.