Skip to content

One envelope reader and one part classifier across the flat parsers (#234) - #260

Open
kurok wants to merge 1 commit into
feat/237-one-tree-traversalfrom
feat/234-shared-flat-helpers
Open

kurok wants to merge 1 commit into
feat/237-one-tree-traversalfrom
feat/234-shared-flat-helpers

Conversation

@kurok

@kurok kurok commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Closes #234 (step 1; step 2 deliberately deferred — see below).

Stacked on #259, which is stacked on #258. Each retargets itself as the one below merges.

The duplication

Mail::from_payload, lazy_from_payload and metadata_from_payload each carried their own copy of the eleven-statement envelope extraction and of the per-part classification. The MIME-tree traversal carried a fourth copy of the identity derivation.

The copies had drifted in the way duplication drifts — header_addresses documented ten call sites when there were fifteen — and the drift that matters here is not cosmetic. The envelope is what strict=True and the whole warning channel are about; the classification rule decides what an attachment is. Three derivations of those are three chances for two views of one message to disagree, which is what parse_agreement's invariants 3 and 7 exist to catch.

The shape

struct Envelope { headers, subject, date, from_, to, cc, bcc, reply_to }
#[inline] fn envelope(mail: &ParsedMail<'_>, warnings: &mut Vec<Warning>) -> Envelope;

struct PartInfo<'p> { mime: &'p str, disposition: ParsedContentDisposition, is_body: bool }
#[inline] fn classify_part<'p>(part: &'p ParsedMail<'_>) -> Option<PartInfo<'p>>;   // None = multipart/*

struct PartIdentity { filename, content_id, disposition }
#[inline] fn part_identity(part: &ParsedMail<'_>, disposition: &ParsedContentDisposition) -> PartIdentity;

Straight-line and #[inline], so each caller emits the instructions it emitted when it owned a copy. What #100 measured at +47% was threading a runtime mode through the parse — a branch in the hot path taken for the benefit of the cold one. There is no mode and no branch here, which is the property that makes sharing free.

Counted in the core:

before after
DispositionType::Attachment (the RFC 2183 rule) 3 1¹
get_first_value("Content-ID") derivations 4 1
starts_with("multipart/") skips 3 1²

¹ the remaining one is disposition_token's own match. ² the other is extract_mail_parts's, which is structural recursion, not classification.

The warn_date check stays at the two callers that want it: metadata mode deliberately does not make it, and a parameter to say so would be exactly the branch this avoids.

One deviation from the issue's sketch

It proposes a single PartInfo { filename, content_id, disposition, is_body }. Bundling the rule with the identity would derive a Content-ID and a disposition token for every text/plain body in every message — work no mode does today. So the rule is separate from the identity, and the identity is read only where it is used. A side effect: body parts stop deriving a filename they never used, which full and lazy mode both did.

Step 2 is not here

The issue's step 2 — FlatMail<A> plus a PartSink trait, to share the loop body itself — is marked optional and splittable "if step 1's A/B is clean". It is clean, and step 2 rewrites Mail::from_payload, the hottest function in the crate. It deserves its own measurement rather than riding along with this one. Happy to do it as a follow-up.

Measurements

Local interleaved A/B on an M4 (Apple M4, 10 vCPU), 3 rounds, against this PR's base:

Benchmark base (#259) this PR
parse_message 0.156 ms 0.156 ms +0.1%
parse_metadata 0.029 ms 0.029 ms +0.7%
parse_lazy_untouched 0.030 ms 0.030 ms +0.3%
parse_lazy_all_attachments 0.163 ms 0.164 ms +0.8%
parse_many 1.289 ms 1.300 ms +0.8%
parse_many_metadata 0.229 ms 0.229 ms +0.1%
parse_tree 0.156 ms 0.156 ms −0.3%
parse_tree_metadata 0.030 ms 0.030 ms +0.0%
parse_tree_lazy_untouched 0.030 ms 0.030 ms +0.7%

Noise floor from the pure-Python controls 1.4%; worst treatment delta +1.8% (parse_qp_message); verdict no significant difference. The issue predicted "every one within about ±1%, nothing expected to improve" — that is what happened.

Verification: 864 Python tests; 13 core Rust tests, two of them new — the RFC 2183 rule and the identity derivation now have direct tests instead of only three end-to-end ones; cargo clippy --workspace --all-targets -D warnings -W clippy::cast_possible_truncation; cargo fmt; mypy --strict; ruff; check_vendored_mailparse.sh; check_bench_lockfile.py. 179K cargo fuzz run -a parse_agreement executions, zero crashes. __init__.pyi, tests/, docs/, vendor/ and src/ are untouched — the diff is the core and the changelog.

…234)

Mail::from_payload, lazy_from_payload and metadata_from_payload each carried
their own copy of two blocks: the eleven-statement envelope extraction (header
map, Subject, Date, From/To/Cc/Bcc/Reply-To) and the per-part classification
(skip multipart/*, apply the RFC 2183 body-vs-attachment rule, derive filename,
Content-ID and disposition token). The MIME-tree traversal carried a fourth copy
of the identity derivation.

The copies had already drifted in the way duplication drifts: header_addresses
documented ten call sites when there were fifteen. And the drift that matters is
not cosmetic -- the envelope is what strict=True and the whole warning channel
are about, and the classification rule decides what an attachment IS. Three
derivations of those were three chances for two views of one message to
disagree, which is the failure parse_agreement's invariants 3 and 7 exist to
catch.

Now: envelope(), classify_part() and part_identity(). Straight-line and
#[inline], so each caller emits the instructions it emitted when it owned a
copy. What #100 measured at +47% was threading a runtime MODE through the parse
-- a branch in the hot path taken for the benefit of the cold one. There is no
mode and no branch here, which is the property that makes sharing free.

Counted in the core: DispositionType::Attachment goes 3 -> 1 (the remaining one
is disposition_token's own match), the Content-ID derivation 4 -> 1, and the
multipart/* skip 3 -> 1 (the other is extract_mail_parts'). The tree traversal
reads a part's identity from the same helper the flat modes do, so a part cannot
answer to a different name depending on which API asked.

One deviation from the issue's sketch, on purpose. It proposes a single
PartInfo { filename, content_id, disposition, is_body }. Bundling the rule with
the identity would derive a Content-ID and a disposition token for every
text/plain body in every message -- work no mode does today. So the rule
(classify_part) is separate from the identity (part_identity), and the identity
is read only where it is used. Body parts consequently stop deriving a filename
they never used, which full and lazy mode did do.

Step 2 of the issue -- FlatMail<A> + a PartSink trait to share the loop itself --
is not here. The issue marks it optional and splittable once step 1's A/B is
clean; it rewrites Mail::from_payload, the hottest function in the crate, and it
deserves its own measurement rather than riding along with this one.

Measured flat: local interleaved A/B on an M4, 3 rounds, worst +1.8%
(parse_qp_message) against a 1.4% control noise floor, verdict "no significant
difference". 864 Python tests, 13 core tests (two new, giving the RFC 2183 rule
and the identity derivation direct tests instead of only three end-to-end ones),
179K fuzz executions with debug assertions live, clippy, fmt, mypy --strict,
ruff and both repo invariant scripts.
@kurok

kurok commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Green on the first run: 11/11, benchmark gate no significant difference (worst +4.6%, threshold 7%, AMD EPYC 9V74).

The four checks that did not register are CodeQL's, for the same reason as on #259 — this repo uses GitHub's default setup for code scanning, which only runs on pull requests targeting the default branch. They will run once this retargets to master as the stack merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant