Skip to content

[fix] 2.14.2 disabling legacy AES-256-CBC decryption breaks pre-migration stored ciphertext (550 Decryption failed for all supported formats) #584

Description

@srgvg

Summary

Release 2.14.2 (commit f4dd436, "fix: harden calendar, redirect, DMARC, and crypto boundaries")
gates legacy AES-256-CBC decryption in helpers/encrypt-decrypt.js behind a new
ALLOW_LEGACY_AES_CBC_DECRYPTION env var, defaulted off. For roughly an hour after that release
went out, outbound mail on one of the domains on our account failed with
550 Decryption failed for all supported formats, Error from: Forward Email.

That is consistent with a stored value still in the pre-migration ciphertext format becoming
unreadable once the flag defaulted to off. The likeliest candidate is domain.dkim_private_key
it is the only decrypt() call on the path after a message has already been accepted
(helpers/process-email.js:760); the other call in the SMTP path
(helpers/on-data-smtp.js:83) surfaces as 535, not 550.

Evidence

  • Deliverability log export (/v1/logs/download) for the affected domain shows delivery failing
    with 550 Decryption failed for all supported formats at 2026-09-04 22:02–22:15 UTC — 22 to 35
    minutes after 2.14.2 was tagged (2026-09-04T21:40:54Z).
  • Delivery resumed cleanly by 23:00 UTC the same day. Every hourly forward from then through
    2026-09-05 is 250, and there have been no further occurrences.
  • The affected domain's record dates to 2025-03-15, predating the encryption-format change tracked
    in [fix] TXT encryption broken #273 (closed 2025-11-23) — consistent with it still holding legacy-CBC ciphertext.
  • Nothing changed on our side, before or during the window. MX, SPF, DMARC, the DKIM TXT record and
    the fe-bounces CNAME were all valid throughout.

Impact

Small for us: four messages lost inside about an hour, self-healed with no action on our part.

Raising it because the failure mode is quiet and generic. The same helper backs SMTP auth password
decryption and webhook signing, so any account whose stored values predate the v2 format could hit
this on the same upgrade, and the only symptom is a 550 whose text does not say which value failed
to decrypt or that a legacy format was involved.

Suggested follow-up

  • Run a one-time migration re-encrypting pre-v2 ciphertext to the current format before the legacy
    path is disabled by default, rather than relying on operators to know to set the flag.
  • Log distinctly when the legacy-CBC branch is reached but disabled, instead of falling through to
    the generic "failed for all supported formats" error — that would have made this self-diagnosing.

Happy to supply the exact log rows if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions