Question

[francofuji]

Hi James, I came across smtpdfix —building an SMTP server as a pytest fixture
with real TLS and auth support tells me you hit friction that the simple
mock_smtp approach couldn't solve.

I'm researching that exact problem. What was the hardest part to get right
when testing email-dependent code in Python?

[bebleo]

Hi Francisco,

Thanks for reaching out.

The motivation behind smtpdfix was that the mock_smtp approach becomes quite complicated in tests to check that code would connect to an SMTP server and successfully transmit an email. This is even more true if there were specific requirements for the connection.

smtpdfix is a wrapper around aiosmtpd extending it - largely by adding authentication methods to work with smtplib.SMTP and smtplib.SMTP_SSL - and creating a Pytest fixture. This seemed like an easy solution when I implemented it originally.

However, the approach of starting an SMTP server and accepting a connection carries several issues, among them:

• somewhat slow particularly on older or underpowered machines,
• prone to network issues (macOS turns out to have a very quirky network/nic configuration),
• the underlying project, aiosmtpd appears to be abandoned -- the same could be said of smtpdfix though I hope to get back to it in the future -- which represents a risk.
• changes to the asyncio library in Python have necessitated fairly frequent updates and changes.

With the advantage of hindsight, I wouldn't use a server if I were to start the project today. My preference would be to have a fully-featured mock_smtp fixture. If not, I would at least use mock_socket - or something similar - to simulate a connection. But I've not had time in the last few years to fully explore either option in detail.

I hope this helps. It you have more questions, please don't hesitate to add to this discussion thread.

-James

P.S. Ironically, I never ended up using smtpdfix myself.

[francofuji]

This is really helpful, James —and the irony note at the end is telling :D

The part I'm still curious about: if you went with mock_smtp or mock_socket, how would you handle tests that need to actually inspect the email content —extract the OTP code, follow the magic link, verify the template?

Is that something you'd mock at the application layer instead, or is that a gap that the approach doesn't fully close?

[bebleo]

Hi Francisco,

Apologies for the delayed response. It has, as they say, been a week!

I would test your scenarios, "extract the OTP code, follow the magic link, verify the template," at the application layer. Universally these require verifying the content to be sent. And, from a design perspective, I would keep the content generation separate from the transmission of the email. If I found myself testing both together, I should - but probably wouldn't because I'm lazy 😊 - ask myself if I've coupled concerns that shouldn't be.

smtpdfix tests whether an email can be successfully sent assuming an SMTP server with a specific configuration. Even this claim is a bit dubious but I like to think it's "good enough" as an approach.

That being said, tests involving smtplib in Python can be painful because figuring out where, when, and how to mock/patch the library and/or its methods is often less than clear. Spinning up an SMTP server to capture the email is tempting as a quick and dirty fix; increasingly, I'm of the opinion that in most cases it's almost certainly the wrong solution.