gh-67022: Document bytes/str inconsistency in email.header.decode_header() and add .decode_header_to_string() as a sane alternative

[dlenski]

This function's possible return types have been surprising and error-prone
for the entirety of its Python 3.x history. It can return either:

typing.List[typing.Tuple[str, None]], of length exactly 1
or typing.List[typing.Tuple[bytes, typing.Optional[str]]]

This function can't be rewritten to be more consistent in a backwards-compatible way, because some users of this function depend on the existing return type(s).

This PR addresses the inconsistency as suggested by @JelleZijlstra in #67022 (comment):

1. we should document the surprising return type at https://docs.python.org/3.10/library/email.header.html.

2. [create] a new function with a sane return type.

The "sane", Pythonic way to handle the decoding of an email/MIME message header value is simply to convert the whole header to a str; the details of exactly which parts of that header were encoded in which charsets are not relevant to the users. Fortunately, the email.header module already contains a mechanism to do this, via the __str__ method of email.header.header, so we can simply create a wrapper function to guide users in the right direction.

Example of the old/inconsistent (decode_header) vs. new/sane (decode_header_to_string) functions:

>>> from email import decode_header, decode_header_to_string
>>>
>>> # Do most users care about this distinction in (sub)encodings? I think not.
>>> print(decode_header('hello =?utf-8?B?ZsOzbw==?= bar'))
[(b'hello ', None), (b'f\xc3\xb3o', 'utf-8'), (b' bar', None)]
>>> print(decode_header('=?iso-8859-1?q?hello_f=F3o_bar?='))
[(b'hello f\xf3o bar', 'iso-8859-1')]
>>>
>>> # Assuming not, this is a much saner interface
>>> print(decode_header_to_string('hello =?utf-8?B?ZsOzbw==?= bar'))
hello fóo bar
>>> print(decode_header_to_string('=?iso-8859-1?q?hello_f=F3o_bar?='))
hello fóo bar

(Closes #30548 and replaces it.)

• Issue: The decode_header() function decodes raw part to bytes or str, depending on encoded part #67022

[ghost]

All commit authors signed the Contributor License Agreement.

[warsaw]

In general, I think this would help users of the legacy API, although I think we should also steer people to the new API. What does @bitdancer think?

[warsaw]

How about adding a link to the non-legacy API, or an example using that newer API?

[dlenski]

Do you mean the non-legacy API as described in https://docs.python.org/3/library/email.html, e.g. email.parser?

To my knowledge, there is not any function/method in that API which can be straightforwardly used instead of email.header.decode_header.

[bitdancer]

>>> from email.headerregistry import HeaderRegistry
>>> decoder = HeaderRegistry()
>>> decoder('To', '=?utf-8?q?M=C3=A4x?= <[email protected]>')
'Mäx <[email protected]>'
>>> decoder('To', '=?utf-8?q?M=C3=A4x?= <[email protected]>').addresses
(Address(display_name='Mäx', username='foo', domain='bar.com'),)

You really don't want to use the legacy decode_header. It has many bugs that the new API fixes.

[bitdancer]

Obviously this needs to be better documented...

[dlenski]

I have made the requested changes; please review again, @warsaw.

And if you don't make the requested changes, you will be put in the comfy chair!

😂

[dlenski]

I have made the requested changes; please review again