Investigate how to handle optional values with `mypy` (e.g. for `lxml`)

[jtherrmann]

Jira: https://asfdaac.atlassian.net/browse/TOOL-3443

Note: The above link is accessible only to members of ASF.

---

A common error when running mypy on a project that uses lxml is that many of the lxml methods for retrieving information from XML trees return optional values (i.e. a value that can be None or some other type such as _Element or str). This causes mypy to report many union-attr errors such as Item "None" of "Optional[_Element]" has no attribute "text".

The only "correct" solution that I'm aware of is to use type narrowing techniques such as assert value is not None or cast(_Element, value). This leads to a lot of verbose code for no real benefit, because in most cases we've tested the code enough to be confident that optional values will not be None at runtime, or we're fine with runtime errors.

In e.g. ASFHyP3/burst2safe#117 we use top-level # mypy: disable-error-code="union-attr" comments to disable that error code for files that report many occurrences. This is probably not a great idea since that's a useful error code that we don't want to completely ignore, but the alternative is to use hundreds of # type: ignore[union-attr] comments throughout the code, which is visually distracting. I'm also not aware of a middle-ground between disabling the error code for an entire file vs. ignoring the error for a single line (it would be nice if the disable/ignore could be scoped to a class or function).

It would be nice to investigate this issue further to see if there's a recommended solution. The lxml library has official type stubs that can be automatically installed via mypy, which makes me think that using mypy with lxml must be fairly common, so other people must have come up with better solutions for this problem.

This issue applies to all usage of optional values with mypy, though so far it's only been a significant inconvenience with lxml. It seems like Python's techniques for type-narrowing optional values are quite verbose compared to something like Rust's unwrap. Raising exceptions seems like the more Pythonic way of signalling errors as opposed to returning None, but if some libraries such as lxml make heavy use of optional values, then it would be nice to find a way to handle them in a non-verbose way.

[jtherrmann]

I'm realizing that union-attr likely always corresponds to a potential AttributeError, so I'm not sure I see any benefit to attempting to address this particular mypy error code for Optional types where None is used to signal an error. Sure, it lets you know that there's a potential error that you might want to handle, but it's not like we handle 100% of all exceptions that might be raised by any operation. So I'm not really concerned about ignoring the error code for a file, though it would still be nice if there were a way to scope the ignore to a particular class/function/method, or even a module (e.g. lxml).

[jtherrmann]

Also asked about this at lxml/lxml-stubs#103