Enable strict mypy settings

[glatterf42]

This PR enables settings that are equivalent to mypy --strict and even slightly more :)

There is still room for improvement and some cleanups, I think, but all tests should be passing already.
There are some dependencies that are not fully annotated (e.g. pint, dask, and genno) so that we have to implore some type: ignores. These feel a bit unnecessary, especially when the underlying functions are documented clearly. For example, genno.Computer.get() is documented, but not typed, so we could think of proxying it here (essentially just calling the super() method) to include type hints (I also looked at genno if I could quickly provide the required type hints there, but got lost by more and more type hints and aborted the project to save some time).

Other than that, I had to change several occurrences of scen.par() in the test suite. Per par()'s docstring (and function logic), it returns a pd.DataFrame for indexed Parameters and a dict[str, float | str] for scalar Parameters. Unfortunately, it was previously annotated to return pd.DataFrame only, which I changed now to the actual return type union. Several places in the test suite expect only a pd.DataFrame to be returned, though.
I'm not sure what the best thing to do is, here. We could maybe come up with another function for scalar Parameters (to allow scen.par() to return only pd.DataFrame), but that also requires code adjustments. Alternatively, we could adjust backend.item_get_elements()to always returnpd.DataFrameforItemType.PAR, even for Scalars. Based on existing usage, scen.par()` doesn't seem to be called often to return Scalar data.

How to review

• Read the diff and note that the CI checks all pass.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update release notes.

[codecov]

Codecov Report

❌ Patch coverage is 99.35588% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.1%. Comparing base (eeb2b2d) to head (b4f9e29).
⚠️ Report is 56 commits behind head on main.

Files with missing lines	Patch %	Lines
ixmp/backend/ixmp4.py	97.9%	2 Missing ⚠️
ixmp/backend/jdbc.py	99.3%	1 Missing ⚠️
ixmp/core/item.py	96.7%	1 Missing ⚠️
ixmp/core/scenario.py	98.7%	1 Missing ⚠️
ixmp/core/timeseries.py	97.0%	1 Missing ⚠️
ixmp/tests/backend/test_base.py	95.0%	1 Missing ⚠️
ixmp/util/__init__.py	98.4%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #581     +/-   ##
=======================================
- Coverage   98.2%   98.1%   -0.1%     
=======================================
  Files         50      51      +1     
  Lines       6063    6260    +197     
=======================================
+ Hits        5955    6143    +188     
- Misses       108     117      +9     

Files with missing lines	Coverage Δ
ixmp/__init__.py	100.0% <100.0%> (ø)
ixmp/_config.py	96.3% <100.0%> (-0.6%)	⬇️
ixmp/backend/__init__.py	100.0% <100.0%> (ø)
ixmp/backend/base.py	99.4% <100.0%> (-0.6%)	⬇️
ixmp/backend/common.py	100.0% <100.0%> (ø)
ixmp/backend/io.py	97.8% <100.0%> (+<0.1%)	⬆️
ixmp/backend/ixmp4_io.py	97.0% <100.0%> (ø)
ixmp/cli.py	100.0% <100.0%> (ø)
ixmp/core/platform.py	98.9% <100.0%> (ø)
ixmp/model/__init__.py	100.0% <100.0%> (ø)
... and 37 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[glatterf42]

By now, this PR does a few other things, too:

• fixes https://github.com/iiasa/ixmp/security/dependabot/17 (bumping requests in docs dependencies)
• enables the docs to build again by pinning pandas 2.2.3 (see Workaround gamsapi dependency bound on pandas <2.3 #582)

Somehow, even after trying extensively to update my local venv to the same versions of packages as pre-commit/mypy uses, I couldn't reproduce some of the errors locally that mypy reports here (though running uvx pre-commit mypy as we do in out check here produces the same errors locally). This is a bit frustrating as vscode/mypy now highlights these errors locally for me (about some type: ignore statements being unused), but I figure it's still best to ensure the checks are passing here (by including them). Still, I'm unsure how to resolve this discrepancy. I'm hoping that some version bump might do that soon.

[glatterf42]

Do we need to edit the release notes for this one?

[khaeru]

I've made some changes on the branch, and will try to summarize here before merging. Most of them are consequent of principles that can be roughly expressed as:

1. Type information should be simple to express and read (and should not outweigh the code it describes).
2. If type information is complicated, this indicates that code architecture can be improved.

For instance, the fact that ixmp.backend.base.Backend has methods that expect certain literal strings (instead of, say, members of some Enum like ItemType, or actual concrete types), is revealed to be a suboptimal design choice when it requires complicated type hints.

The added commits try to make changes where possible in line with those principles, while refraining from making big changes to internal and external APIs.

• In general:
  • Extend ixmp.types with additional type aliases including:
    • Simple and union return types from Scenario.par() et al.
    • Arguments like Scenario.par(…, filters=…).
  • Use these everywhere except where there are specific differences.
  • Restore the TimeSeries._backend() method so that we don't immediately break message_ix main, but add a deprecation warning.
• For IXMP4Backend:
  • For non-public methods (those part of the .base.Backend API), use ixmp4's own classes/types directly, instead of string literals. This allows e.g. _find_item() and its type hints to be much simpler.
  • Drop _get_item(). This was a case where the @overloads and type information were many times longer (20–30 lines of code) than the function body itself (1 line) and places it was used (only 2).
• For JDBCBackend:
  • Introduce ixmp.core.item.Item and concrete subclasses corresponding to .backend.common.ItemType members. These (a) make more explicit the class hierarchy followed by JDBCBackend (based on ixmp_source) and thus the original Backend API and (b) allow to simplify JDBCBackend._get_item() and its type hints.
    • Subclasses of these can used eventually to absorb the functionality of message_ix.models.Item and help us deal with the >1700 lines of message_ix.util.scenario_data.

Also some improvements:

• Handle the ixmp4 Scalar type.
• Simplify .util.diff().
• Add .util.ixmp4.is_ixmp4backend() as a type guard for internal and downstream use.
• from ixmp.types import .. only within an if TYPE_CHECKING: block. This follows usage that I see in e.g. xarray, where type checking info is separated from runtime behaviour as much as possible.

[khaeru]

Rebased after dealing with #586; I cherry picked two commits from this branch to ensure the RTD build succeed there.

[khaeru]

Were you intending to open an issue upstream to request this? Should I?

[glatterf42]

Just did :)

[khaeru]

As of 3.0.0, pyam-iamc is distributed without a py.typed marker file (see the distributions), so mypy considers the entire package to be untyped.

This is also why we list it in pyproject.toml with tool.mypy.overrides.ignore_missing_imports.

[glatterf42]

As far as I can tell, pyam never had this marker file (at least since v2.0.0). I think I was still wondering why it only showed up in this one spot.
If I find a bit of time, I may try adding a py.typed file to pyam to resolve this :)

[khaeru]

Do we need to edit the release notes for this one?

Definitely yes. The added type hints may trigger new errors in downstream packages that use ixmp and do type checking. So we need to (a) inform about what has changed, and (b) document ixmp.types to make it easier for those downstream applications to update their internal hints/checks.

[khaeru]

Unfortunately the removal of a positional argument here broke the entire message_ix test suite; see iiasa/message_ix#963. I'll make another PR to revert.

[khaeru]

#587.