Adjust to ixmp4 versions v0.16.0 and higher

[meksor]

Follows the recent refactor of ixmp4 and cleans up existing work to prefer using the "facade layer" (ixmp4.core) wherever possible. There are still a few missing features and uses of private functionality. I had an LLM summarize all the relevant comments as most of them were not written by me. I think many points are moot, but it might serve as a reference later:

Output

1. Upstream ixmp4 Features Needed

These are the gaps that look hard or wrong to solve purely in ixmp, because ixmp4.py is already compensating with placeholders, private API calls, or lossy behavior.

• Hierarchical metadata storage and retrieval. ixmp expects metadata at model, scenario, and model-scenario-version levels, plus inheritance and strict reads. ixmp4 currently appears to store only run-level meta, which blocks most of test_meta.py.
• Public lock lifecycle APIs. ixmp currently relies on private ixmp4 methods for checkout and discard, noted in ixmp4.py and ixmp4.py. A stable public lock, unlock, and revert/discard API would remove fragile backend coupling.
• Proper run audit fields. get_scenarios() and last_update() need real created/updated/locked metadata instead of placeholders, as called out in ixmp4.py and ixmp4.py.
• Platform-level timeslice model, or an equivalent compatibility API. ixmp expects predefined timeslices and explicit add/list semantics; ixmp4 currently infers them from IAMC categorical datapoints, which does not match the contract in test_platform.py.
• Region synonym support. The backend currently discards synonym and partly discards parent semantics in ixmp4.py, and the related tests remain xfailed in test_platform.py and test_timeseries.py.
• Better run-scoped item repository behavior. The backend works around an ixmp4 listing issue noted in ixmp4.py. If repository list() were correctly run-scoped by default, ixmp would not need defensive filtering everywhere.
• Stronger model-item constraints and clearer uniqueness semantics. ixmp assumes item names are effectively unique across item classes in a scenario, while ixmp4 only enforces uniqueness per repository, as noted in ixmp4.py.
• Bulk docs/meta mutation helpers. The current code loops item-by-item and uses backend-level deletion tables; see ixmp4.py and ixmp4.py.
• IAMC-side partial solution removal. clear_solution(from_year=...) cannot be faithfully implemented until ixmp4 exposes removal semantics beyond optimization-only state; see ixmp4.py.
• More complete GDX import support. The helper in ixmp4_io.py still assumes required items already exist, ignores solution-status checking, and only partially mirrors ixmp_source behavior; see ixmp4_io.py and ixmp4_io.py.

2. ixmp-Side Compatibility Shims

These are gaps that look feasible to improve inside ixmp without waiting on new ixmp4 APIs, though some are tradeoffs rather than perfect fixes.

• Stop hardcoding scenario-list placeholders where ixmp can derive values itself. scheme is already stored in run meta, so get_scenarios() can likely surface that immediately instead of "IXMP4Run" from ixmp4.py. The remaining audit fields could stay blank or None rather than fake strings.
• Make clone semantics stricter and clearer in ixmp. The backend can reject unsupported cross-platform or first_model_year clone modes with explicit exceptions instead of warning-and-continue behavior from ixmp4.py.
• Normalize run_id() semantics at the ixmp layer. If ixmp conceptually wants a stable run identifier rather than version, ixmp can map to run.id instead of run.version, provided tests and callers are updated consistently. The current implementation in ixmp4.py looks like a compatibility shortcut, not a requirement.
• Improve last_update() behavior in ixmp even before upstream support. Returning None would be more honest than the invalid placeholder date currently shown in ixmp4.py.
• Emulate region synonyms in ixmp-owned metadata tables. If true ixmp4 synonym support is absent, ixmp could maintain a lightweight synonym mapping in run or platform docs/meta and apply it during lookup and filtering. That would not be perfect, but it could satisfy some read/write expectations without touching ixmp4 internals.
• Emulate platform timeslices in ixmp configuration or reserved metadata. A shim could maintain a synthetic registry for add_timeslice() and timeslices() so tests and platform APIs behave more like JDBC, even if ixmp4 still only materializes actual usage through datapoints.
• Tighten backend-side item lookup rules. ixmp can proactively reject ambiguous duplicate names across item types instead of assuming JDBC-like uniqueness in _find_item() at ixmp4.py.
• Improve CSV and URL compatibility in ixmp CLI. The xfailed CLI items in test_cli.py and test_cli.py look partly like ixmp-side argument and URL-shaping issues, not purely ixmp4 storage issues.
• Make GDX import more self-healing in ixmp before calling ixmp4. ixmp can inspect the requested equ_list and var_list, initialize missing items for MESSAGE-like scenarios, and then hand off to the existing reader. That would reduce the gap noted in ixmp4_io.py without requiring new ixmp4 primitives.
• Replace fake compatibility strings with explicit warnings or NotImplementedError in narrow places. In a few methods, honest partial support is better than fabricated JDBC-shaped data.

How to review

Read the diff and note that the CI checks all pass. Ensure old and new code snippets in encapsulated logic (i.e. functions) still expresses the same intent.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update release notes.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[khaeru]

I had an LLM summarize all the relevant comments as most of them were not written by me.

Beyond code, can you please give a clear indication what code was written by you, and which is output from your LLM(s)? This would help me detect when changes convey intent vs. not.

I notice failures of:

• some "pytest" CI workflow jobs, at the iiasa/actions/setup-gams step. I think this is unrelated to the changes in the PR, and will investigate/address separately.
• other "pytest" jobs here; this appears to be because the pytest.yaml workflow is not yet updated to drop the force-installation of the branch for Fixes for ixmp support September/October 2025 ixmp4#208. I think we can simply drop this line.
• the "code quality" job, in particular mypy. Could you please have a look and let me know if these are indications of places where the current PR is still incomplete and will need further work?

I will also at some point rewrite the branch history to:

• Ensure the commit messages to conform to our code style.
• Drop the merge commit from main into the PR branch/rebase as needed.

[meksor]

No actual LLM /code/ here, only that summary.
Failures: I'll look at the mypy issues next week, as for the other failures: I am not sure which parts of the tests already worked and which didn't, but I am pretty sure any code that relies on missing features def. won't have worked so I tried to make everything else work.

I looked through the top level md files for guidelines like merge strategy and commit format, but I must have missed it. Do you have a link for future reference?

[meksor]

Looks like i forgot about test_ixmp4_io.py and there was a comment that was too long. Mypy still fails however, I think due to a wrongly cached dependency. pytest.yaml:157 says "Comment this line to force clear". Is the idea that I push a commit with that line commented out?

Unrelated to that, locally I still get a few errors:

(.venv) wolschlager@nb-wolschlager:~/Code/ixmp$ mypy ixmp
/home/wolschlager/Code/ixmp/ixmp/backend/io.py:226: error: "ExcelFile" has no attribute "parse"  [attr-defined]
/home/wolschlager/Code/ixmp/ixmp/backend/io.py:228: error: Unused "type: ignore" comment  [unused-ignore]
/home/wolschlager/Code/ixmp/ixmp/backend/io.py:238: error: "ExcelFile" has no attribute "parse"  [attr-defined]
/home/wolschlager/Code/ixmp/ixmp/backend/io.py:245: error: "ExcelFile" has no attribute "parse"  [attr-defined]
/home/wolschlager/Code/ixmp/ixmp/backend/io.py:248: error: Returning Any from function declared to return "DataFrame"  [no-any-return]
/home/wolschlager/Code/ixmp/ixmp/tests/backend/test_ixmp4.py:66: error: Unused "type: ignore[arg-type]" comment  [unused-ignore]
/home/wolschlager/Code/ixmp/ixmp/tests/core/test_scenario.py:939: error: Argument 1 to "from_records" of "DataFrame" has incompatible type "dict[str, object]"; expected "ndarray[tuple[int, int], dtype[Any]] | Iterable[SequenceNotStr[Any]] | Iterable[Mapping[Never, Any]]"  [arg-type]
Found 7 errors in 3 files (checked 59 source files)

Most of them are in files I did not touch. One is because one of two specifiers in a mypy-ignore comment is unused, I added one specifier because the one that was already there did not catch my error. I assume that this has something to do with different mypy versions or the type hints were still a work-in-progress, so I will leave it like this.

[khaeru]

No actual LLM /code/ here, only that summary.

OK, thank for the details. Knowing such things really lowers the cognitive load of review, because I know what to treat as reliable vs. not.

I looked through the top level md files for guidelines like merge strategy and commit format, but I must have missed it. Do you have a link for future reference?

We have a common set of pages that apply to message_ix, ixmp, message-ix-models, and message_data: https://docs.messageix.org/en/latest/contributing.html. CONTRIBUTING.rst should link to that URL.

Unrelated to that, locally I still get a few errors: [...]

These predate your PR, see e.g. here. I suspect (have not had time to check) these are due to a new release of pandas or pandas-stubs (which is not necessarily synced with pandas per se). I'd say these are also out of scope for this PR; I'll try to make another one to fix them, then we can rebase this branch.

pytest.yaml:157 says "Comment this line to force clear". Is the idea that I push a commit with that line commented out?

You could, yes, but let's see if that happens as a result of the other things.