Skip to content

Conversation

@glatterf42
Copy link
Member

@glatterf42 glatterf42 commented Oct 14, 2025

This PR expands the functionality of IXMP4Backend, bringing it closer to full parity with JDBCBackend. It does so by:

  • adding implementations for some functions
  • clarifying that some functions won't be supported
  • migrating the ixmp4 tests to running on a PostgreSQL instance (enabling full functionality from ixmp4 in terms of versioning and rollback logic)

Setting up Postgres on GitHub Actions

The GHA runners only provide Postgres installations on Windows and Linux, but not on Macos. Thus, I'm trying to use this third-party action, which works on all OSs and is actively maintained.

How to review

  • Read the #NOTE, #TODO and #FIXME marks in the diff and respond if necessary.
  • Note that the CI checks all pass.
  • Check the pytest marks @pytest.mark.ixmp4, @pytest.mark.ixmp4_not_yet, and @pytest.mark.ixmp4_never to see if the usage/explanations are clear.
  • Read the documentation of differences between ixmp4 and JDBC and see if it's clear or if anything might be missing.
  • Respond to comments I made on the files here on GitHub.

TBD:

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests ; coverage checks both ✅ (see below)
  • Add, expand, or update documentation.
  • Update release notes.

@glatterf42 glatterf42 added this to the 3.12 milestone Oct 14, 2025
@glatterf42 glatterf42 self-assigned this Oct 14, 2025
@glatterf42 glatterf42 added enh New features & functionality backend.ixmp4 Interaction with ixmp4 via IXMP4Backend labels Oct 14, 2025
@glatterf42 glatterf42 force-pushed the enh/implement-more-ixmp4backend-functions branch from 97f6c44 to 74154b8 Compare October 14, 2025 13:35
@glatterf42 glatterf42 force-pushed the enh/implement-more-ixmp4backend-functions branch from 74154b8 to 642d21e Compare October 14, 2025 13:49
@glatterf42 glatterf42 force-pushed the enh/implement-more-ixmp4backend-functions branch 3 times, most recently from 10382f5 to cad9d8c Compare October 21, 2025 13:53
@codecov
Copy link

codecov bot commented Oct 21, 2025

Codecov Report

❌ Patch coverage is 96.81529% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.2%. Comparing base (bae4917) to head (95f8d72).
⚠️ Report is 49 commits behind head on main.

⚠️ Current head 95f8d72 differs from pull request most recent head 00d2867

Please upload reports for the commit 00d2867 to get more accurate results.

Files with missing lines Patch % Lines
ixmp/testing/__init__.py 85.9% 9 Missing ⚠️
ixmp/backend/ixmp4.py 97.0% 6 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #601     +/-   ##
=======================================
- Coverage   98.5%   98.2%   -0.3%     
=======================================
  Files         51      51             
  Lines       6345    6557    +212     
=======================================
+ Hits        6251    6444    +193     
- Misses        94     113     +19     
Files with missing lines Coverage Δ
ixmp/_config.py 94.5% <ø> (ø)
ixmp/backend/base.py 99.5% <100.0%> (+<0.1%) ⬆️
ixmp/backend/ixmp4_io.py 97.1% <100.0%> (+<0.1%) ⬆️
ixmp/backend/jdbc.py 97.3% <100.0%> (-0.1%) ⬇️
ixmp/core/scenario.py 99.2% <100.0%> (-0.4%) ⬇️
ixmp/core/timeseries.py 97.4% <100.0%> (+<0.1%) ⬆️
ixmp/tests/backend/test_base.py 98.0% <100.0%> (ø)
ixmp/tests/backend/test_ixmp4.py 100.0% <100.0%> (ø)
ixmp/tests/backend/test_ixmp4_io.py 100.0% <100.0%> (ø)
ixmp/tests/backend/test_jdbc.py 100.0% <100.0%> (ø)
... and 12 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

glatterf42 and others added 22 commits November 18, 2025 14:21
* Implement caching behaviour correctly
* Handle meta= argument to timeseries functions
* Handle from_year argument to clear_solution()
* Reraise dedicated IXMP4 errors as expected Exceptions
* Fix various small things to satisfy the test suite
* Remove outdated 'jdbc_only' marker
* Use 'jdbc' and 'ixmp4' markers consistently
* Add 'ixmp4_never' and 'ixmp4_not_yet' markers
* Enable numerous tests for ixmp4
* Clarify remaining markers limiting ixmp4 tests
* Reduce re-usage of fixtures
- Organize changes according to public API methods.
- Use higher-level heading.
- Link to existing descriptions in doc/data-model.rst instead of
  repeating these.
- Change reST links to non-public ixmp_source code to comments.
- Move implementation details that do not change user API to
  IXMP4Backend method docstrings.
- Use :attr:/:meth: reST roles instead of :func: where needed.
@khaeru
Copy link
Member

khaeru commented Nov 18, 2025

Here is the run in which all checks pass.

@khaeru khaeru force-pushed the enh/implement-more-ixmp4backend-functions branch from 95f8d72 to 00d2867 Compare November 18, 2025 13:23
@khaeru khaeru merged commit 1bf6d34 into main Nov 18, 2025
18 checks passed
@khaeru khaeru deleted the enh/implement-more-ixmp4backend-functions branch November 18, 2025 13:23
@khaeru khaeru mentioned this pull request Nov 18, 2025
5 tasks
@glatterf42
Copy link
Member Author

Thanks for picking this up and pushing it over the finish line :)

@khaeru
Copy link
Member

khaeru commented Nov 19, 2025

  • I note that it is no longer possible to run pytest at all without exactly the right local PostgreSQL installation. In a future PR, I'd:

    • document this requirement, and
    • try to relax this requirement, letting tests simply fail if the required database service is not available.

It turns out I/we should have done this before merging, as today workflows in message-ix, message-ix-models, and message_data are failing with the updated main branch.

I think this happens because ixmp.testing is used as a pytest plugin in those test suites, so ixmp.testing.pytest_sessionstart is run.

I'll open a PR to fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend.ixmp4 Interaction with ixmp4 via IXMP4Backend enh New features & functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants