Enhance functionality of IXMP4Backend

.github/workflows/pytest.yaml

@@ -81,13 +81,28 @@ jobs:
         version: ${{ matrix.gams-version }}
         license: ${{ secrets.GAMS_LICENSE }}
 
+    - uses: ikalnytskyi/action-setup-postgres@v8
+      with:
+        # username: postgres
+        # password: postgres
+        database: ixmp_test
+        # port: 5432
+        postgres-version: "16"
+        # ssl: true
+      id: setup-postgres
+
     - name: Set RETICULATE_PYTHON
       # Retrieve the Python executable set up above
       run: echo "RETICULATE_PYTHON=$(uv python find)" >> $GITHUB_ENV
 
     - name: Install the package and dependencies
+      # TEMPORARY Use branch for https://github.com/iiasa/ixmp4/pull/208
       # [docs] → [tests] → [ixmp4,report,tutorial]
-      run: uv pip install .[docs]
+      run: |
+        uv pip install \
+            "ixmp4 @ git+https://github.com/iiasa/ixmp4@enh/fixes-for-ixmp-support-september-2025; python_version > '3.9'" \
+            .[docs]
+      shell: bash
 
     - name: "Install libpng-dev"  # for R 'png', required by reticulate
       if: startsWith(matrix.os, 'ubuntu-')
@@ -104,6 +119,13 @@ jobs:
         IRkernel::installspec()
       shell: Rscript {0}
 
+    - name: Setup template & default-ixmp4-local Postgres schema
+      if: matrix.python-version != '3.9'
+      run: |
+        ixmp4 platforms add --dsn "postgresql+psycopg://postgres:postgres@localhost:5432/template1" template1
+        ixmp4 platforms add --dsn "postgresql+psycopg://postgres:postgres@localhost:5432/ixmp_test" ixmp_test
+        IXMP4_MANAGED=false ixmp4 platforms upgrade
+
     - name: Run tests
       run: |
         pytest ixmp \

.pre-commit-config.yaml

@@ -7,7 +7,8 @@ repos:
     additional_dependencies:
     - genno
     - GitPython
-    - "ixmp4 >= 0.13, < 0.14"
+    # TEMPORARY Use branch for https://github.com/iiasa/ixmp4/pull/208
+    - "ixmp4 @ git+https://github.com/iiasa/ixmp4@enh/fixes-for-ixmp-support-september-2025; python_version > '3.9'"
     - nbclient
     - pandas-stubs
     - pytest

RELEASE_NOTES.rst

@@ -38,6 +38,7 @@ All changes
 - Improve :class:`.IXMP4Backend` (:pull:`581`):
 
   - Support creation and modification of 0-dimensional parameters (:class:`ixmp4.Scalar`).
+  - Streamline and fix various functions to enable more tests to pass (:pull:`601`).
 
 - New method :meth:`.Scenario.iter_par_data` (:pull:`581`).
   :meth:`.Scenario.items` no longer supports iterating over item *contents*.
@@ -60,6 +61,7 @@ All changes
   Calling this method emits :class:`DeprecationWarning`,
   and the method will be removed in a future version of :mod:`ixmp`.
 - Improve performance of :meth:`.Scenario.remove_par` and :meth:`.Scenario.remove_set` (:pull:`598`).
+- Document all differences between :class:`.JDBCBackend` and :class:`.IXMP4Backend` (:pull:`601`).
 
 .. _v3.11.1:
 

doc/api-backend.rst

@@ -80,7 +80,21 @@ IXMP4Backend
 ------------
 
 .. note::
-   As of version 0.10, `ixmp4 <https://github.com/iiasa/ixmp4/>`_ supports only Python 3.10 and above.
+   See :ref:`jdbc-vs-ixmp4`, below.
+
+   As of :mod:`ixmp` version 3.11,
+   IXMP4Backend uses :mod:`ixmp4` version 0.10.
+   This is a development release;
+   a 'stable' version 1.0 or later has not yet been released.
+   For this reason:
+
+   - The ixmp4 API is not yet finalized and may change at any time.
+   - Complete documentation of the ixmp4 API itself is not yet available.
+
+   The same holds for IXMP4Backend.
+   Consequently you **may** but probably **should not** use it for 'production' scientific scenario work.
+
+   As of version 0.10, :mod:`ixmp4` supports only Python 3.10 and above.
 
    - If you want to use IXMP4Backend,
      please ensure you are using a sufficiently recent Python version.
@@ -92,27 +106,232 @@ IXMP4Backend
    :exclude-members: IXMP4Backend
 
 .. autoclass:: ixmp.backend.ixmp4.IXMP4Backend
+   :members:
+
+.. automodule:: ixmp.util.ixmp4
+   :members:
 
-   .. note::
-      As of ixmp version 3.11,
-      IXMP4Backend has only *partial* support for the APIs of :class:`ixmp.Platform`,
-      :class:`ixmp.TimeSeries`, :class:`ixmp.Scenario`, and :py:`message_ix.Scenario`,
-      and may not be as performant as JDBCBackend.
-      See `Support roadmap for ixmp4 <https://github.com/iiasa/message_ix/discussions/939>`_
-      and `issues labeled 'backend.ixmp4' <https://github.com/iiasa/ixmp/labels/backend.ixmp4>`_
-      for details of future work to expand support and improve performance.
+.. _jdbc-vs-ixmp4:
 
-      Because IXMP4Backend uses ixmp4 version 0.10 (that is, prior to a 1.0 'stable' release):
+Differences between JDBCBackend and IXMP4Backend
+================================================
 
-      - The ixmp4 API is not yet finalized and may change at any time.
-      - Complete documentation of the ixmp4 API itself is not yet available.
+The long-term goal is that :mod:`ixmp` used with IXMP4Backend
+has **feature and performance parity** with JDBCBackend.
+As of ixmp version 3.12, IXMP4Backend has support for *most* features.
+This section lists known differences in behaviour when using IXMP4Backend,
+compared to JDBCBackend.
+This includes cases where:
 
-      Consequently you **may** but probably **should not** use it for 'production' scientific scenario work.
+1. Behaviour is the same, but performance is known to differ.
+2. IXMP4Backend currently has different behaviour,
+   but work is planned to address the difference.
+3. IXMP4Backend has different behaviour, and this difference is permanent.
 
-   IXMP4Backend supports storage in local, SQLite databases.
+   This occurs where :mod:`ixmp4` intentionally omits certain functionality
+   or implements different behaviour,
+   IXMP4Backend does not cover the difference,
+   and no work is planned to change this.
+   In some cases, this is because inspection of publicly-visible :mod:`ixmp` user code
+   shows no usage of certain features.
+   The text “Not in visible use” is used.
 
-.. automodule:: ixmp.util.ixmp4
-   :members:
+For (1) and (2),
+code that works with JDBCBackend will work or should work with IXMP4Backend.
+For (3), code may need to be tested and adapted.
+
+The differences are listed according to components of the :doc:`ixmp Python API <api>`,
+with individual items and class attributes and methods in alphabetical order.
+Behaviours that are identical are not mentioned.
+For planned future work see the `support roadmap for ixmp4 <https://github.com/iiasa/message_ix/discussions/939>`_
+and `issues labeled 'backend.ixmp4' <https://github.com/iiasa/ixmp/labels/backend.ixmp4>`_.
+
+.. NOTE What do we do about major difference such as Run being the new Scenario, IndexSet and Table vs Set, etc?
+.. Functionally, all things are equivalent, but these *are* changes we'd like people to adjust to eventually.
+
+.. contents::
+   :backlinks: none
+   :local:
+
+General behaviour
+-----------------
+
+Command-line interface (CLI)
+   Some CLI tests do not pass yet with IXMP4Backend, but all should.
+   In the meantime, use care with CLI commands including
+   :program:`ixmp platform list` and :program:`ixmp solve`.
+
+Default contents
+   JDBCBackend pre-populates Platform and Scenario instances with certain contents.
+   On IXMP4Backend, the configuration option :attr:`.ixmp4.Options.jdbc_compat`
+   controls whether these default contents are created (:any:`True`) or not.
+   See the sections below for further details.
+
+   As of :mod:`ixmp` v3.12, these default contents only include items necessary
+   for the test suite and tutorials.
+
+Exceptions
+   JDBCBackend raises exceptions from the Python standard exception hierarchy
+   (:class:`RuntimeError`, :class:`ValueError`, :class:`KeyError`, etc.),
+   or occasionally a generic :class:`IxException`.
+   ixmp4 provides a large number of exception classes with specific meanings
+   such as :class:`.RunLockRequired`, :class:`.NoDefaultRunFound`, or
+   :class:`.OptimizationDataValidationError`
+   that are not subclasses of the same standard exceptions.
+   In cases covered by the ixmp test suite,
+   IXMP4Backend intercepts ixmp4-specific exceptions
+   and re-raises the same classes of exceptions as JDBCBackend.
+   However:
+
+   - Exception messages may differ.
+   - In other cases, ixmp4 exceptions may be raised directly.
+
+   Users **should** inspect code that uses Python :py:`try:` statements,
+   :func:`isinstance` calls, or similar, to ensure it has the same behaviour
+   when using IXMP4Backend.
+   Code **may** be adapted by replacing single except types with tuples of types:
+
+   .. code-block::
+
+      try:
+          ...
+      # except RuntimeError:        # Old code: single exception type
+      except (RuntimeError, ixmp4.exceptions.RunLockRequired):  # New
+          ...
+
+Platform class
+--------------
+
+Default contents
+   If :attr:`.ixmp4.Options.jdbc_compat` is :any:`True`,
+   regions and units are populated.
+   See :issue:`608` for details.
+
+:meth:`.Platform.add_region`
+   - ixmp4 and IXMP4Backend do not and will not support the :py:`parent`
+     argument.
+     If passed, the argument is not stored, and warnings are logged.
+   - IXMP4Backend supplies a default, meaningless :py:`hierarchy` value
+     if no argument is given.
+
+:meth:`.Platform.add_region_synonym`
+   ixmp4 and IXMP4Backend do not and will not support 'synonyms' for region IDs.
+
+:meth:`.Platform.add_timeslice`, :meth:`.Platform.timeslices`
+   Not in visible use.
+   ixmp4 does not and will not provide a dedicated way to handle :ref:`data-timeslice`
+   for :ref:`data-tsdata`, so these methods are not supported on IXMP4Backend.
+
+   When using ixmp4 directly,
+   the :class:`ixmp4.data.db.iamc.datapoint.DataPoint` class
+   and its :py:`type`, :py:`step_category`, :py:`step_year`, and
+   :py:`step_datetime` methods can be used to define time slices.
+
+:meth:`Platform.close_db <.base.Backend.close_db>` called twice
+    JDBCBackend logs a warning if this method is called
+    after the database connection has already been closed.
+    IXMP4Backend does not;
+    if the connection was already closed, the call has no effect.
+
+    .. On IXMP4Backend there is no straightforward way to provide this:
+       sqlalchemy does not provide a way to check if a database session
+       or engine was closed.
+
+:meth:`.Platform.export_timeseries_data`
+   On JDBCBackend, the "meta" column (:py:`meta` parameter to :meth:`.TimeSeries.add_timeseries`)
+   is exported as :type:`int` 0 or 1.
+   On IXMP4Backend, it is exported as the string "False" or "True".
+
+   .. NB see https://github.com/iiasa/ixmp_source/blob/889b51f7731b3fdfed2e241c3d6596723e83202e/src/main/resources/db/migration/postgresql/V1__postgresql_base_version.sql#L219
+
+:meth:`Platform.get_doc <.Backend.get_doc>`, :meth:`set_doc <.Backend.set_doc>`
+   On JDBCBackend, the argument :py:`domain="metadata"` is supported.
+   On IXMP4Backend, it is not and will not be supported.
+
+   Code that will use only IXMP4Backend **may** set values on :attr:`ixmp4.core.run.Run.meta`.
+
+:meth:`Platform.get_meta <.Backend.get_meta>`, :meth:`remove_meta <.Backend.remove_meta>`, :meth:`set_meta <.Backend.set_meta>`
+   On JDBCBackend, the :py:`meta=...` parameter can be a mapping of :class:`str` keys to arbitrary Python values,
+   including :class:`list` and other collections.
+   On IXMP4Backend, collections are not supported.
+
+   On JDBCBackend, any one or two of the :py:`model=...`, :py:`scenario=...`, or :py:`version=...` parameters
+   may be given.
+   On IXMP4Backend, this is not supported; all 3 parameters are required.
+
+:meth:`.Platform.set_log_level`
+    On JDBCBackend, this sets the log level of the Python logger named :py:`"ixmp.backend.jdbc"`
+    and the underlying Java code.
+    On IXMP4Backend, only the log level of the logger named :py:`"ixmp.backend.ixmp4"` is changed,
+    and not the log level of any underlying code.
+    User code **may** use standard :mod:`logger` methods to access and
+    change the level of loggers within :mod:`ixmp4` or its dependencies:
+
+    .. code-block:: python
+
+       import logging
+
+       log = logging.getLogger("ixmp4")  # or other module
+       log.setLevel(logging.DEBUG)  # or other level
+
+TimeSeries class
+----------------
+
+:attr:`.TimeSeries.model`, :attr:`.TimeSeries.scenario`
+   On ixmp4 and IXMP4Backend these names may be up to 255 characters long
+   (see `here <https://github.com/iiasa/ixmp4/blob/main/ixmp4/db/migrations/versions/c71efc396d2b_initial_migration.py#L38>`__).
+   On JDBCBackend the maximum is 1000 characters.
+
+   .. NB see https://github.com/iiasa/ixmp_source/blob/889b51f7731b3fdfed2e241c3d6596723e83202e/src/main/resources/db/migration/postgresql/V1.31__model_scenario_names.sql
+
+:attr:`.TimeSeries.version`
+   On IXMP4Backend, a new, uncommitted Timeseries has :py:`version = 1`.
+   (ixmp4 sets a value of 0, but IXMP4Backend always stores certain metadata,
+   which increases the value to 1.)
+   On JDBCBackend, the value is -1.
+
+:meth:`.TimeSeries.add_geodata`, :meth:`.get_geodata`, and :meth:`.remove_geodata`
+   Not in visible use.
+   ixmp4 does not support storing and retrieving geodata,
+   so these methods are and will not be supported on IXMP4Backend.
+
+:meth:`.TimeSeries.add_timeseries`
+   On IXMP4Backend, ‘variable’ column entries may be up to 255 characters long
+   (see `here <https://github.com/iiasa/ixmp4/blob/main/ixmp4/db/migrations/versions/c71efc396d2b_initial_migration.py#L24>`__).
+   On JDBCBackend, the maximum is 256 characters.
+
+   .. NB see https://github.com/iiasa/ixmp_source/blob/889b51f7731b3fdfed2e241c3d6596723e83202e/src/main/resources/db/migration/postgresql/V1__postgresql_base_version.sql#L184
+
+:meth:`.TimeSeries.last_update`
+   On IXMP4Backend, the return value is never :any:`None`.
+   For newly-created TimeSeries objects, it is the date-time of creation.
+
+:meth:`.TimeSeries.transact`
+   JDBCBackend logs certain warnings when exceptions occur within a :py:`with ts.transact(): ...` block.
+   ixmp4/IXMP4Backend does not yet emit the same warnings.
+
+Scenario class
+--------------
+
+Default contents 
+   If :attr:`.ixmp4.Options.jdbc_compat` is :any:`True`,
+   the sets ``technology`` and ``year`` are created,
+   matching the behaviour of JDBCBackend.
+
+:attr:`Scenario.version <.TimeSeries.version>`
+   On IXMP4Backend, a new, empty Scenario has :py:`version = 1`.
+   On JDBCBackend, the value is 0.
+
+:meth:`.Scenario.par`
+   On IXMP4Backend, data returned as :class:`pandas.DataFrame`
+   have "unit" and "value" as the last 2 columns, in that order.
+   On JDBCBackend, the columns are in the order "value", "unit".
+
+   .. The former aligns with the common order in the IAMC data format.
+      The latter “seems more aligned with natural language”.
+
+   To accommodate this difference,
+   user code **should** access the columns by name, rather than by index.
 
 .. currentmodule:: ixmp.backend
 

doc/conf.py

@@ -118,6 +118,7 @@ def local_inv(name: str, *parts: str) -> str | None:
 intersphinx_mapping = {
     "dask": ("https://docs.dask.org/en/stable/", None),
     "genno": ("https://genno.readthedocs.io/en/latest/", (local_inv("genno"), None)),
+    "ixmp4": ("https://docs.ece.iiasa.ac.at/projects/ixmp4/en/latest/", None),
     "jpype": ("https://jpype.readthedocs.io/en/stable", None),
     "message_ix": ("https://docs.messageix.org/en/latest/", None),
     "message-ix-models": (

doc/data-model.rst

@@ -195,6 +195,8 @@ Documentation
 Data associated with a TimeSeries object
 ========================================
 
+.. _data-tsdata:
+
 Time series data
 ----------------
 
@@ -210,6 +212,10 @@ Time series data
   - **unit**: a value from the “Unit” list (see :ref:`codelists`).
   - **subannual**: a value from the “Sub-annual time slices” list (see :ref:`codelists`).
   - **meta**: a boolean value.
+    If set, the time series data are handled specially in 2 ways:
+
+    1. The data are not removed by :meth:`.Scenario.remove_solution`.
+    2. The data are preserved by :meth:`.Scenario.clone` regardless of the :py:`keep_solution=...` parameter.
 
     .. note:: This is distinct from :ref:`data-meta`, above.
 

ixmp/_config.py

@@ -87,12 +87,18 @@ def _platform_default() -> dict[str, "str | PlatformInitKwargs"]:
             "driver": "hsqldb",
             "path": next(_iter_config_paths())[1].joinpath("localdb", "default"),
         },
-        "ixmp4-local": {
+        "sqlite-local": {
             "class": "ixmp4",
             "dsn": f"sqlite:///{ixmp4_databases.joinpath('local.sqlite3')}",
             "ixmp4_name": "local",
             "jdbc_compat": True,
         },
+        "ixmp4-local": {
+            "class": "ixmp4",
+            "dsn": "postgresql+psycopg://postgres:postgres@localhost:5432/ixmp_test",
+            "ixmp4_name": "ixmp_test",
+            "jdbc_compat": True,
+        },
     }