Skip to content

Separate models.py into submodules #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khaeru merged 8 commits into from
Aug 21, 2025
Merged

Separate models.py into submodules #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1efe903
Change mode of core.py to match other files
khaeru Aug 19, 2025
d7acd06
Add model/README
khaeru Aug 19, 2025
204352f
Move .macro → .macro.calibrate
khaeru Aug 19, 2025
f70eb6d
Split .models to 4 submodules
khaeru Aug 19, 2025
8615910
Adjust 1st-party imports of .models contents
khaeru Aug 19, 2025
73aaa8b
Allow Scenario(…, scheme="MESSAGE-MACRO")
khaeru Aug 19, 2025
6baab9e
Rearrange and expand tests for #972
khaeru Aug 19, 2025
df1bfa9
Update docs, release notes for #972
khaeru Aug 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions RELEASE_NOTES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,24 @@
Next release
============

Migration notes
---------------

Adjust any imports like the following:

.. code-block:: python

from message_ix.models import DIMS, Item, MACRO, MESSAGE, MESSAGE_MACRO

…to:

.. code-block:: python

from message_ix.common import DIMS, Item
from message_ix.macro import MACRO
from message_ix.message import MESSAGE
from message_ix.message_macro import MESSAGE_MACRO

All changes
-----------

Expand All @@ -20,6 +38,13 @@ All changes
- Revise :ref:`equation_commodity_balance_aux` to include input and output flows based on |CAP| and |CAP_NEW|.
- New :class:`.MESSAGE` / :meth:`.Scenario.solve` option :py:`cap_comm=True` to enable this representation.

- The former module :py:`message_ix.models` is split to distinct submodules (:pull:`972`):

- :mod:`message_ix.common` includes :class:`.GAMSModel` and related code.
- :mod:`message_ix.macro` includes :class:`.MACRO`.
- :mod:`message_ix.message` includes :class:`.MESSAGE`.
- :mod:`message_ix.message_macro` includes :class:`.MESSAGE_MACRO`.

- Document the :ref:`minimum version of Java <install-java>` required for :class:`ixmp.JDBCBackend <ixmp.backend.jdbc.JDBCBackend>` (:pull:`962`).
- Improve type hinting (:pull:`963`).

Expand Down Expand Up @@ -61,7 +86,7 @@ Users **should**:
All changes
-----------

- Some MESSAGEix :doc:`tutorials <tutorials>` are runnable with the :class:`.IXMP4Backend` introduced in :mod:`ixmp` version 3.11 (:pull:`894`, :pull:`941`).
- Some MESSAGEix :doc:`tutorials <tutorials>` are runnable with the :class:`~ixmp.backend.ixmp4.IXMP4Backend` introduced in :mod:`ixmp` version 3.11 (:pull:`894`, :pull:`941`).
See `Support roadmap for ixmp4 <https://github.com/iiasa/message_ix/discussions/939>`__ for details.
- Add the :py:`concurrent=...` model option to :class:`.MACRO` (:pull:`808`).
- Adjust use of :ref:`type_tec <mapping-sets>` in :ref:`equation_emission_equivalence` (:pull:`930`, :issue:`929`, :pull:`935`).
Expand Down Expand Up @@ -174,7 +199,7 @@ Migration notes
NOTE: this may result in changes to the solution.
In order to use the previous default `lpmethod`, the user-specific default setting can be set through the user's ixmp configuration file.
Alternatively, the `lpmethod` can be specified directly as an argument when solving a scenario.
Both of these configuration methods are further explained :meth:`here <message_ix.models.GAMSModel>`.
Both of these configuration methods are further documented at :class:`.GAMSModel`.

- The dimensionality of one set and two parameters (``map_tec_storage``, ``storage_initial``, and ``storage_self_discharge``) are extended to allow repesentation of the mode of operation of storage technologies and the temporal level of storage containers.
If these items are already populated with data in a Scenario, this data will be incompatible with the MESSAGE GAMS implementation in this release; a :class:`UserWarning` will be emitted when the :class:`.Scenario` is instantiated, and :meth:`~.message_ix.Scenario.solve` will raise a :class:`ValueError`.
Expand Down Expand Up @@ -421,7 +446,7 @@ All changes
- :pull:`281`: Test and improve logic of :meth:`.years_active` and :meth:`.vintage_and_active_years`.
- :pull:`269`: Enforce ``year``-indexed columns as integers.
- :pull:`256`: Update to use :obj:`ixmp.config` and improve CLI.
- :pull:`255`: Add :mod:`message_ix.testing.nightly` and ``message-ix nightly`` CLI command group for slow-running tests.
- :pull:`255`: Add :py:`message_ix.testing.nightly` and ``message-ix nightly`` CLI command group for slow-running tests.
- :pull:`249`, :pull:`259`: Build MESSAGE and MESSAGE_MACRO classes on ixmp model API; adjust Scenario.
- :pull:`235`: Add a reporting tutorial.
- :pull:`236`, :pull:`242`, :pull:`263`: Enhance reporting.
Expand Down
47 changes: 28 additions & 19 deletions doc/api.rst
View file
Open in desktop
Copy link
Member

@glatterf42 glatterf42 Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm looking at the docs build and I'm surprised to see lots and lots of errors. Most of them already existed for the latest build, though, it seems. What's new is that doc/macro.rst complains about not being able to import various things from message_ix.macro. I'm guessing they moved somewhere (too many to list, but starting here in the build logs).

I also notice that there is an actual error about the table that contains PRICE_COMMODITY in model_core.rst, which is likely just imported from the GAMS file. This seems to be malformed, so is not rendered in our docs. While this error isn't new, it would be nice to fix it to have our docs be complete again. The error says "Bottom/header table border does not match top border.", which sounds like we just have a mismatch in the number of = used for the borders.

Copy link
Member

@glatterf42 glatterf42 Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc/install-adv.rst and doc/reporting.rst also complain about missing reference targets, as well as the RELEASE_NOTES.rst, curiously.

Copy link
Member Author

@khaeru khaeru Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll fix up these examples you mention. It would be good to have someone do a more systematic sweep, at some point.

Copy link
Member Author

@khaeru khaeru Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the table that contains PRICE_COMMODITY in model_core.rst,

This happened in #451: https://github.com/iiasa/message_ix/blame/cc014c24c7084f252f3827a86187a0ad467155a5/message_ix/model/MESSAGE/model_core.gms#L150

I've changed it to a ReST list-table, which makes it easier to edit.

doc/macro.rst complains about not being able to import various things from message_ix.macro

That file is now message_ix.macro.calibrate. Adjusted.

doc/install-adv.rst

home/khaeru/vc/iiasa/message-ix/doc/install-adv.rst:71: WARNING: py:mod reference target not found: jpype [ref.mod]

Like a lot of Sphinx packages, the intersphinx inventory for JPype1 does not actually contain a target for the top-level module:

$ python -m sphinx.ext.intersphinx https://jpype.readthedocs.io/en/stable/objects.inv | grep -B1 -A6 ^py:module
    jpype.dbapi2.JDBCType.set         : dbapi2.html#jpype.dbapi2.JDBCType.set
py:module
    jpype.beans                       : api.html#module-jpype.beans
    jpype.imports                     : api.html#module-jpype.imports
    jpype.pickle                      : api.html#module-jpype.pickle
    jpype.types                       : api.html#module-jpype.types
py:property
    jpype.dbapi2.Connection.adapters  : dbapi2.html#jpype.dbapi2.Connection.adapters

I added it to reference_aliases in doc/conf.py alongside the other such cases.

/home/khaeru/vc/iiasa/message-ix/doc/install-adv.rst:233: WARNING: py:class reference target not found: IXMP4Backend [ref.class]

This one looked like:

message_ix/doc/install-adv.rst

Line 233 in cc014c2

- ``ixmp4`` includes packages require to use :class:`ixmp.IXMP4Backend <.IXMP4Backend>`,

The shorthand whereby .IXMP4Backend will find a fully-qualified name (FQN) like ixmp.backend.ixmp4.IXMP4Backend only functions within the same project, not across projects via intersphinx. I replaced with the FQN.

doc/reporting.rst

Most of these cases occur with references that work/are valid within the genno docs, but are not properly resolved when those docs are rendered within the context of the message_ix docs. Some of those might be resolvable by copying some of the genno doc/conf.py settings. I'll omit those for now.

RELEASE_NOTES.rst

/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:43: WARNING: py:mod reference target not found: message_ix.common [ref.mod]
/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:44: WARNING: py:mod reference target not found: message_ix.macro [ref.mod]
/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:45: WARNING: py:mod reference target not found: message_ix.message [ref.mod]
/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:46: WARNING: py:mod reference target not found: message_ix.message_macro [ref.mod]

Added the appropriate targets.

/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:89: WARNING: py:class reference target not found: IXMP4Backend [ref.class]
/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:302: WARNING: py:mod reference target not found: macro [ref.mod]

Same as above.

/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:198: WARNING: py:meth reference target not found: message_ix.models.GAMSModel [ref.meth]

This was trying to use :meth: to reference a :class:. Fixed.

/home/khaeru/vc/iiasa/message-ix/RELEASE_NOTES.rst:449: WARNING: py:mod reference target not found: message_ix.testing.nightly [ref.mod]

This happens when things are removed. I resolve these by using the :py: role, which looks code-like but no longer tries to link to anything.

Copy link
Member

@glatterf42 glatterf42 Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this already looks a lot better. And agreed that a more systematic sweep would be useful, but out of scope here :)

Original file line number Diff line number Diff line change
Expand Up @@ -129,17 +129,21 @@ The full API is also available from R; see :doc:`rmessageix`.
Model classes
-------------

.. currentmodule:: message_ix.models
.. currentmodule:: message_ix

.. autosummary::

MESSAGE
MACRO
MESSAGE_MACRO
GAMSModel
DEFAULT_CPLEX_OPTIONS
Item
ItemType
~message.MESSAGE
~macro.MACRO
~message_macro.MESSAGE_MACRO
~common.GAMSModel
~common.DEFAULT_CPLEX_OPTIONS
~common.Item
~ixmp.backend.ItemType

.. currentmodule:: message_ix.common

.. automodule:: message_ix.common

.. autodata:: DEFAULT_CPLEX_OPTIONS

Expand All @@ -149,7 +153,8 @@ Model classes
:members:
:exclude-members: defaults

The :class:`.MESSAGE`, :class:`MACRO`, and :class:`MESSAGE_MACRO` child classes encapsulate the GAMS code for the core MESSAGE (or MACRO) mathematical formulation.
The :class:`.MESSAGE`, :class:`.MACRO`, and :class:`.MESSAGE_MACRO` child classes
encapsulate the GAMS code for the core MESSAGE (or MACRO) mathematical formulation.

The class receives `model_options` via :meth:`.Scenario.solve`. Some of these are passed on to the parent class :class:`ixmp.model.gams.GAMSModel` (see there for a list); others are handled as described below.

Expand Down Expand Up @@ -244,6 +249,10 @@ Model classes
* - **var_list**
- :obj:`None`

.. currentmodule:: message_ix.message

.. automodule:: message_ix.message

.. autoclass:: MESSAGE
:members: initialize
:exclude-members: defaults
Expand All @@ -267,6 +276,10 @@ Model classes
Keys are the names of items (sets, parameters, variables, and equations); values are :class:`.Item` instances.
These include all items listed in the MESSAGE mathematical specification, i.e. :ref:`sets_maps_def` and :ref:`parameter_def`.

.. currentmodule:: message_ix.macro

.. automodule:: message_ix.macro

.. autoclass:: MACRO
:members:
:exclude-members: items
Expand All @@ -283,6 +296,10 @@ Model classes
.. autoattribute:: items
:no-value:

.. currentmodule:: message_ix.message_macro

.. automodule:: message_ix.message_macro

.. autoclass:: MESSAGE_MACRO
:members:
:exclude-members: items
Expand All @@ -307,12 +324,10 @@ Model classes
.. autoattribute:: items
:no-value:

.. autodata:: DIMS
.. autoclass:: Item
.. autodata:: message_ix.common.DIMS
.. autoclass:: message_ix.common.Item
:members:

.. currentmodule:: message_ix.macro

.. _utils:

Utility methods
Expand All @@ -321,14 +336,8 @@ Utility methods
.. automodule:: message_ix.util
:members: expand_dims, copy_model, make_df

.. automodule:: message_ix.util.sankey
:members: map_for_sankey

Testing utilities
-----------------

.. automodule:: message_ix.testing
:members: make_austria, make_dantzig, make_westeros, tmp_model_dir

.. automodule:: message_ix.testing.nightly
:members:
1 change: 1 addition & 0 deletions doc/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@
# top-level module in objects.inv. Resolve these using :doc:`index` or similar for
# each project.
"dask$": ":std:doc:`dask:index`",
"jpype$": ":std:doc:`jpype:index`",
"plotnine$": ":class:`plotnine.ggplot`",
}

Expand Down
2 changes: 1 addition & 1 deletion doc/install-adv.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ This implies five of the six available groups of extra requirements:

- ``docs`` includes packages required to build this documentation locally,
including ``message_ix[report]`` and all *its* requirements,
- ``ixmp4`` includes packages require to use :class:`ixmp.IXMP4Backend <.IXMP4Backend>`,
- ``ixmp4`` includes packages require to use :class:`ixmp.IXMP4Backend <ixmp.backend.ixmp4.IXMP4Backend>`,
- ``report`` includes packages required to use the built-in :doc:`reporting <reporting>` features of :mod:`message_ix`,
- ``sankey`` includes packages required to use :meth:`.Reporter.add_sankey`,
- ``tests`` includes packages required to run the test suite,
Expand Down
10 changes: 6 additions & 4 deletions doc/macro.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,12 @@ The required dimensions or symbol of each item are given in the same notation us

- ``price_ref`` (:math:`n, s`): prices of MACRO sector output in the reference year.
These can be constructed from the MESSAGE variable ``PRICE_COMMODITY``, using the ``config`` mapping.
If not provided, :mod:`message_ix.macro` will identify the reference year and extrapolate reference values using an exponential function fitted to ``PRICE_COMMODITY`` values; see :func:`.macro.extrapolate`.
If not provided, :mod:`message_ix.macro.calibrate` will identify the reference year
and extrapolate reference values using an exponential function fitted to ``PRICE_COMMODITY`` values.
See :func:`~.macro.calibrate.extrapolate`.
- ``cost_ref`` (:math:`n`): total cost of the energy system in the reference year.
These can be constructed from the MESSAGE variable ``COST_NODAL_NET``, including dividing by a factor of 1000.
If not provided, :mod:`message_ix.macro` will extrapolate using :func:`.macro.extrapolate`.
If not provided, :mod:`message_ix.macro.calibrate` will extrapolate using :func:`~.macro.calibrate.extrapolate`.
- ``demand_ref`` (:math:`n, s`): demand for MACRO sector output in the reference year.
- ``lotol`` (:math:`n`): tolerance factor for lower bounds on MACRO variables.
- ``esub`` (:math:`\epsilon_n`): elasticity of substitution between capital-labor and energy.
Expand Down Expand Up @@ -228,9 +230,9 @@ Alternatively the arguments can be specified either in :file:`models.py`.
Code documentation
==================

.. currentmodule:: message_ix.macro
.. currentmodule:: message_ix.macro.calibrate

.. automodule:: message_ix.macro
.. automodule:: message_ix.macro.calibrate
:members:

The functions :func:`add_model_data` and :func:`calibrate` are used by :meth:`.Scenario.add_macro`.
Expand Down
2 changes: 1 addition & 1 deletion doc/reporting.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ These include:

Other added keys include:

- :mod:`message_ix` adds the standard short symbols for |MESSAGEix| dimensions (sets) based on :data:`.models.DIMS`.
- :mod:`message_ix` adds the standard short symbols for |MESSAGEix| dimensions (sets) based on :data:`.common.DIMS`.
Each of these is also available in a Reporter: for example :py:`rep.get("n")` returns a list with the elements of the |MESSAGEix| set named "node";
:py:`rep.get("t")` returns the elements of the set "technology", and so on.
These keys can be used as input to other computations.
Expand Down
4 changes: 3 additions & 1 deletion message_ix/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
from ixmp.util import DeprecatedPathFinder

from .core import Scenario
from .models import MACRO, MESSAGE, MESSAGE_MACRO
from .macro import MACRO
from .message import MESSAGE
from .message_macro import MESSAGE_MACRO
from .report import Reporter
from .util import make_df

Expand Down
Loading
Loading