Test launcher based on pytest by svalat · Pull Request #366 · idefix-code/idefix

svalat · 2026-03-19T10:47:19Z

The problem

When running locally the full test suite (tests in /test of Idefix) we might want to avoid getting a long log with no summary of what passed and what failed.

What it does

In order a ease running the suite, I implemented a way to call the tests via the pytest library which is used to implement python unit tests. It offers by default :

Summary logging (just SUCCESS if the test is valid)
Filtering test to select what to run (eg. only MPI => -k mpi)
Get a detailed output only for failed tests.
Can get verbose mode (-v) if desired
Get report in junit format which can be taken by github or we can generate an html page from it (via junit2html).

testme.json

I provided a way to describe the test running option combinations via a testme.json file which might be more readable
and less programmatic than the current testme.py.

Example:

{
     "variants": [
        {
            "dumpname": "dump.0001.dmp",
            "ini": ["idefix.ini","idefix-rkl.ini"],
            "noplot": true,
            "single": [false, true],
            "reconstruction": 2,
            "mpi": [false, true],
            "tolerance": 0
        }
    ]
}

This PR provides the testme.json equivalent to each testme.py in the test directory.

Documentation

I updated the documentation by adding a new dedicated page in doc/source/testing/testLauncher.rst

Missing

Make a default configuration for re-generate the reference results.

Legacy

I currently let untouched the current testme.py files to let the time to validate and complete the new process.

glesur

a few typos, but otherwise ok

glesur · 2026-03-29T18:58:26Z

+Overview
+--------
+
+The class :doc:`idfxTest <idfxTest>` provides the basement to implement an *Idefix* integration test for validation.


Suggested change

The class :doc:`idfxTest <idfxTest>` provides the basement to implement an *Idefix* integration test for validation.

The class :doc:`idfxTest <idfxTest>` provides the toolbox to implement an *Idefix* integration test for validation.

glesur · 2026-03-29T18:59:12Z

+Depencencies
+------------
+
+Before using :doc:`idfxTest <idfxTest>` you need to install some Python Depencencies (possibly in a ``virtual env``):


Suggested change

Before using :doc:`idfxTest <idfxTest>` you need to install some Python Depencencies (possibly in a ``virtual env``):

Before using :doc:`idfxTest <idfxTest>` you need to install some Python depencencies (possibly in a ``virtual env``):

glesur · 2026-03-29T19:00:17Z

+    ---------- generated xml file: idefix-tests.junit.xml ---------------------------------------------------------
+    =============================== 8 passed, 44 deselected in 73.03s (0:01:13) ===================================
+
+When seeing an error, the output of the command will be printed at the end.


Suggested change

When seeing an error, the output of the command will be printed at the end.

When an error is detected, the output of the command will be printed at the end.

glesur · 2026-03-29T19:01:44Z

+Definition of the tests
+-----------------------
+
+The script is simply searching all the files names ``testme.json`` into the ``test`` directory.


Suggested change

The script is simply searching all the files names ``testme.json`` into the ``test`` directory.

The script looks for all the files named ``testme.json`` in the ``test`` directory and all its sub-directories.

glesur · 2026-03-29T19:02:29Z

+-----------------------
+
+The script is simply searching all the files names ``testme.json`` into the ``test`` directory.
+This file describe the combination of parameters to use to produce the list of *Idefix* run to perform.


Suggested change

This file describe the combination of parameters to use to produce the list of *Idefix* run to perform.

This file describes the combination of parameters used to produce the list of *Idefix* runs to check.

glesur · 2026-03-29T19:03:17Z

+    }
+
+
+By default, the alphabetical order will be used.


Suggested change

By default, the alphabetical order will be used.

By default, the alphabetical order is used.

… some paths handling in idfx_test.py

…est.py

… pytools by default

…e.json

### Changed - fixed a bug that could lead to diverging results around the spherical axis in non-ideal MHD in 2.5D and 3D (#356) - refactor of the MPI exchange routines and boundary routines to avoid buildup of roundoff errors at domain faces/edges that could lead to the sudden burst of div(B) or incoherences between MPI sub-domains (#357) - fixed a bug that could lead to compilations error when targetting AMD APUs (#359) - fixed a bug that led to the generation of incorrect subviews in 2.5D with vector_potential enabled (#362) - fixed a bug that could lead to memory corruption when using the UCT_HLLD emf reconstruction scheme and DIMENSIONS < COMPONENTS (#363) - reorganise the test to separate specific IO tests from physics tests (#367) ### Added - magnetic vector potential is now accessible from pydefix when enabled (#361) - use ccache in the test suite to reduce the runtime of continuous integration (#364) - automatically detects pybind11 path with cmake when using pydefix (#367) - skeleton to run the test suite with pytest (#366) --------------------------------------------------------------------------- Co-authored-by: Clément Robert <cr52@protonmail.com> Co-authored-by: Sébastien Valat <sebastien.valat.dev@orange.fr>

svalat added documentation Improvements or additions to documentation tests labels Mar 19, 2026

svalat changed the title ~~Feature/test with pytest~~ Test launcher based on pytest Mar 19, 2026

svalat force-pushed the feature/test-with-pytest branch from 7eaebeb to f39d6a6 Compare March 19, 2026 12:23

glesur self-requested a review March 26, 2026 22:54

glesur requested changes Mar 29, 2026

View reviewed changes

svalat added 12 commits April 21, 2026 12:32

test: prepare the work to introduce pytest by adding logging + fixing…

a31712a

… some paths handling in idfx_test.py

test: introduce the new runner based on pytest (./test.py)

1c1fd5c

test: port all the idefix tests to the testme.json new semantic for t…

9573483

…est.py

test: document the semantic of test.py

455ef19

test: document -ccache option of idfx_test.py

5069ab3

test: fix some comments

55714e2

test: remove the CMakeCache.txt check, it is not required

062dbf0

test: some fixes in documentation

6952bd1

test: fix test paddings to pass linter in CI

1afed8f

test: change configuration of the root pytest.ini to run the tests in…

c732ca2

… pytools by default

test: fix PR remarks about the doc

ae0fc3f

test: adapt the new IO/ tests to the new semantic and add their testm…

a0aedba

…e.json

svalat force-pushed the feature/test-with-pytest branch from 3f16d43 to a0aedba Compare April 21, 2026 14:17

glesur self-requested a review April 21, 2026 16:14

glesur merged commit a6eebd1 into idefix-code:develop Apr 21, 2026
41 checks passed

glesur mentioned this pull request Apr 22, 2026

v2.3.0 #376

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test launcher based on pytest#366

Test launcher based on pytest#366
glesur merged 12 commits into
idefix-code:developfrom
svalat:feature/test-with-pytest

svalat commented Mar 19, 2026

Uh oh!

glesur left a comment

Uh oh!

glesur Mar 29, 2026

Uh oh!

glesur Mar 29, 2026

Uh oh!

glesur Mar 29, 2026

Uh oh!

glesur Mar 29, 2026

Uh oh!

glesur Mar 29, 2026

Uh oh!

glesur Mar 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	The class :doc:`idfxTest <idfxTest>` provides the basement to implement an Idefix integration test for validation.
	The class :doc:`idfxTest <idfxTest>` provides the toolbox to implement an Idefix integration test for validation.

	Before using :doc:`idfxTest <idfxTest>` you need to install some Python Depencencies (possibly in a ``virtual env``):
	Before using :doc:`idfxTest <idfxTest>` you need to install some Python depencencies (possibly in a ``virtual env``):

	When seeing an error, the output of the command will be printed at the end.
	When an error is detected, the output of the command will be printed at the end.

	The script is simply searching all the files names ``testme.json`` into the ``test`` directory.
	The script looks for all the files named ``testme.json`` in the ``test`` directory and all its sub-directories.

	This file describe the combination of parameters to use to produce the list of Idefix run to perform.
	This file describes the combination of parameters used to produce the list of Idefix runs to check.

	By default, the alphabetical order will be used.
	By default, the alphabetical order is used.

Conversation

svalat commented Mar 19, 2026

The problem

What it does

testme.json

Documentation

Missing

Legacy

Uh oh!

glesur left a comment

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

glesur Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants