Checksum Reset via Env Variable (#5105)

* Checksum Reset via Env Variable

In preparation for CTest adoption, we want to make it easier
to quickly run and reset checksums locally. This adds the
feature to reset instead of evaluate in regular test runs.

* Improve wording

Co-authored-by: Edoardo Zoni <[email protected]>

Author: ax3l

Docs/source/developers/checksum.rst

@@ -65,6 +65,21 @@ Since this will automatically change the JSON file stored on the repo, make a se
    git add <test name>.json
    git commit -m "reset benchmark for <test name> because ..." --author="Tools <[email protected]>"
 
+Automated reset of a list of test benchmarks
+--------------------------------------------
+
+If you set the environment variable ``export CHECKSUM_RESET=ON`` before running tests that are compared against existing benchmarks, the test analysis will reset the benchmarks to the new values, skipping the comparison.
+
+With `CTest <https://cmake.org/cmake/help/latest/manual/ctest.1.html>`__ (coming soon), select the test(s) to reset by `name <https://cmake.org/cmake/help/latest/manual/ctest.1.html#run-tests>`__ or `label <https://cmake.org/cmake/help/latest/manual/ctest.1.html#label-matching>`__.
+
+.. code-block:: bash
+
+   # regex filter: matched names
+   CHECKSUM_RESET=ON ctest --test-dir build -R "Langmuir_multi|LaserAcceleration"
+
+   # ... check and commit changes ...
+
+
 Reset a benchmark from the Azure pipeline output on Github
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

Regression/Checksum/checksumAPI.py

@@ -42,6 +42,9 @@ def evaluate_checksum(test_name, output_file, output_format='plotfile', rtol=1.e
     Read checksum from output file, read benchmark
     corresponding to test_name, and assert their equality.
 
+    If the environment variable CHECKSUM_RESET is set while this function is run,
+    the evaluation will be replaced with a call to reset_benchmark (see below).
+
     Parameters
     ----------
     test_name: string
@@ -65,9 +68,17 @@ def evaluate_checksum(test_name, output_file, output_format='plotfile', rtol=1.e
     do_particles: bool, default=True
         Whether to compare particles in the checksum.
     """
-    test_checksum = Checksum(test_name, output_file, output_format,
-                             do_fields=do_fields, do_particles=do_particles)
-    test_checksum.evaluate(rtol=rtol, atol=atol)
+    # Reset benchmark?
+    reset = ( os.getenv('CHECKSUM_RESET', 'False').lower() in
+              ['true', '1', 't', 'y', 'yes', 'on'] )
+
+    if reset:
+        print(f"Environment variable CHECKSUM_RESET is set, resetting benchmark for {test_name}")
+        reset_benchmark(test_name, output_file, output_format, do_fields, do_particles)
+    else:
+        test_checksum = Checksum(test_name, output_file, output_format,
+                                 do_fields=do_fields, do_particles=do_particles)
+        test_checksum.evaluate(rtol=rtol, atol=atol)
 
 
 def reset_benchmark(test_name, output_file, output_format='plotfile', do_fields=True, do_particles=True):