Simplifying regression testing with docker

[lbutler]

I believe we can significantly simplify the process of running regression tests in this repository.

Currently, there are multiple scripts dedicated to this task, with separate versions depending on whether the system is Windows or Unix-based. Maintaining duplicate scripts increases complexity.

Personally, I've also found it very difficult to run the scripts and only succeeded after much manual effort.

By bundling the regression testing workflow into a Docker container, we could:

1. Reduce maintenance overhead by consolidating all scripts into a single environment.
2. Streamline the testing process, making it easier to run the same tests locally and in CI/CD pipelines.
3. Allow users to test private models, making it easier to detect regressions in their own networks.
4. Enable commit-specific testing, allowing users to identify exactly when a regression was introduced.

The same Docker container could build, test, and run regression tests in one place, simplifying the CI/CD workflow for both contributors and maintainers.

I've created a proof of concept (PoC) that demonstrates this approach:
https://github.com/lbutler/epanet-regression-testing

Action	Command
Run standard regression tests (latest master vs. reference v2.2)	docker run --rm epanet-regtest
Test a specific branch (e.g., feature-branch)	docker run --rm -e SUT_TAG="feature-branch" epanet-regtest
Test a specific commit (e.g., d4e5f8c)	docker run --rm -e SUT_TAG="d4e5f8c" epanet-regtest
Run both standard and custom tests	docker run --rm -e DO_CUSTOM_TEST=true -v /path/to/private/tests:/custom_tests epanet-regtest

Before investing more time into refining and testing it, I’d like to hear the general consensus on this approach.

(previous discussions related to regression testing: #635 & #736 )

[LRossman]

@lbutler I agree that the current unit and regression testing protocol is too complex and requires too many dependencies. I suggest that before you pursue using your Docker approach to simplify it, we totally replace the current protocol with a more simpler and maintainable one.

I suggest that we replace the Boost test library for the unit tests with the Catch2 system (no dependencies needed). As for the regression tests, we could replace the Python-based testing code with a simple C program that compares binary benchmark files to the binary output files produced by the EPANET DLL being tested using code similar to what is already used in the Toolkit's report.c unit.

We could pursue these changes in a separate fork of this project so that we don't delay the release of v.2.3. If they prove to be successful they could then be merged into the dev branch for the next version of the toolkit.

[lbutler]

Thanks for your feedback @LRossman.

Perhaps as an intermediate step, before completely replacing the existing regression tests, we could perform the minimal work needed to ensure they are functional in their current state. Alongside this, we could introduce Docker as an optional method for running the tests, simplifying dependency management for those who opt to use it.

This approach would allow us to integrate working regression tests into the CI/CD process sooner, providing immediate confidence. Then, as suggested, we could pursue the major overhaul in a separate fork.