Have tests use a temporary default config file and cache directory

[sol1105]

Tests may fail or interact with the user configuration file when users or developers run them locally. Several tests already make use of a fixture to make use of a temporary cache directory and config file, but this has to be set up for all tests.

[sol1105]

#135 does some work on that.

[mpmoine]

from TISG meeting 38:

• MS: Didn’t get to this yet. We want to make the tests independent of where they are running. Should not interact with personal config or cache. This is not the case currently. Might be a connection to the pull request issues discussed earlier Update tests #131

[sol1105]

@JamesAnstey I will have another close look at this as soon as I have the time. I plan to set up fixtures for test classes and test functions, that everyone can then just reuse for their tests. The main point is that when importing any module of the dreq software, the config is already loaded and now one has to use some tricks to overwrite / reload it. In the following I just list what I got so far (which is also the product of some trial and error), also for my reference:

• Fixtures for the cli test classes to set up a default config file in test_cli.py.
  

  CMIP7_DReq_Software/data_request_api/data_request_api/tests/test_cli.py

  Lines 12 to 27 in 630296c

  	@pytest.fixture(scope="class")
  	def monkeyclass():
  	with pytest.MonkeyPatch.context() as mp:
  	yield mp
  	@pytest.fixture(scope="class")
  	def temp_config_file(tmp_path_factory, monkeyclass):
  	temp_dir = tmp_path_factory.mktemp("data")
  	config_file = temp_dir / ".CMIP7_data_request_api_config"
  	monkeyclass.setenv("CMIP7_DR_API_CONFIGFILE", str(config_file))
  	# Provide the test with the config file
  	try:
  	yield config_file
  	finally:
  	config_file.unlink(missing_ok=True)

• For the test class, one can have a setup_method automatically applied to all tests of that class as follows:

class TestExportDreqListsJson:
    @pytest.fixture(scope="function", autouse=True)
    def setup_method(self, request):
        # Initialize config and load v1.2.2 content version
        self.temp_config_file = request.getfixturevalue("temp_config_file")
        with open(self.temp_config_file, "w") as fh:
            config = {
                "consolidate": False,   # custom config values you might want to test that differ from default
                "cache_dir": str(self.temp_config_file.parent), # cache dir needs to be set if the cache should be test only as well
            }
            yaml.dump(config, fh)
        dc.load("v1.2.2")

• If setting up a test function instead of a class, the following fixture should work to have a fresh default config file:

@pytest.fixture(scope="function")
def temp_config_file(tmp_path_factory):
    temp_dir = tmp_path_factory.mktemp("data")
    config_file = temp_dir / ".CMIP7_data_request_api_config"
    try:
        yield config_file
    finally:
        config_file.unlink(missing_ok=True)
        dreqcfg.CONFIG = {}

@pytest.fixture(scope="function")
def monkeypatch(monkeypatch):
    return monkeypatch

• In the test function one can then write to the config file and make sure the correct config file is referenced:

def test_load_config_file_exists(temp_config_file, monkeypatch):
    with open(temp_config_file, "w") as f:
        yaml.dump(DEFAULT_CONFIG, f)
    monkeypatch.setattr(
        "data_request_api.utilities.config.CONFIG_FILE", temp_config_file
    )

• Some remarks:
  In test_cli.py the cli tools are called (self contained start and stop of the software), so it is necessary to set the CMIP7_DR_API_CONFIGFILE environment variable but not necessary to reset dreqcfg.CONFIG = {}.

It probably needs some further trial and error to get this fixtures and logic universally set up for all tests. It seems more complex than it needs to be, but I failed with simpler approaches.