Ensure that pytest is the only strict requirement for running test suite for all installation modes (#738)

* Skip addheader tests if package is not available

* Assume that pytest is the only additional testing requirement

* Parametrize installation target

* Prioritize WIP jobs while debugging

* Move omlt to optional requirements

* Reorder keras_surrogate imports

* Use pytest.importorskip() to check for optional modules sooner

* Display skipped tests and CI and docs use exactly the same commands

* Add more context to post-installation pytest instructions

* Revert "Prioritize WIP jobs while debugging"

This reverts commit e1f1892.

* Move omlt back to default dependencies in setup.py

* Extend CI check to simulate environment after Conda installation

* Prioritize WIP jobs while debugging

* Track renamed matrix parameter

* Reassure pip that we really meant to uninstall

* Try to see if -rs flag to pytest causes failure in test_enrtl

TestParameters.test_parameters_alpha_symmetry_duplicate

* Try to see if the culprit is -W ignore instead

* Fix outdated module used for caplog that was causing test to fail

* Use module-level pytest.importorskip()

* Add remaining individual pytest.skipif()

* Fix typo in conditionally imported module name

* Revert "Prioritize WIP jobs while debugging"

This reverts commit 084a8c2.

(cherry picked from commit 90f762a)

Author: lbianchi-lbl

Diff for: .github/workflows/integration.yml

@@ -193,7 +193,7 @@ jobs:
           cat *errors.txt || echo "No error logs found"
 
   pytest-site-packages:
-    name: Run tests from site-packages (non-editable) installation
+    name: Run tests from site-packages (non-editable) installation (${{ matrix.install-mode }})
     # NOTE: using ubuntu-latest (20.04) instead of 18.04 results in failures in power_generation/carbon_capture/mea_solvent_system/unit_models/tests/test_column.py
     runs-on: ubuntu-18.04
     needs: [precheck]
@@ -203,6 +203,21 @@ jobs:
         || needs.precheck.outputs.workflow-trigger == 'not_pr'
         || needs.precheck.outputs.workflow-trigger == 'approved_pr'
       )
+    strategy:
+      fail-fast: false
+      matrix:
+        install-mode:
+          - pip-default
+          - pip-optional
+          - conda-like
+        include:
+          - install-mode: pip-default
+            pip-install-target: '.'
+          - install-mode: pip-optional
+            pip-install-target: .[optional]
+          - install-mode: conda-like
+            pip-install-target: '.'
+            dependencies-to-uninstall: omlt
     steps:
       - uses: actions/checkout@v2
       - name: Set up Conda environment
@@ -213,13 +228,18 @@ jobs:
       - name: Set up idaes (non-editable installation)
         uses: ./.github/actions/setup-idaes
         with:
-          install-target: '.[optional]'
-      - name: Install test dependencies and run pytest in empty directory
+          install-target: ${{ matrix.pip-install-target }}
+      - name: Remove dependencies installable with pip but not with conda
+        # NOTE some dependencies that are installed by default with pip are not available through conda
+        # so they're not installed if IDAES is installed with `conda -c conda-forge -c IDAES-PSE idaes-pse`
+        # to ensure this scenario is handled properly, since we don't have (yet) the conda-build process integrated with the CI,
+        # we manually remove the "pip-but-not-conda" dependencies after installing with pip
+        # as an approximation of the enviroment that we'd get from `conda install`
+        if: matrix.dependencies-to-uninstall
+        run: |
+          pip uninstall --yes ${{ matrix.dependencies-to-uninstall }}
+      - name: Install and run pytest in empty directory
         run: |
           pushd "$(mktemp -d)"
-          # TODO this could be an optional [testing] dependency
-          pip install pytest addheader
-          # TODO downloading pytest.ini is still needed for the tests to pass
-          # eventually it will be moved to the root conftest.py so that is installed as Python code
-          wget https://raw.githubusercontent.com/IDAES/idaes-pse/main/pytest.ini 
-          pytest --pyargs idaes
+          pip install pytest
+          pytest --pyargs idaes -W ignore -rs

Diff for: docs/tutorials/getting_started/index.rst

@@ -187,15 +187,21 @@ Powershell Prompt.  Regardless of OS and shell, the following steps are the same
     Refer to the full :doc:`idaes get-examples command documentation <../../../reference_guides/commands/get_examples>`
     for more information.
 
-4. Run tests::
+4. To test that the installation was successful, you can run the IDAES test suite.
+    This is not strictly required, but can offer more confidence that the installation environment is as expected (if the tests pass),
+    and more insight into possible issues (if any of the tests fail).
+    The ``pytest`` package is required for running the test suite. After installing it using e.g. ``pip``, run the tests::
 
-    pytest --pyargs idaes -W ignore
+    pip install pytest
+    pytest --pyargs idaes -W ignore -rs
 
-5. You should see the tests run and all should pass to ensure the installation worked. You
-   may see some "Error" level log messages, but they are okay, and produced by tests for
+5. You should see the tests run and all should pass to ensure the installation worked.
+   You may see some "Error" level log messages, but they are okay, and produced by tests for
    error handling. The number of tests that failed and succeeded is reported at the end of the pytest
-   output. You can report problems on the |github-issues|
-   (Please try to be specific about the command and the offending output.)
+   output.
+   If the optional ``-rs`` flag is given, the output will also display tests that were skipped because of
+   e.g. optional dependencies that can be installed separately (see below).
+   You can report problems on the |github-issues| (Please try to be specific about the command and the offending output.)
 
 **Install IDAES using Conda**
 

Diff for: idaes/generic_models/properties/core/coolprop/tests/test_coolprop_wrapper.py

@@ -41,16 +41,16 @@
     Constant
 
 
+CoolProp = pytest.importorskip("CoolProp.CoolProp", reason="CoolProp not installed")
+
 from idaes.generic_models.properties.core.coolprop.coolprop_wrapper import (
     CoolPropWrapper,
     CoolPropExpressionError,
     CoolPropPropertyError)
 
-CoolProp, coolprop_available = attempt_import('CoolProp.CoolProp')
 solver = get_solver()
 
 
-@pytest.mark.skipif(not coolprop_available, reason="CoolProp not installed")
 class TestWrapper:
     @pytest.mark.unit
     def test_load_component(self):
@@ -233,7 +233,6 @@ def test_pressure_sat_unrecognised_form(self):
             CoolPropWrapper.pressure_sat_comp.build_parameters(m.TestComp)
 
 
-@pytest.mark.skipif(not coolprop_available, reason="CoolProp not installed")
 class TestCoolPropProperties(object):
     @pytest.fixture(scope="class")
     def m(self):
@@ -671,7 +670,6 @@ def test_psat(self, m):
                         m.fs.state[0].pressure_sat_comp["benzene"])
 
 
-@pytest.mark.skipif(not coolprop_available, reason="CoolProp not installed")
 class TestVerifyExcessLiq(object):
     @pytest.fixture(scope="class")
     def m(self):
@@ -802,7 +800,6 @@ def test_cubic_liquid_entr(self, m):
                         m.fs.state[0].entr_mol_phase["Liq"] - S0_I)
 
 
-@pytest.mark.skipif(not coolprop_available, reason="CoolProp not installed")
 class TestVerifyExcessVap(object):
     @pytest.fixture(scope="class")
     def m(self):

Diff for: idaes/generic_models/properties/core/eos/tests/test_enrtl.py

@@ -190,7 +190,7 @@ def test_parameters_alpha_symmetry_duplicate(self, caplog):
         caplog.set_level(
             idaeslog.INFO,
             logger=("idaes.generic_models.properties.core."
-                    "generic.generic_property"))
+                    "eos.enrtl_parameters"))
 
         test_config = dict(configuration)
         test_config["parameter_data"] = {}

Diff for: idaes/surrogate/keras_surrogate.py

@@ -13,20 +13,30 @@
 """
 Interface for importing Keras models into IDAES
 """
-import numpy as np
+from enum import Enum
 import json
 import os.path
 
+import numpy as np
+import pandas as pd
+
 from pyomo.common.dependencies import attempt_import
 keras, keras_available = attempt_import('tensorflow.keras')
+omlt, omlt_available = attempt_import("omlt")
+
+if omlt_available:
+    from omlt import OmltBlock, OffsetScaling
+    from omlt.neuralnet import (
+        FullSpaceContinuousFormulation,
+        ReducedSpaceContinuousFormulation,
+        ReLUBigMFormulation,
+        ReLUComplementarityFormulation,
+        load_keras_sequential
+    )
 
-from enum import Enum
-import pandas as pd
 from idaes.surrogate.base.surrogate_base import SurrogateBase
 from idaes.surrogate.sampling.scaling import OffsetScaler
-from omlt import OmltBlock, OffsetScaling
-from omlt.neuralnet import (FullSpaceContinuousFormulation, ReducedSpaceContinuousFormulation,
-                            ReLUBigMFormulation, ReLUComplementarityFormulation, load_keras_sequential)
+
 
 class KerasSurrogate(SurrogateBase):
     def __init__(self, keras_model, input_labels, output_labels, input_bounds,

Diff for: idaes/surrogate/tests/test_keras_surrogate.py

@@ -15,19 +15,20 @@
 """
 import pytest
 
+pytest.importorskip("tensorflow.keras", reason="tensorflow.keras not available")
+pytest.importorskip("omlt", reason="omlt not available")
+
 import os.path
 import pandas as pd
 from pyomo.common.fileutils import this_file_dir
 from pyomo.common.tempfiles import TempfileManager
 from pyomo.environ import (ConcreteModel, Var, SolverFactory,
                            assert_optimal_termination, value,
                            Objective)
-from idaes.surrogate.keras_surrogate import KerasSurrogate, load_keras_json_hd5, keras_available
+from idaes.surrogate.keras_surrogate import KerasSurrogate, load_keras_json_hd5
 from idaes.surrogate.surrogate_block import SurrogateBlock
 from idaes.surrogate.sampling.scaling import OffsetScaler
 
-if not keras_available:
-    pytestmark = pytest.mark.skip("tensorflow.keras not available")
 
 rtol = 1e-4
 atol = 1e-4

Diff for: idaes/tests/test_headers.py

@@ -17,11 +17,13 @@
 from pathlib import Path
 import os
 # third-party
-from addheader.add import FileFinder, detect_files
 import pytest
 import yaml
 
 
+addheader_add = pytest.importorskip("addheader.add", reason="`addheader` package is not available")
+
+
 @pytest.fixture
 def package_root():
     """Determine package root.
@@ -50,8 +52,8 @@ def test_headers(package_root, patterns):
         print(f"ERROR: Did not get glob patterns: skipping test")
     else:
         # modify patterns to match the files that should have headers
-        ff = FileFinder(package_root, glob_patterns=patterns)
-        has_header, missing_header = detect_files(ff)
+        ff = addheader_add.FileFinder(package_root, glob_patterns=patterns)
+        has_header, missing_header = addheader_add.detect_files(ff)
         # ignore empty files (probably should add option in 'detect_files' for this)
         nonempty_missing_header = list(filter(lambda p: p.stat().st_size > 0, missing_header))
         #
@@ -62,4 +64,3 @@ def test_headers(package_root, patterns):
             print(f"Missing headers from files under '{pfx}{os.path.sep}': {file_list}")
         # uncomment to require all files to have headers
         assert len(nonempty_missing_header) == 0
-        

Diff for: setup.py

@@ -109,7 +109,7 @@ def dmf_data_files(root: str = DMF_DATA_ROOT) -> List[Tuple[str, List[str]]]:
             "tensorflow",  # idaes.surrogate.keras_surrogate
             # A Lee 11-Jan-22: no precompiled version of CoolProp available for Pyhton 3.9
             "coolprop; python_version < '3.9'"  # idaes.generic_models.properties.general.coolprop
-        ]
+        ],
     },
     package_data={
         # If any package contains these files, include them: