Only run cleanup if tests ran, adds pytest marker config for slow tests (#595)

### Description

We don't need to fail the cleanup step if we never ran the tests, this
deletes the image from the GPU worker to save space. Also adds the
missing pytest marker configuration to our pytest.ini for skipping slow
tests.

### Type of changes
<!-- Mark the relevant option with an [x] -->

- [x]  Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (non-breaking change which adds functionality)
- [ ]  Refactor
- [ ]  Documentation update
- [ ]  Other (please describe):

### CI Pipeline Configuration
Configure CI behavior by checking relevant boxes below. This will
automatically apply labels.

- [ ]
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
- [ ]
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest

> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.

### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```

### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->

 - [ ] I have tested these changes locally
 - [ ] I have updated the documentation accordingly
 - [ ] I have added/updated tests as needed
 - [ ] All existing tests pass successfully

---------

Signed-off-by: Peter St. John <pstjohn@nvidia.com>

Author: pstjohn

.github/pull_request_template.md

@@ -11,10 +11,10 @@
 - [ ]  Other (please describe):
 
 ### CI Pipeline Configuration
-Configure CI behavior by checking relevant boxes below. This will automatically apply labels.
+Configure CI behavior by applying the relevant labels:
 
-- [ ] [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests
-- [ ] [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest
+- [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests
+- [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest
 
 > [!NOTE]
 > By default, the notebooks validation tests are skipped unless explicitly enabled.

.github/workflows/pr-labels.yml

@@ -1,4 +1,4 @@
-name: "[Optional] BioNemo Image Build and Unit Tests"
+name: "BioNemo Image Build and Unit Tests"
 
 on:
   pull_request:
@@ -100,6 +100,12 @@ jobs:
           BIONEMO_DATA_SOURCE: ngc
         run: ./ci/scripts/run_pytest.sh --no-nbval --skip-slow
 
+      - name: Run notebook tests
+        if: ${{ contains(github.event.pull_request.labels.*.name, 'INCLUDE_NOTEBOOKS_TESTS') }}
+        env:
+          BIONEMO_DATA_SOURCE: ngc
+        run: pytest --nbval-lax -p no:python docs/ sub-packages/
+
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v5
         with:
@@ -116,10 +122,9 @@ jobs:
   clean-up:
     needs: run-tests
     runs-on: self-hosted-nemo-gpus-1
-    if: ${{ always() }}
+    if: ${{ success() || failure() }}
     steps:
       - name: clean up image
         run: docker rmi nemoci.azurecr.io/bionemo:${{ github.run_id }}
-
 # TODO: exclude tests from base image; run tests from github workspace mounted in the image.
 # TODO: figure out way of cleaning up working directory (requires sudo or for us to fix file ownership from release container)

.github/workflows/unit-tests.yml

@@ -139,9 +139,9 @@
         "filename": "pyproject.toml",
         "hashed_secret": "79670e9c9d1c7ea5b81a96a2053d81437712c78e",
         "is_verified": false,
-        "line_number": 47
+        "line_number": 43
       }
     ]
   },
-  "generated_at": "2024-11-01T22:26:03Z"
+  "generated_at": "2025-01-14T17:26:27Z"
 }

.secrets.baseline

@@ -36,11 +36,7 @@ dependencies = [
 build = ['flash-attn', 'pip']
 
 [tool.uv.workspace]
-members = [
-    "3rdparty/*",
-    "sub-packages/bionemo-*/",
-    "internal/infra-bionemo/",
-]
+members = ["3rdparty/*", "internal/infra-bionemo/", "sub-packages/bionemo-*/"]
 
 [tool.uv.sources]
 # external
@@ -114,9 +110,10 @@ convention = "google"
 [tool.pytest.ini_options]
 norecursedirs = ["3rdparty"]
 addopts = ["--ignore=3rdparty"]
+markers = ["slow: marks tests as slow (deselect with '-m \"not slow\"')"]
 
 [tool.pyright]
-include = ["./scripts/", "./sub-packages/", "./internal/"]
+include = ["./internal/", "./scripts/", "./sub-packages/"]
 exclude = ["*/tests/"]
 executionEnvironments = [
     { "root" = ".", pythonVersion = "3.10", extraPaths = [