Skip to content

testing.md

Latest commit

 

History

History
134 lines (101 loc) · 5.69 KB

testing.md

File metadata and controls

134 lines (101 loc) · 5.69 KB

Testing Guide

This document is the test catalog for the notebooks repository. It describes what tests exist, where they live, how to run them, and how they map to CI.

For operational gotchas (worktree naming, pyfakefs, KONFLUX matching, CI -n metadata), see CONTRIBUTING.md.

Setup

uv venv --python "$(which python3.14)"
uv sync --locked

On macOS, install Homebrew GNU Make so make resolves to GNU Make 4.x:

brew install make
PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"

Test targets

Target What it runs Requires
make test Static tests (pytest) + Dockerfile alignment check Nothing
make test-unit Python unit tests + doctests + Go tests Nothing (Go auto-downloads)
make test-integration PYTEST_ARGS="--image=<img>" Container integration tests Podman/Docker
make test-<notebook> Notebook smoke test via papermill kubectl + deployed workbench

Deploying for notebook smoke tests

make test-<notebook> requires a deployed workbench on OpenShift. For the full deploy/test/undeploy cycle, see README.md § Notebooks and README.md § Runtimes.

Test types and locations

Type Location Naming Organization
Static / manifest tests/test_*.py Module-level functions Group with subtests
Unit tests/unit/ test_*.py mirroring source layout Module-level functions
Container integration tests/containers/ *_test.py (class-based) Test* classes with fixtures
Browser E2E tests/browser/tests/ *.spec.ts Playwright + pnpm
Go scripts/buildinputs/ Standard Go test files make test-unit runs these
K8s notebook smoke scripts/ Shell scripts make test-<notebook>

Collection behavior

  • tests/containers/ is excluded from default collection via collect_ignore in tests/conftest.py. Run container tests explicitly with pytest tests/containers --image=<img>.
  • Default make test collects from tests/, ntb/, and ci/ (doctests).
  • --strict-markers is on — unregistered markers fail the run.

Markers

All markers must be registered in pytest.ini:

Marker Meaning Excluded from default run
openshift Needs live OpenShift cluster Yes
cuda Needs NVIDIA GPU Yes
rocm Needs AMD GPU Yes
manifest_validation Slow registry/skopeo checks Yes
buildonlytest Runs inside docker build only Yes (different filter)
codeserver Code-Server workbench specific No

CI parity

Local make test + make test-unit covers the pytest-tests job in .github/workflows/code-quality.yaml. Other CI checks that are not yet exposed as make targets:

  • yamllint (inline in workflow)
  • hadolint (inline in workflow)
  • gotestsum (inline in workflow)
  • prek (inline in workflow)

Closing this gap (moving inline CI logic into Makefile targets) is tracked in #3174.

Frameworks and tools

Tool Purpose
pytest Test runner for all Python tests
pytest-subtests Granular sub-assertions within a single test
pytest-cov Coverage (XML + terminal)
allure-pytest Issue tracking + step decoration
testcontainers Container lifecycle for integration tests
pyfakefs Filesystem mocking for unit tests
Playwright Browser tests (TypeScript)
papermill Notebook execution verification

Troubleshooting

  • Container tests hang: Ensure the container runtime (podman/docker) is running. On Linux: systemctl --user start podman.service.
  • Dependency conflicts after lock regen: Run make refresh-lock-files and check for "unsatisfiable" errors. See docs/cves/python.md for constraint resolution.
  • make test fails on stray files: Extra top-level directories (.cursor-tmp-*) break repo-wide assertions. Clean clone or git clean -fdx the offending paths.
  • Security scanning: Weekly Quay vulnerability reports are generated by ci/security-scan/quay_security_analysis.py (triggered via .github/workflows/sec-scan.yml).

External test suites

The images built by this repo are also tested by other projects:

Suite Framework What it tests
odh-dashboard Cypress Workbench CRUD, image selection, RBAC via ODH dashboard
ods-ci Robot Framework GPU/CUDA validation, Elyra pipelines, plugin consistency
opendatahub-tests Pytest ImageStream health, Notebook CR spawning, package availability

Running images locally

Build with a predictable tag and push disabled, then run:

# JupyterLab (port 8888)
make jupyter-minimal-ubi9-python-3.12 IMAGE_TAG=latest PUSH_IMAGES=no
podman run -it -p 8888:8888 quay.io/opendatahub/workbench-images:jupyter-minimal-ubi9-python-3.12-latest

# Code-Server (port 8787)
make codeserver-ubi9-python-3.12 IMAGE_TAG=latest PUSH_IMAGES=no
podman run -it -p 8787:8787 quay.io/opendatahub/workbench-images:codeserver-ubi9-python-3.12-latest

For published image references, see the README.md Image Inventory. Override the entrypoint to inspect an image without starting the workbench (see AGENTS.md operational notes).