@@ -5,6 +5,38 @@ The tests are high-level integration tests at the Kubernetes API level.
55
66You are an expert QE engineer writing maintainable pytest tests that other engineers can understand without deep domain knowledge.
77
8+ ## Setup
9+
10+ ### Prerequisites
11+
12+ - Python 3.14+
13+ - [ uv] ( https://docs.astral.sh/uv/ ) package manager
14+ - Access to an OpenShift cluster with OpenDataHub or OpenShift AI installed
15+
16+ ### Install and Configure
17+
18+ ``` bash
19+ # 1. Log in to the target OpenShift cluster
20+ oc login --server=< cluster-api-url> --token=< token>
21+
22+ # 2. Install dependencies
23+ uv sync
24+ ```
25+
26+ All test commands should be run with ` uv run ` to use the project's virtual environment.
27+
28+ ### Run Prerequisites
29+
30+ Tests run against a live OpenShift cluster. Before running tests:
31+
32+ - ** Cluster access** : ` oc login ` must be completed (or ` KUBECONFIG ` env var set)
33+ - ** Jira connectivity** (for xfail/skip of known bugs): ` PYTEST_JIRA_URL ` , ` PYTEST_JIRA_USERNAME ` , ` PYTEST_JIRA_TOKEN `
34+ - ** S3 credentials** (model serving tests): ` AWS_ACCESS_KEY_ID ` , ` AWS_SECRET_ACCESS_KEY ` , ` CI_S3_BUCKET_NAME ` , ` CI_S3_BUCKET_REGION ` , ` CI_S3_BUCKET_ENDPOINT `
35+ - ** Model storage** (model serving tests): ` MODELS_S3_BUCKET_NAME ` , ` MODELS_S3_BUCKET_REGION ` , ` MODELS_S3_BUCKET_ENDPOINT `
36+ - ** Optional overrides** : ` VLLM_RUNTIME_IMAGE ` , ` OVMS_RUNTIME_IMAGE ` , ` TRITON_RUNTIME_IMAGE ` , ` MLSERVER_RUNTIME_IMAGE `
37+
38+ These can also be passed as pytest command-line options (see ` conftest.py ` for ` addoption ` definitions).
39+
840## Commands
941
1042### Validation (run before committing)
@@ -31,6 +63,69 @@ uv run pytest -m "model_serving and tier1"
3163uv run pytest --setup-plan tests/model_serving/
3264```
3365
66+ ### Debugging
67+
68+ ``` bash
69+ # Drop into pdb on failure
70+ uv run pytest --pdb tests/model_serving/
71+
72+ # Drop into pdb at a specific point (add to code)
73+ import pdb; pdb.set_trace ()
74+
75+ # Verbose output with full tracebacks
76+ uv run pytest -vvv --tb=long tests/model_serving/
77+ ```
78+
79+ ### Make Targets
80+
81+ ``` bash
82+ make check # Install tox and run tox (CI validation)
83+ make build # Build container image (podman/docker)
84+ make push # Push container image to registry
85+ make build-and-push-container # Build and push
86+ ```
87+
88+ To run tests in a container: ` make build ` then run the image via ` podman run ` .
89+
90+ Image defaults: ` quay.io/opendatahub/opendatahub-tests:latest ` (override with ` IMAGE_REGISTRY ` , ` REGISTRY_NAMESPACE ` , ` IMAGE_TAG ` ).
91+
92+ ### Tox Targets
93+
94+ ``` bash
95+ tox -e unused-code # Check for unused code
96+ tox -e pytest # Collect tests and verify setup plan
97+ tox # Run all targets
98+ ```
99+
100+ ## Architecture
101+
102+ ### Component Relationships
103+
104+ Tests are organized by product component. Each component maps to a feature area of OpenDataHub/OpenShift AI:
105+
106+ - ` model_serving ` — Model deployment and inference (KServe, ModelMesh, vLLM, OVMS, Triton)
107+ - ` model_registry ` — Model registration, catalog, RBAC, REST API
108+ - ` model_explainability ` — TrustyAI, LM-Eval, guardrails
109+ - ` llama_stack ` — Llama Stack inference and safety
110+ - ` workbenches ` — Jupyter notebook controllers and images
111+ - ` cluster_health ` — Operator and cluster health checks
112+
113+ ### Fixture Flow
114+
115+ Fixtures follow a hierarchical scope with composition:
116+
117+ 1 . ** Session fixtures** (` tests/conftest.py ` ) — cluster-wide resources (admin client, DSC, namespaces)
118+ 2 . ** Component fixtures** (` tests/<component>/conftest.py ` ) — component-scoped resources
119+ 3 . ** Test fixtures** (local ` conftest.py ` ) — per-test resources
120+
121+ Resources flow top-down: session fixtures create shared state, component fixtures build on them, test fixtures add test-specific resources. All use context managers for cleanup.
122+
123+ ### Key Conventions
124+
125+ - ` conftest.py ` files contain ** fixtures only** — no utility functions or constants
126+ - Utility functions go in ` utils.py ` (component-level) or ` utilities/<topic>_utils.py ` (shared)
127+ - All K8s resources use [ openshift-python-wrapper] ( https://github.com/RedHatQE/openshift-python-wrapper ) ; for missing resources, generate with ` class_generator ` and contribute upstream
128+
34129## Project Structure
35130
36131``` text
@@ -49,16 +144,21 @@ utilities/ # Shared utility functions
49144### Tests
50145
51146- Every test MUST have a docstring explaining what it tests (see ` tests/cluster_health/test_cluster_health.py ` )
52- - Apply relevant markers from ` pytest.ini ` : tier (` smoke ` , ` sanity ` , ` tier1 ` , ` tier2 ` ), component (` model_serving ` , ` model_registry ` , ` llama_stack ` ), infrastructure (` gpu ` , ` parallel ` , ` slow ` )
147+ - Apply relevant markers from ` pytest.ini ` : tier (` smoke ` , ` tier1 ` , ` tier2 ` ), component (` model_serving ` , ` model_registry ` , ` llama_stack ` ), infrastructure (` gpu ` , ` parallel ` , ` slow ` )
53148- Use Given-When-Then format in docstrings for behavioral clarity
54149
55150### Fixtures
56151
152+ - Every fixture MUST have a one-line docstring explaining what it does
57153- Fixture names MUST be nouns: ` storage_secret ` not ` create_secret `
58154- Use context managers for resource lifecycle (see ` tests/conftest.py:544-550 ` for pattern)
59155- Fixtures do one thing only—compose them rather than nesting
60156- Use narrowest scope that meets the need: function > class > module > session
61157
158+ ### Utilities
159+
160+ - Every utility function MUST have a full Google-format docstring including description, Args, Returns, and Raises sections
161+
62162### Kubernetes Resources
63163
64164- Use [ openshift-python-wrapper] ( https://github.com/RedHatQE/openshift-python-wrapper ) for all K8s API calls
@@ -97,6 +197,23 @@ utilities/ # Shared utility functions
97197- Skip pre-commit or type checking
98198- Create abstractions for single-use code
99199
200+ ## Key Rules (from Constitution)
201+
202+ The [ Constitution] ( ./CONSTITUTION.md ) supersedes all other docs. Critical rules:
203+
204+ - ** Simplicity first** : Favor the simplest solution; no YAGNI; every function/variable must be used or removed
205+ - ** Consistency** : Follow existing patterns; use absolute imports; import specific functions not modules
206+ - ** Security** : Never log secrets; use detect-secrets and gitleaks hooks; do not reference internal-only resources
207+ - ** Conftest discipline** : ` conftest.py ` files contain fixtures only — no utility functions or constants
208+ - ** Fixture ordering** : Call pytest native fixtures first, then session-scoped, then others
209+
210+ ## Style Summary (from Style Guide)
211+
212+ - Follow [ Google Python Style Guide] ( https://google.github.io/styleguide/pyguide.html )
213+ - Use descriptive names; meaningful names over short names; no single-letter names
214+ - Use [ Google-format] ( https://google.github.io/styleguide/pyguide.html#381-docstrings ) docstrings
215+ - Add type annotations to all new code; enforced by mypy (rules in ` pyproject.toml ` )
216+
100217## Documentation Reference
101218
102219Consult these for detailed guidance:
0 commit comments