Skip to content

Commit 0366c2b

Browse files
IrakusIrakus
authored
Merge branch 'feature/tests-rework-to-pytest' into ci_update_pytest_execution
2 parents b2ead09 + 77dcb6d commit 0366c2b

5 files changed

Lines changed: 170 additions & 60 deletions

File tree

‎.gitignore‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,6 @@
1212
.env
1313
.pytest_cache
1414
.vscode
15-
!.vscode/settings.json
16-
!.vscode/launch.json
17-
!.vscode/tasks.json
18-
!.vscode/extensions.json
1915
__pycache__
2016
*.egg-info
2117
admin-pass.txt

‎tests/README.md‎

Lines changed: 100 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,39 @@
1-
# Running tests for Intel® SceneScape on Docker
1+
# Running tests for Intel® SceneScape
22

3-
## Setup environment
3+
Tests support two deployment backends controlled by the `--backend` flag:
4+
5+
| Backend | Description |
6+
| ------------ | --------------------------------------------------- |
7+
| `docker` | Docker Compose (default) |
8+
| `kubernetes` | KinD cluster + Helm chart |
9+
| `all` | Run each test against both backends |
10+
11+
## Prerequisites
12+
13+
### Docker backend
414

515
```bash
616
# Build images, generate secrets, and install the pytest virtualenv
7-
SUPASS=change_me make build-all && make setup-tests
8-
make setup-pytest # creates tests/.venv if not present
17+
SUPASS=change_me make && make setup-tests
918
```
1019

20+
### Kubernetes backend
21+
22+
In addition to the Docker prerequisites, the following tools must be installed
23+
and available on `PATH`:
24+
25+
| Tool | Purpose |
26+
| ----------- | ------------------------------ |
27+
| `kind` | Creates the local KinD cluster |
28+
| `kubectl` | Manages Kubernetes resources |
29+
| `helm` | Deploys the SceneScape chart |
30+
31+
The Python dependencies are installed automatically by `make setup-pytest`.
32+
33+
The Kubernetes backend creates a fully self-contained KinD cluster, deploys
34+
SceneScape via the Helm chart, and tears the cluster down at the end of the
35+
test session.
36+
1137
## Running tests
1238

1339
Tests are orchestrated by pytest. The `scenescape_env` fixture in
@@ -17,14 +43,11 @@ only for failed tests. Use `--collect-container-logs {failed,all,none}` to
1743
change this behavior. Test specs are defined in the individual test
1844
modules as Python dataclasses.
1945

20-
### Using make (recommended)
46+
### Running tests via make
2147

2248
Use make targets from the repository root.
2349

2450
```bash
25-
# One-time venv setup (auto-called by group targets)
26-
make setup-pytest
27-
2851
# Run all basic acceptance tests
2952
make run_basic_acceptance_tests
3053

@@ -34,28 +57,28 @@ make run_standard_tests
3457
# Run all functional tests
3558
make run_functional_tests
3659

60+
# Run all UI/Selenium tests
61+
make run_ui_tests
62+
3763
# Run all unit tests
3864
make run_unit_tests
3965

40-
# Run all UI/Selenium tests
41-
make run_ui_tests
4266

4367
```
4468

45-
### Using pytest directly
69+
### Running tests via pytest directly
4670

4771
Run from the **repository root**:
4872

4973
```bash
5074
# Activate the venv
5175
source tests/.venv/bin/activate
5276

77+
# ── Docker backend (default) ───────────────────────────────────────────────
78+
5379
# Run a single test by its pytest ID (use underscores)
5480
pytest -k mqtt_roi -v
5581

56-
# Run multiple tests matching a keyword
57-
pytest -k "mqtt" -v
58-
5982
# Run all functional tests
6083
pytest tests/functional -v
6184

@@ -65,7 +88,21 @@ pytest tests/sscape_tests -v
6588
# Run all UI tests
6689
pytest tests/ui -v
6790

68-
# Container log collection modes (default: failed)
91+
# ── Kubernetes backend ─────────────────────────────────────────────────────
92+
93+
# Run a specific test against Kubernetes
94+
pytest tests/ui/test_out_of_box.py --backend=kubernetes -v
95+
96+
# Run all Kubernetes-capable tests
97+
pytest --backend=kubernetes -v
98+
99+
# Run only tests that require Kubernetes
100+
pytest -m kubernetes_only --backend=kubernetes -v
101+
102+
# Run all tests against both backends (parametrized)
103+
pytest --backend=all -v
104+
105+
# ── Container log collection ───────────────────────────────────────────────
69106
pytest tests/functional -v --collect-container-logs failed
70107
pytest tests/functional -v --collect-container-logs all
71108
pytest tests/functional -v --collect-container-logs none
@@ -74,11 +111,11 @@ pytest tests/functional -v --collect-container-logs none
74111

75112
### Environment variables
76113

77-
| Variable | Default | Description |
78-
| --------------- | ------------------ | -------------------------------------------- |
79-
| `SUPASS` | random | Superuser password passed to test containers |
80-
| `SECRETSDIR` | `manager/secrets/` | Path to the secrets directory |
81-
| `IMAGE_VERSION` | `latest` | Docker image tag to use for test containers |
114+
| Variable | Default | Backend | Description |
115+
| --------------- | ------------------ | ---------- | -------------------------------------------- |
116+
| `SUPASS` | random | both | Superuser password for the test deployment |
117+
| `SECRETSDIR` | `manager/secrets/` | docker | Path to the secrets directory |
118+
| `IMAGE_VERSION` | `latest` | docker | Docker image tag to use for test containers |
82119

83120
### Log files
84121

@@ -126,6 +163,47 @@ or directly with pytest:
126163
pytest tests/sscape_tests -v
127164
```
128165

129-
## Running tests on kubernetes
166+
## Test markers
167+
168+
| Marker | Description |
169+
| ------------------ | ---------------------------------------------------- |
170+
| `kubernetes_only` | Test runs only with `--backend=kubernetes` or `--backend=all`; skipped for Docker |
171+
| `preserve_db` | Skip post-test DB restore so the next test can verify persistence |
172+
173+
## Using the VS Code Test Extension
174+
175+
The workspace is pre-configured for the **Python Testing** extension
176+
(`ms-python.python`). Tests are discovered automatically from the `tests/`
177+
directory.
178+
179+
### Docker backend (default)
180+
181+
No extra configuration is needed. Open the **Testing** sidebar, click
182+
**Run All Tests** or select individual tests.
183+
184+
### Kubernetes backend
185+
186+
To run tests against the Kubernetes backend from the Test Explorer, add
187+
`--backend=kubernetes` to the pytest args in your VS Code workspace settings:
188+
189+
1. Open `.vscode/settings.json` (repository root).
190+
2. Add `--backend=kubernetes` to `python.testing.pytestArgs`:
191+
192+
```json
193+
{
194+
"python.testing.pytestArgs": [
195+
"tests",
196+
"--backend=kubernetes"
197+
],
198+
"python.testing.pytestEnabled": true,
199+
"python.testing.unittestEnabled": false
200+
}
201+
```
202+
203+
3. Click **Refresh Tests** in the Testing sidebar, then run as normal.
204+
205+
### Running both backends
130206

131-
Refer to [Running tests on kubernetes](kubernetes/README.md)
207+
Set `--backend=all` to parametrize every test across Docker and Kubernetes in
208+
a single session. Each test appears in the Test Explorer as
209+
`test_name[docker]` and `test_name[kubernetes]`.

‎tests/kubernetes/README.md‎

Lines changed: 69 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,83 @@
11
# Running tests for Intel® SceneScape on Kubernetes
22

3-
Run Kubernetes tests on a Docker test host against our local Kind testing setup or a remote Kubernetes cluster.
3+
All Kubernetes tests are driven by pytest with the `--backend=kubernetes` flag.
4+
This creates a KinD cluster, deploys SceneScape via Helm, runs the tests, and
5+
destroys the cluster automatically at the end of the session. No
6+
separately-running cluster is required.
47

5-
# Usage
8+
## Prerequisites
69

7-
## Local Kubernetes cluster tests
10+
Install the following tools and make them available on `PATH`:
11+
12+
| Tool | Installation |
13+
| ----------- | ----------------------------------- |
14+
| `kind` | https://kind.sigs.k8s.io/docs/user/quick-start/#installation |
15+
| `kubectl` | https://kubernetes.io/docs/tasks/tools/ |
16+
| `helm` | https://helm.sh/docs/intro/install/ |
17+
18+
Python dependencies (`pytest-kubernetes`, `python-on-whales`) are installed
19+
automatically by `make setup-tests`.
20+
21+
## Running tests
822

923
```bash
10-
# make sure SS docker isn't running
11-
make -C kubernetes VALIDATION=1 # this starts kubernetes with tests.enabled = true in the helm chart
12-
# wait until containers are ready, check with k9s
13-
# run tests with the SUPASS in values.yaml and KUBERNETES=1
14-
make -C tests out-of-box SUPASS=change_me KUBERNETES=1
15-
# containers won't be stopped after a test, you can go ahead and execute the next test.
16-
make -C kubernetes clean-all # to clear all kubernetes infra after you are done
24+
# One-time setup — build images and create the test virtualenv
25+
SUPASS=change_me make && make setup-tests
26+
27+
# Activate the virtualenv
28+
source tests/.venv/bin/activate
29+
30+
# Run the out-of-box test on Kubernetes
31+
pytest tests/ui/test_out_of_box.py --backend=kubernetes -v
32+
33+
# Run all Kubernetes-capable tests
34+
pytest --backend=kubernetes -v
35+
36+
# Run only tests that are Kubernetes-specific
37+
pytest -m kubernetes_only --backend=kubernetes -v
1738
```
1839

19-
## Remote Kubernetes cluster tests
40+
The cluster setup (KinD creation, Helm deploy, image loading) takes roughly
41+
15–20 minutes on first run. Subsequent runs reuse the pulled images from the
42+
local Docker cache.
2043

21-
```bash
22-
# install SS on Kubernetes with tests.enabled: true in values.yaml, wait until all pods are ready
23-
# prepare docker test host on the same network
24-
# copy kubeconfig to docker test host
25-
# execute the test cases on a remote cluster with scenescape installed with tests enabled
26-
KUBECONFIG=/home/ubuntu/config make -C tests out-of-box SUPASS=change_me KUBERNETES=1 KUB_CLUSTER_FRP_ADDRESS=192.168.122.42 KUB_CLUSTER_FRP_PORT=7000
27-
# Variables to set
28-
KUBECONFIG=<absolute-path-to-remote-cluster-kubeconfig>
29-
KUB_CLUSTER_FRP_ADDRESS=<ip-of-frps-service>
30-
KUB_CLUSTER_FRP_PORT=<port-of-frps-service>
31-
KUB_CLUSTER_FRP_SECRET_KEY=<frps-secret-key>
32-
CERT_KUB_BROKER_URL=<broker-cert-url> # in the format broker.<namespace> where ns is the ns in where scenescape is installed
33-
CERT_KUB_WEB_URL=<web-cert-url> # in the format web.<namespace> where ns is the ns in where scenescape is installed
44+
## VS Code Test Extension
45+
46+
Add `--backend=kubernetes` to `python.testing.pytestArgs` in
47+
`.vscode/settings.json`:
48+
49+
```json
50+
{
51+
"python.testing.pytestArgs": [
52+
"tests",
53+
"--backend=kubernetes"
54+
],
55+
"python.testing.pytestEnabled": true,
56+
"python.testing.unittestEnabled": false
57+
}
3458
```
3559

36-
# How it works
60+
Then click **Refresh Tests** and run from the Testing sidebar as normal.
61+
62+
## How it works
3763

38-
The kubernetes `runtest`, which will be run when `make -C tests` is started with `KUBERNETES=1` does the following:
64+
When `--backend=kubernetes` is passed to pytest the `K8sManager` in
65+
`tests/utils/k8s.py` performs the following steps:
3966

40-
- expects Intel® SceneScape to be running in validation mode on a Kubernetes cluster with `tests.enabled: true`
41-
- this will start one FRP server (frps) and multiple FRP client (frpc) containers to proxy pod ports
42-
- require an additional `init-tests` image to copy our test database into our pgserver pod to run tests against
43-
- uses `kubectl` which uses the kubeconfig defined by the `KUBECONFIG` environment variable to connect to a cluster (local or remote) to manage PVCs and deployments
44-
- starts multiple [FRP](https://github.com/fatedier/frp) clients as docker containers which proxy most of the useful ports out from Kubernetes back into the docker environment, through one frps server port. These containers proxy the ports from the Kubernetes pods back into the docker test host so our existing test infrastructure does not need to be modified.
67+
1. Creates a KinD cluster named `pytest-test-cluster` using the config in
68+
`tests/kubernetes/config/kind_config.yaml`.
69+
2. Installs the Nginx Ingress Controller and cert-manager.
70+
3. Tags and loads all SceneScape Docker images (plus external dependencies
71+
parsed from `helm template` output) into the KinD node.
72+
4. Runs `make copy-files` in `kubernetes/` to populate the Helm chart files.
73+
5. Installs the Helm chart with generated values (passwords, proxy settings).
74+
6. Waits for all core services to become ready (web, scene controller,
75+
autocalibration, broker, pgserver, vdms, mediaserver, kubeclient, cameras,
76+
DL Streamer warmup).
77+
7. Extracts `controller.auth` and `scenescape-ca.pem` from Kubernetes secrets.
78+
8. Sets up `kubectl port-forward` for MQTT (`localhost:1883`) and the web
79+
service (`localhost:<random-port>`).
80+
9. Injects the port-forwarded URLs, auth file, and root certificate into the
81+
test fixtures so tests connect transparently.
4582

46-
Additional loadbalancer, service or nodeport values can be changed in the chart's values.yaml depending on the cluster.
83+
After all tests finish the cluster is torn down automatically.

‎tests/pytest.ini‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ filterwarnings =
2727
ignore::pytest.PytestCollectionWarning
2828

2929
markers =
30-
skip_init: skip init-sample-data step
3130
test_name(name): sets the XML test name attribute
3231
preserve_db: skip post-test DB restore so the next test can verify persistence
3332
kubernetes_only: test only runs with --backend=kubernetes or --backend=all

‎tests/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ paho-mqtt==2.1.0
1616
numpy==2.2.6
1717
requests==2.31.0
1818
selenium==4.15.0
19-
Pillow==12.2.0
19+
Pillow==10.0.0
2020
scikit-image==0.24.0
2121
opencv-python==4.11.0.86
2222
pyvirtualdisplay==3.0

0 commit comments

Comments
 (0)