Skip to content

[ViPPET] Add functional tests#1891

Open
ktaube26 wants to merge 52 commits intomainfrom
vippet-functional-tests
Open

[ViPPET] Add functional tests#1891
ktaube26 wants to merge 52 commits intomainfrom
vippet-functional-tests

Conversation

@ktaube26
Copy link
Contributor

@ktaube26 ktaube26 commented Mar 4, 2026

Description

This PR adds a functional test suite for the VIPPET API. The functional tests exercise the live API over HTTP and are designed to run against a deployed VIPPET instance.

Key changes:

  • New functional test suite under vippet/tests/functional/ covering:

    • test_health.pyGET /health and GET /status endpoints
    • test_devices_list.pyGET /devices schema validation
    • test_videos_list.pyGET /videos schema validation and default recordings presence
    • test_models_list.pyGET /models schema validation and YAML config cross-checks (default models, all models)
    • test_cameras.py – Camera CRUD, profile loading, and error cases
    • test_pipelines.py – Pipeline/variant CRUD, read-only enforcement on PREDEFINED pipelines, convert-to-simple/advanced graph endpoints
    • test_pipeline_templates.py – Pipeline template listing and retrieval
    • test_convert.pyPOST /convert/to-graph and POST /convert/to-description happy paths and error cases
    • test_density_job_flow.py – Full density test job lifecycle (submit → poll → COMPLETED), parametrized across all discovered device families
    • test_performance_job_flow.py – Full performance test job lifecycle with disabled, file, and live_stream output modes, parametrized across all discovered device families
    • test_performance_job_usb_camera.py – Performance jobs driven by a live USB camera input; verifies camera-compatible pipelines stay RUNNING and file-only pipelines correctly reach FAILED
    • test_pipeline_optimize_flow.pypreprocess and optimize pipeline optimization job flows
    • test_pipelines_validate.py – Pipeline validation job flow
    • test_jobs_management.py – Job status, summary, and stop endpoints for performance, density, and optimization jobs
  • Shared infrastructure:

    • config.py – Centralised environment-variable-driven configuration (VIPPET_BASE_URL, VIPPET_JOB_TIMEOUT_SECONDS, VIPPET_JOB_POLL_INTERVAL)
    • conftest.py – Session-scoped http_client, supported_models_config, and default_recordings_config fixtures
    • helpers/api_helpers.py – Reusable HTTP helpers (fetch_*, start_performance_job, wait_for_job_completion, poll_job_not_failed, run_job_with_retry, etc.)
    • helpers/pipeline_case_helpers.py – Dynamic pipeline/variant discovery keyed on live device families; yields pytest parametrize values with graceful skip when no cases are found
    • pytest.inismoke and full marker definitions with structured log format
  • Test classification via markers:

    • @pytest.mark.smoke – Fast, read-only API queries; no jobs submitted
    • @pytest.mark.full – Submit jobs and poll until completion; may take several minutes
  • Unit test reorganisation: All existing unit tests moved from vippet/tests/ into vippet/tests/unit/ (flat rename, no content changes) to cleanly separate them from the new functional suite.

  • Makefile targets added:

    • make test-smoke – Run only smoke functional tests via pytest -m smoke
    • make test-full – Run the complete functional test suite via pytest
    • Existing make test updated to discover unit tests from tests/unit/
  • .gitignore updates: Added shared/videos/output/* and shared/onvif/*

  • New dependency: pytest==9.0.2 added to requirements-dev.txt

How Has This Been Tested?

The functional tests are designed to run against a live VIPPET API instance. To execute:

# Smoke tests only (fast, no job submission)
make test-smoke

# Full functional test suite
make test-full

# Or directly with pytest
python3 -m pytest vippet/tests/functional/
python3 -m pytest -m smoke vippet/tests/functional/

Configuration (via environment variables):

Variable Default Description
VIPPET_BASE_URL http://localhost/api/v1 Base URL of the running VIPPET API
VIPPET_JOB_TIMEOUT_SECONDS 600 Max polling time for job completion
VIPPET_JOB_POLL_INTERVAL 2.0 Polling interval in seconds

Existing unit tests verified to pass without regressions after the tests/tests/unit/ rename.

Checklist:

  • I agree to use the APACHE-2.0 license for my code changes.
  • I have not introduced any 3rd party components incompatible with APACHE-2.0.
  • I have not included any company confidential information, trade secret, password or security token.
  • I have performed a self-review of my code.

@ktaube26 ktaube26 added the vippet label Mar 4, 2026
ktaube26 and others added 22 commits March 4, 2026 00:55
@ktaube26 ktaube26 marked this pull request as ready for review March 16, 2026 09:22
@p-zak
Copy link
Contributor

p-zak commented Mar 16, 2026

It might be worth considering adding functionality that checks whether the entire available API is covered by the functional tests. The main goal would be to have a kind of reminder that whenever we add or change something in the API, we should also adjust the functional tests.

As a first step, this could simply check whether all endpoints have been called at least once (regardless of whether it’s the happy path or not).
Alternatively, it might be enough to collect test coverage for files/methods, but only for those located in api/routes - that would already cover more than just the happy paths.

(this doesn’t necessarily have to be done in this PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants