Open
Conversation
…os. Enhance models and videos test cases.
…e-platform/edge-ai-libraries into vippet-functional-tests
…e-platform/edge-ai-libraries into vippet-functional-tests
This reverts commit d25764e.
ktaube26
commented
Mar 6, 2026
tools/visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/test_pipelines.py
Outdated
Show resolved
Hide resolved
tools/visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/test_pipelines.py
Outdated
Show resolved
Hide resolved
tools/visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/test_pipelines.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Dariusz Lemiech <dariusz.lemiech@intel.com>
p-zak
requested changes
Mar 16, 2026
tools/visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/helpers/config.py
Show resolved
Hide resolved
.../visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/helpers/api_helpers.py
Outdated
Show resolved
Hide resolved
tools/visual-pipeline-and-platform-evaluation-tool/vippet/tests/functional/README.md
Show resolved
Hide resolved
Contributor
|
It might be worth considering adding functionality that checks whether the entire available API is covered by the functional tests. The main goal would be to have a kind of reminder that whenever we add or change something in the API, we should also adjust the functional tests. As a first step, this could simply check whether all endpoints have been called at least once (regardless of whether it’s the happy path or not). (this doesn’t necessarily have to be done in this PR) |
p-zak
approved these changes
Mar 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a functional test suite for the VIPPET API. The functional tests exercise the live API over HTTP and are designed to run against a deployed VIPPET instance.
Key changes:
New functional test suite under
vippet/tests/functional/covering:test_health.py–GET /healthandGET /statusendpointstest_devices_list.py–GET /devicesschema validationtest_videos_list.py–GET /videosschema validation and default recordings presencetest_models_list.py–GET /modelsschema validation and YAML config cross-checks (default models, all models)test_cameras.py– Camera CRUD, profile loading, and error casestest_pipelines.py– Pipeline/variant CRUD, read-only enforcement on PREDEFINED pipelines,convert-to-simple/advancedgraph endpointstest_pipeline_templates.py– Pipeline template listing and retrievaltest_convert.py–POST /convert/to-graphandPOST /convert/to-descriptionhappy paths and error casestest_density_job_flow.py– Full density test job lifecycle (submit → poll → COMPLETED), parametrized across all discovered device familiestest_performance_job_flow.py– Full performance test job lifecycle withdisabled,file, andlive_streamoutput modes, parametrized across all discovered device familiestest_performance_job_usb_camera.py– Performance jobs driven by a live USB camera input; verifies camera-compatible pipelines stay RUNNING and file-only pipelines correctly reach FAILEDtest_pipeline_optimize_flow.py–preprocessandoptimizepipeline optimization job flowstest_pipelines_validate.py– Pipeline validation job flowtest_jobs_management.py– Job status, summary, and stop endpoints for performance, density, and optimization jobsShared infrastructure:
config.py– Centralised environment-variable-driven configuration (VIPPET_BASE_URL,VIPPET_JOB_TIMEOUT_SECONDS,VIPPET_JOB_POLL_INTERVAL)conftest.py– Session-scopedhttp_client,supported_models_config, anddefault_recordings_configfixtureshelpers/api_helpers.py– Reusable HTTP helpers (fetch_*,start_performance_job,wait_for_job_completion,poll_job_not_failed,run_job_with_retry, etc.)helpers/pipeline_case_helpers.py– Dynamic pipeline/variant discovery keyed on live device families; yields pytest parametrize values with graceful skip when no cases are foundpytest.ini–smokeandfullmarker definitions with structured log formatTest classification via markers:
@pytest.mark.smoke– Fast, read-only API queries; no jobs submitted@pytest.mark.full– Submit jobs and poll until completion; may take several minutesUnit test reorganisation: All existing unit tests moved from
vippet/tests/intovippet/tests/unit/(flat rename, no content changes) to cleanly separate them from the new functional suite.Makefile targets added:
make test-smoke– Run only smoke functional tests viapytest -m smokemake test-full– Run the complete functional test suite viapytestmake testupdated to discover unit tests fromtests/unit/.gitignoreupdates: Addedshared/videos/output/*andshared/onvif/*New dependency:
pytest==9.0.2added torequirements-dev.txtHow Has This Been Tested?
The functional tests are designed to run against a live VIPPET API instance. To execute:
Configuration (via environment variables):
VIPPET_BASE_URLhttp://localhost/api/v1VIPPET_JOB_TIMEOUT_SECONDS600VIPPET_JOB_POLL_INTERVAL2.0Existing unit tests verified to pass without regressions after the
tests/→tests/unit/rename.Checklist: