You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
skip_suite under RELEASE_GATE=1 is a HARD FAIL by design, to catch the silent-success class (#870/#871/#888) where a suite quietly skips and the gate goes green without actually testing anything.
That default is correct. But several suites legitimately skip_suite because a capability is genuinely not provisioned / not shipped in the gate deploy, which is an environment fact, not a code defect. With the hard-fail default, those suites fail the gate even though there is nothing for the backend to fix.
Observed in release-gate run 26616763325 (5 hard-failing skip_suites, all not-provisioned):
Suite
Reason
Why not a backend bug
security/test-auto-scan-on-upload.sh
scan-config endpoint 404 (POST and PUT)
auto-scan-on-upload feature not shipped on the gate backend image
security/test-scheduled-scan.sh
scan-schedules endpoint 404
scheduled-scan feature not shipped on the gate backend image
security/test-dependencytrack-integration.sh
DEPENDENCY_TRACK_API_KEY/DEPENDENCY_TRACK_URL unset (and 404 when integration route absent)
Dependency-Track is not deployed in the gate namespace (see #200)
security/test-openscap-scanner.sh
scanners.openscap_enabled=false and OPENSCAP_URL unset
OpenSCAP sidecar is not provisioned in the gate deploy
platform/test-wasm-plugin-roundtrip.sh
plugin list empty / no plugin list endpoint
no WASM plugin fixture is loaded against the gate backend (helm/values-test.yaml plugins.enabled)
Additionally:
mesh/test-sync-filter-application.sh run-now (POST /api/v1/sync-policies/{id}/run) is documented as an unshipped sync worker (TODO Epic 12: Federation depth E2E coverage (v1.1.9) #78.4, artifact-keeper-fzj). A 404 there is "endpoint not shipped", not a backend defect.
Approach
Add a small, explicit, documented allowlist of capability keys to tests/lib/common.sh (where skip_suite lives). Each entry carries:
When skip_suite is called under RELEASE_GATE=1 with a reason that matches an allowlisted capability, emit EXEMPT: <capability> (tracked by #N), write a JUnit <skipped/> testcase (not a failure), and exit 0.
Any OTHER skip_suite under RELEASE_GATE=1 continues to hard-fail. This preserves the silent-success protection for everything that is not a known not-provisioned capability.
Out of scope (must keep hard-failing)
Any skip that could mask a real backend bug.
twine upload 401 in formats/test-pypi-native-client.sh: the test already passes valid Basic credentials (verified identical to the curl -u upload that PASSES in the same gate). This is not a not-provisioned capability and is left hard-failing pending backend triage.
Tracking
This issue tracks the allowlist itself. Per-capability provisioning work is tracked separately (DTrack: #200).
Problem
skip_suiteunderRELEASE_GATE=1is a HARD FAIL by design, to catch the silent-success class (#870/#871/#888) where a suite quietly skips and the gate goes green without actually testing anything.That default is correct. But several suites legitimately
skip_suitebecause a capability is genuinely not provisioned / not shipped in the gate deploy, which is an environment fact, not a code defect. With the hard-fail default, those suites fail the gate even though there is nothing for the backend to fix.Observed in release-gate run 26616763325 (5 hard-failing skip_suites, all not-provisioned):
security/test-auto-scan-on-upload.shsecurity/test-scheduled-scan.shsecurity/test-dependencytrack-integration.shDEPENDENCY_TRACK_API_KEY/DEPENDENCY_TRACK_URLunset (and 404 when integration route absent)security/test-openscap-scanner.shscanners.openscap_enabled=falseandOPENSCAP_URLunsetplatform/test-wasm-plugin-roundtrip.shhelm/values-test.yaml plugins.enabled)Additionally:
mesh/test-sync-filter-application.shrun-now (POST /api/v1/sync-policies/{id}/run) is documented as an unshipped sync worker (TODO Epic 12: Federation depth E2E coverage (v1.1.9) #78.4, artifact-keeper-fzj). A 404 there is "endpoint not shipped", not a backend defect.Approach
Add a small, explicit, documented allowlist of capability keys to
tests/lib/common.sh(whereskip_suitelives). Each entry carries:dependency_track,openscap,scan_config_autoscan,scan_schedules,wasm_plugin_fixture,mesh_run_now)skip_suitereasonWhen
skip_suiteis called underRELEASE_GATE=1with a reason that matches an allowlisted capability, emitEXEMPT: <capability> (tracked by #N), write a JUnit<skipped/>testcase (not a failure), and exit 0.Any OTHER
skip_suiteunderRELEASE_GATE=1continues to hard-fail. This preserves the silent-success protection for everything that is not a known not-provisioned capability.Out of scope (must keep hard-failing)
twine upload401 informats/test-pypi-native-client.sh: the test already passes valid Basic credentials (verified identical to the curl-uupload that PASSES in the same gate). This is not a not-provisioned capability and is left hard-failing pending backend triage.Tracking
This issue tracks the allowlist itself. Per-capability provisioning work is tracked separately (DTrack: #200).