Skip to content

[gate] Documented capability-exemption allowlist for not-provisioned gate suites #211

@brandonrc

Description

@brandonrc

Problem

skip_suite under RELEASE_GATE=1 is a HARD FAIL by design, to catch the silent-success class (#870/#871/#888) where a suite quietly skips and the gate goes green without actually testing anything.

That default is correct. But several suites legitimately skip_suite because a capability is genuinely not provisioned / not shipped in the gate deploy, which is an environment fact, not a code defect. With the hard-fail default, those suites fail the gate even though there is nothing for the backend to fix.

Observed in release-gate run 26616763325 (5 hard-failing skip_suites, all not-provisioned):

Suite Reason Why not a backend bug
security/test-auto-scan-on-upload.sh scan-config endpoint 404 (POST and PUT) auto-scan-on-upload feature not shipped on the gate backend image
security/test-scheduled-scan.sh scan-schedules endpoint 404 scheduled-scan feature not shipped on the gate backend image
security/test-dependencytrack-integration.sh DEPENDENCY_TRACK_API_KEY/DEPENDENCY_TRACK_URL unset (and 404 when integration route absent) Dependency-Track is not deployed in the gate namespace (see #200)
security/test-openscap-scanner.sh scanners.openscap_enabled=false and OPENSCAP_URL unset OpenSCAP sidecar is not provisioned in the gate deploy
platform/test-wasm-plugin-roundtrip.sh plugin list empty / no plugin list endpoint no WASM plugin fixture is loaded against the gate backend (helm/values-test.yaml plugins.enabled)

Additionally:

  • mesh/test-sync-filter-application.sh run-now (POST /api/v1/sync-policies/{id}/run) is documented as an unshipped sync worker (TODO Epic 12: Federation depth E2E coverage (v1.1.9) #78.4, artifact-keeper-fzj). A 404 there is "endpoint not shipped", not a backend defect.

Approach

Add a small, explicit, documented allowlist of capability keys to tests/lib/common.sh (where skip_suite lives). Each entry carries:

  • a stable capability key (e.g. dependency_track, openscap, scan_config_autoscan, scan_schedules, wasm_plugin_fixture, mesh_run_now)
  • a substring matched against the skip_suite reason
  • a tracking-issue reference

When skip_suite is called under RELEASE_GATE=1 with a reason that matches an allowlisted capability, emit EXEMPT: <capability> (tracked by #N), write a JUnit <skipped/> testcase (not a failure), and exit 0.

Any OTHER skip_suite under RELEASE_GATE=1 continues to hard-fail. This preserves the silent-success protection for everything that is not a known not-provisioned capability.

Out of scope (must keep hard-failing)

  • Any skip that could mask a real backend bug.
  • twine upload 401 in formats/test-pypi-native-client.sh: the test already passes valid Basic credentials (verified identical to the curl -u upload that PASSES in the same gate). This is not a not-provisioned capability and is left hard-failing pending backend triage.

Tracking

This issue tracks the allowlist itself. Per-capability provisioning work is tracked separately (DTrack: #200).

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-gateHardening Coretest-calibrationTest needs adjustment to match actual API behavior

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions