feat(reporting): distinguish "not applicable" from "could not verify" and make unproven checks fail the run

[ian-flores]

VIP's value is the claim that a deployment was validated. The most dangerous failure mode for a validation tool is not a red run — it is a green run that proved nothing. Today those two outcomes are indistinguishable: a check that was legitimately N/A and a check that could not be executed both land in the same skipped bucket, and neither affects the exit code.

Evidence that this is a class, not a one-off

#596 (configured product never authenticates, every test skips, exit 0), #602 (test_workbench_login skips under SSO, so no lane verifies a Workbench login at all), #606 (test_connect_login_ui can never fail — its only assertion is satisfied before credentials are submitted), #550 and #546 (suites that cannot run in CI because [runtimes] is unpopulated and nothing terminates TLS). Each has been triaged individually; nothing prevents the next one.

Structural evidence

src/vip_tests contains 160 pytest.skip() call sites. ReportData (src/vip/reporting.py:90) counts skips as a single undifferentiated total, and the only classification that exists is na_version. Exit status is pytest's raw exitstatus (src/vip/plugin.py:1266), so a run in which every meaningful check skipped exits 0 and renders a report with zero failures. To someone reading that report as an audit artifact, "we did not check" and "we checked and it is fine" look identical.

Proposal

Introduce a third outcome, unproven, sitting between passed and failed.

1. Classify at the skip site. vip.attest.not_applicable(reason) marks a skip as deliberate and non-fatal (product not configured, tier lacks the feature, IDE not installed) and keeps it out of the exit code; vip.attest.unproven(reason) marks one VIP was asked to run and could not. A bare pytest.skip() keeps the not_applicable meaning, so sites convert deliberately rather than by flag day — defaulting to unproven would make "product not configured" fatal, which is exactly the case fix(cli): vip verify exits 0 when a configured product never authenticates #596's note on the reverse risk says must stay non-fatal.
2. Surface it everywhere the outcome is reported: report counters and badges, JUnit (unproven as a distinct message), SARIF (warning, between none and error).
3. Make it affect the exit code, behind --allow-unproven for anyone who needs the old behaviour.
4. Split the report's top line three ways — Verified / Failed / Not verified — with "not verified" broken into "N/A for this version" and "could not run here".

The tradeoff, stated plainly

This makes VIP louder on day one. Existing lanes that quietly skipped will start failing, and the first run against a real deployment will surface gaps nobody has had to look at. That is the intended effect, but it is a real behaviour change and the escape hatch exists for that reason.

Why it is worth it

A tool that reports "31 of your 47 configured capabilities were verified, here are the 16 I could not reach and why" is more credible as an IQ/OQ artifact than one that reports 47/47 green with the interesting half silently skipped. #596 already names the required distinction — "separate 'nothing to do here' from 'we could not do what was asked'" — this issue supplies the mechanism, and closes #596 as its first consumer.

Scope note

Landing in slices: this issue covers the unproven outcome, its plumbing through the report formats, and the exit-code contract. Retriaging the existing 160 skip sites is follow-up work, as is the capability-attestation layer (asserting that a declared capability had a check that actually exercised it) and a mutation meta-test that proves each scenario can fail.

[ian-flores]

Slice 1 shipped in #617 (merged): the unproven outcome, vip.attest, plumbing through report/JUnit/SARIF/PDF, and the exit-code contract (6, with --allow-unproven). #596 is closed by it.

Two things landed differently from this issue as filed. The polarity is opt-in rather than default, for the reason now in Proposal item 1. And summary_status gained an UNPROVEN verdict, so a report where nothing was verified no longer headlines PASS — that fell out of the collision with #618's report refactor and is arguably the most valuable single line of the change.

Remaining, and what this issue now tracks: retriaging the ~160 existing pytest.skip() sites; the capability-attestation layer (asserting a declared capability had a check that actually exercised it); and the mutation meta-test that proves each scenario can fail, which is what would have caught #606.

Worth recording that the exit-6 path has not yet run against a real deployment — every smoke lane passes because authentication succeeds there. #263's SAML lane is where it would first be exercised for real.