Skip to content

[QA] Add cross-agent conformance and E2E matrix #27

Description

@teseo

Kind

acceptance-gate

Goal

Prove that every support claim in the matrix is backed by repeatable conformance and lifecycle tests.

Non-goals

  • Re-testing undocumented agent behavior without a fixture.
  • Replacing host-specific smoke tests.
  • Treating advisory behavior as enforcement.

Invariants

  • Every adapter runs the same canonical decision scenarios.
  • Tests distinguish allow, deny, advisory, timeout, stale state, and session end.
  • Fail-open and fail-closed semantics are asserted explicitly.
  • Test artifacts identify agent, version, surface, and result.

Behavioral contract / States

A conformance runner provisions an isolated home, installs one adapter, executes the shared scenario set, and emits machine-readable results plus a human summary.

Scope surface

tests/conformance, tests/test.sh, adapter fixtures, CI configuration, and result documentation.

Acceptance

  • Add scenarios for outside-hours, blocked period, wind-down, max session, stale state, concurrent sessions, and uninstall.
  • Run: bash tests/test.sh, bash tests/test_install.sh, python3 -m pytest tests/test_guard.py -v, and node --test tests/test_guard.mjs.
  • CI fails when a claimed full-support adapter lacks a passing conformance result.
  • Publish one result artifact per supported agent/surface.

Split triggers

  • A host cannot execute the shared fixture model.
  • A desktop-only scenario requires a separate manual acceptance gate.

Dependencies / Parent

Blocked by: #4, #5, #6, #7, #8, #9, #10, #11
Parent Epic: #15

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions