Skip to content

feat(verifier): support bounded certified test plans - #10

Merged
Punisheroot merged 2 commits into
mainfrom
feat/js/bounded-certified-test-plans
Aug 3, 2026
Merged

feat(verifier): support bounded certified test plans#10
Punisheroot merged 2 commits into
mainfrom
feat/js/bounded-certified-test-plans

Conversation

@Punisheroot

@Punisheroot Punisheroot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #5.

Needle's independent verifier can now process a deterministic set of up to four distinct certified TestPlans. Every available plan is executed sequentially, at most once, using its exact canonical command, and a change cannot be marked verified unless the complete expected plan set has fresh, successful runtime evidence.

Over-cap, stale, unavailable, failed, partial, mismatched, or undeclared plan evidence remains fail-closed.

Implementation

  • Add a stable plan identity over the canonical runner, argv, relative working directory, and test identifier, excluding execution provenance.
  • Collect certified artifact and claim plans into an exactly deduplicated, digest-ordered set with a hard limit of four distinct plans.
  • Preserve plans with differing commands or identifiers as separate identities and never execute a truncated subset after cap overflow.
  • Reuse certificate freshness, dependency, evidence-path, trusted-repository, and test-command validation.
  • Add verifier-specific ordered approval state enforcing exact argv/cwd, sequential execution, once-per-plan limits, and an in-flight evidence gate.
  • Persist the complete per-plan expected, available, executed, passed, evidence, and failure projection in canonical verification artifacts and change attempts.
  • Enforce the complete-plan invariant during output normalization, canonical artifact admission, persistence, and verified-change application.
  • Version the verifier definition while retaining backward-compatible decoding and identity behavior for legacy artifacts.
  • Expose plan results through MCP and the web control plane.
  • Extend the offline simulator and fixtures with one-, two-, and four-plan execution, exact duplicates, cap overflow, stale dependencies, evidence mismatch, partial execution, ordering, and cleanup coverage.

Validation

cd crates/needle-app/web
npm ci
# completed; 415 packages audited, with 2 high-severity advisories reported;
# no dependency or lockfile changes

npm run build
# passed; 1,964 modules transformed

npm run typecheck
# passed

npm test -- --run
# 2 passed

cd ../../..

cargo test --locked -p needle-core --lib
# 59 passed

cargo test --locked -p needle-runtime --lib
# 107 passed

cargo test --locked -p needle-platform-codex --lib
# 81 passed, 1 ignored (MSVC-specific)

cargo test --locked -p needle-platform-codex --test patcher_offline
# 3 passed

cargo test --locked -p needle-platform-codex --test offline_n1
# 6 passed

cargo test --locked -p needle-platform-codex --test main_interrupt
# 14 passed

cargo test --locked --workspace
# 383 passed, 1 ignored

cargo clippy --locked --workspace --all-targets -- -D warnings
# passed

cargo fmt --all --check
# passed

git diff --check
# passed

No provider-backed verifier or live repository command was run; validation used the existing offline simulator and fixtures.

Risks/limitations

  • Certified commands intentionally remain sequential and capped at four, so verifier latency can approach the sum of the individual command runtimes.
  • The compact web table prioritizes the evidence ID when both evidence and a failure reason are present. The complete failure reason remains available in the API and persisted artifact.
  • A pre-existing error path while reading runtime settings after sandbox materialization can leave that temporary sandbox behind. It occurs before session or command execution and is outside this change's authority and correctness boundary.
  • npm ci reports two high-severity dependency advisories; this change does not modify dependencies or the lockfile.
  • The artifact schema additions use defaulted fields and require no database migration. Legacy artifacts continue to decode, while the new verifier-definition digest prevents stale cache reuse under the updated contract.

Documentation/evidence

  • Updated docs/VERIFIED_CHANGES.md with the bounded multi-plan verification contract and safety boundary.
  • Updated PROJECT_STATUS.md to replace the previous one-plan limitation with the implemented four-plan bound.
  • Added deterministic offline evidence for successful execution, deduplication, overflow, stale dependencies, adversarial evidence handling, and sandbox cleanup.

AI assistance and human verification

AI assistance: code assistance only. AI was used only for code assistance.
Human verification: I reviewed the complete 14-file issue #5 diff against the issue acceptance criteria, including canonical identity, deterministic ordering and deduplication, cap overflow, exact-command approval, runtime evidence binding, fail-closed normalization, legacy compatibility, persistence and apply admission, and offline cleanup. I also verified the reported validation results and confirmed that the issue-scoped diff passes formatting and diff checks.

Ownership checklist

  • I read and understand the complete diff and can explain every material change.
  • I personally verified the reported tests and technical or performance claims.
  • I finalized and personally published the commits and this pull request.
  • I checked for unrelated edits, credentials, generated files, and live artifacts.
  • Public documentation and PROJECT_STATUS.md are updated when applicable.

@Punisheroot
Punisheroot merged commit 20eeeb0 into main Aug 3, 2026
2 checks passed
@Punisheroot
Punisheroot deleted the feat/js/bounded-certified-test-plans branch August 3, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support bounded multiple certified TestPlans in the independent verifier

1 participant