Skip to content

feat(quality): add AI-assisted OSPS-QA-06.02 evaluation#346

Open
vinayada1 wants to merge 1 commit into
ossf:mainfrom
vinayada1:feat/osps-qa-06-02-ai-validation
Open

feat(quality): add AI-assisted OSPS-QA-06.02 evaluation#346
vinayada1 wants to merge 1 commit into
ossf:mainfrom
vinayada1:feat/osps-qa-06-02-ai-validation

Conversation

@vinayada1

@vinayada1 vinayada1 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements an AI-assisted evaluator for OSPS-QA-06.02 (documentation explains when and how tests are run), replacing the previous hard-coded NeedsReview stub.

  • Maps OSPS-QA-06.02 to the new quality.TestExecutionDocumentation evaluator.
  • Sends a conservative evidence boundary (README + CONTRIBUTING) to the configured AI provider and parses a strict, schema-validated verdict.
  • Persists AI evidence packets via the SDK writer (sdkai.WritePacket), gated behind the SDK's double opt-in (config.Write + ai_write_evidence: true), with SDK-side redaction.
  • Caches successful verdicts per run; provider/config/schema failures remain retryable.

Backward compatibility

  • AI not configured: behavior is identical to today. When Config is nil, the client is nil, or ai_provider/ai_api_key are missing, the evaluator returns the same gemara.NeedsReview result and message as the previous stub. All failure paths fall back to the same NeedsReview verdict.
  • SDK version: requires privateer-sdk v1.28.0 symbols (WritePacket, PacketAttempt, NewSanitizer, RedactPatterns, NewClientFromConfig). go.mod already pins v1.28.0, and Go's minimum version selection guarantees the floor at build time, so there is no silent old-SDK runtime risk.

@vinayada1 vinayada1 requested a review from a team as a code owner June 12, 2026 15:05
@vinayada1 vinayada1 force-pushed the feat/osps-qa-06-02-ai-validation branch from ad4eb4f to 833cf87 Compare June 12, 2026 15:07
Signed-off-by: Vinaya Damle <vinayada1@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant