What happened
PR #1736 added new deny rules for experimental Hermeto backends across SPDX and CycloneDX SBOM policy files. The fullsend-ai-review bot ran 4 successful review iterations (starting June 19) and flagged several legitimate issues (annotator string mismatch, pattern duplication, OPA rule conflicts, test gaps). However, it never flagged that the new deny rules were missing effective_on dates — a convention used throughout this codebase to provide migration windows before new enforcement takes effect. Human reviewer st3penta caught this on June 22, noting the rules would enforce immediately on deployment. The PR author subsequently added an effective_on date of 2026-08-01.
What could go better
The review agent had access to AGENTS.md which mentions effective_on in passing ("These files have effective_on dates — rules with future dates are warnings, not failures") but only in the context of rule data files in example/data/, not as a review checklist item for new policy rules. The agent explored the codebase and saw many existing rules using effective_on, but did not connect the convention to the new rules being added. Confidence: high that adding explicit review guidance would help — the pattern is well-established (61 files reference effective_on) and the agent clearly understands conventions when they are explicitly stated in AGENTS.md. Note: conforma/policy issue #1749 covers documenting effective_on usage for policy authors, but that is about authoring documentation, not review-time checking.
Proposed change
Add a review checklist section to AGENTS.md in conforma/policy. Under "Common Change Patterns" or as a new "Review Checklist" section, add guidance like:
## Review Checklist for New Policy Rules
- **effective_on date required:** New deny/warn rules MUST include an `effective_on` date in their rule data to provide a migration window. Rules without `effective_on` enforce immediately on deployment, which can break existing builds without warning. Check `example/data/` for the rule data entry and verify it has a future `effective_on` date.
- **Collection membership:** New rules must be added to the appropriate collection(s) in `policy/*/collection/` or they won't be evaluated.
This is repo-specific guidance and belongs in the source repo, not upstream.
Validation criteria
The next 3 PRs to conforma/policy that add new deny or warn rules should have the review agent flag missing effective_on dates if they are absent. Verify by checking review agent comments on those PRs for mention of effective_on.
Generated by retro agent from #1736
What happened
PR #1736 added new deny rules for experimental Hermeto backends across SPDX and CycloneDX SBOM policy files. The fullsend-ai-review bot ran 4 successful review iterations (starting June 19) and flagged several legitimate issues (annotator string mismatch, pattern duplication, OPA rule conflicts, test gaps). However, it never flagged that the new deny rules were missing
effective_ondates — a convention used throughout this codebase to provide migration windows before new enforcement takes effect. Human reviewer st3penta caught this on June 22, noting the rules would enforce immediately on deployment. The PR author subsequently added aneffective_ondate of 2026-08-01.What could go better
The review agent had access to AGENTS.md which mentions
effective_onin passing ("These files haveeffective_ondates — rules with future dates are warnings, not failures") but only in the context of rule data files inexample/data/, not as a review checklist item for new policy rules. The agent explored the codebase and saw many existing rules usingeffective_on, but did not connect the convention to the new rules being added. Confidence: high that adding explicit review guidance would help — the pattern is well-established (61 files referenceeffective_on) and the agent clearly understands conventions when they are explicitly stated in AGENTS.md. Note: conforma/policy issue #1749 covers documentingeffective_onusage for policy authors, but that is about authoring documentation, not review-time checking.Proposed change
Add a review checklist section to
AGENTS.mdin conforma/policy. Under "Common Change Patterns" or as a new "Review Checklist" section, add guidance like:This is repo-specific guidance and belongs in the source repo, not upstream.
Validation criteria
The next 3 PRs to conforma/policy that add new deny or warn rules should have the review agent flag missing
effective_ondates if they are absent. Verify by checking review agent comments on those PRs for mention ofeffective_on.Generated by retro agent from #1736