What happened
On PR #1805, a new bypass condition was added to test.test_data_found: when verified test-result attestations exist, skip the denial and delegate enforcement to the test_attestation package. The PR included an inline comment (same collections) claiming the destination package had equivalent collection membership. The fullsend-ai-review agent ran three times (run 31598428096, run 31706850329, run 31709939640) and approved each time without catching the issue. Human reviewer st3penta identified on Aug 13 that (same collections) was incorrect for @redhat_security — four rules in test_attestation lacked redhat_security collection membership, creating an enforcement gap. The fix required adding redhat_security to four rule METADATA annotations (commit 89ded27).
What could go better
The AGENTS.md review checklist (lines 88–101) covers three items: effective_on dates, collection membership in METADATA annotations, and test coverage. None address the enforcement-delegation pattern where one rule conditionally bypasses to another package. The agent's challenger sub-agent read test_attestation.rego during the first review but had no instruction to compare collection annotations between the bypassed rule and the destination rules. Existing open issues #1801 (cross-file consistency) and #1772 (dual test-result architecture) are complementary but neither specifies this check: #1801 focuses on patterns within a PR's own files, and #1772 documents the architecture without adding a specific review step for collection parity. Confidence is high that this gap caused the miss — the agent had file access and sub-agent capacity, but no review criterion prompted the comparison.
Proposed change
Add a fourth item to the 'Review Checklist for New Policy Rules' section in AGENTS.md (after the existing three items at lines 97–101):
- **Enforcement delegation parity.** When a deny rule conditionally skips its
denial (bypass path), verify that the package receiving enforcement provides
equivalent coverage. Specifically: compare the `collections:` list of the
bypassed rule against every rule in the destination package. If the bypassed
rule belongs to a collection that the destination rules do not, the delegation
creates an enforcement gap for consumers of that collection.
This is a targeted, actionable instruction the agent can follow mechanically: read the bypassed rule's collections:, read the destination package's rules' collections:, and flag any collection present in the former but missing from any rule in the latter.
Validation criteria
On the next PR that introduces or modifies a conditional enforcement bypass between policy packages, the review agent should verify collection-membership parity and flag any gap. Retroactively, a review of PR #1805's initial diff with this instruction should produce a finding about the missing redhat_security collection on the four test_attestation rules. Measure over the next 3 PRs touching enforcement-delegation logic: the agent should not miss a collection-parity gap that a human reviewer would catch.
Generated by retro agent from #1805
What happened
On PR #1805, a new bypass condition was added to
test.test_data_found: when verified test-result attestations exist, skip the denial and delegate enforcement to thetest_attestationpackage. The PR included an inline comment(same collections)claiming the destination package had equivalent collection membership. The fullsend-ai-review agent ran three times (run 31598428096, run 31706850329, run 31709939640) and approved each time without catching the issue. Human reviewer st3penta identified on Aug 13 that(same collections)was incorrect for@redhat_security— four rules intest_attestationlackedredhat_securitycollection membership, creating an enforcement gap. The fix required addingredhat_securityto four rule METADATA annotations (commit 89ded27).What could go better
The AGENTS.md review checklist (lines 88–101) covers three items:
effective_ondates, collection membership in METADATA annotations, and test coverage. None address the enforcement-delegation pattern where one rule conditionally bypasses to another package. The agent's challenger sub-agent readtest_attestation.regoduring the first review but had no instruction to compare collection annotations between the bypassed rule and the destination rules. Existing open issues #1801 (cross-file consistency) and #1772 (dual test-result architecture) are complementary but neither specifies this check: #1801 focuses on patterns within a PR's own files, and #1772 documents the architecture without adding a specific review step for collection parity. Confidence is high that this gap caused the miss — the agent had file access and sub-agent capacity, but no review criterion prompted the comparison.Proposed change
Add a fourth item to the 'Review Checklist for New Policy Rules' section in
AGENTS.md(after the existing three items at lines 97–101):This is a targeted, actionable instruction the agent can follow mechanically: read the bypassed rule's
collections:, read the destination package's rules'collections:, and flag any collection present in the former but missing from any rule in the latter.Validation criteria
On the next PR that introduces or modifies a conditional enforcement bypass between policy packages, the review agent should verify collection-membership parity and flag any gap. Retroactively, a review of PR #1805's initial diff with this instruction should produce a finding about the missing
redhat_securitycollection on the fourtest_attestationrules. Measure over the next 3 PRs touching enforcement-delegation logic: the agent should not miss a collection-parity gap that a human reviewer would catch.Generated by retro agent from #1805