Skip to content

Commit 6d0939e

Browse files
committed
Block PRs that use AI without disclosing it
Add an "AI Use Without Disclosure" pre-merge check in error mode that flags a blocker when a PR shows AI assistance (a named AI tool or agent, an AI trailer or bot commit author, or two indirect signals) but the PR template's AI Disclosure section is deleted, unedited, still placeholders, or answers None against direct evidence. The existing "AI Contribution Disclosure" check stays a warning and now only reviews disclosure quality, accepting the PR template's table as well as the bullet form.
1 parent 8eddcf9 commit 6d0939e

3 files changed

Lines changed: 80 additions & 29 deletions

File tree

‎.coderabbit.yaml‎

Lines changed: 67 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -658,9 +658,10 @@ reviews:
658658
# (a rule changed without its test, a `.conf` regex changed without its
659659
# `.ra`) that no per-file instruction can express.
660660
#
661-
# All checks are mode: warning. Escalate an individual check to mode: error
662-
# (paired with request_changes_workflow: true) only after confirming a low
663-
# false-positive rate on real CRS PRs. Check names are capped at 50 chars
661+
# Checks are mode: warning, except "AI Use Without Disclosure" (error). An
662+
# error-mode check blocks merge only while auto approvals are enabled
663+
# (request_changes_workflow: true). Escalate another check to mode: error
664+
# only after confirming a low false-positive rate on real CRS PRs. Check names are capped at 50 chars
664665
# and instructions at 10,000 chars by the schema.
665666
pre_merge_checks:
666667
description:
@@ -941,25 +942,72 @@ reviews:
941942
a rule ID retired and left unused rather than reassigned; a new
942943
default introduced alongside the old one with the switch documented.
943944
945+
# The one check in error mode. It blocks merge only while
946+
# reviews.auto_approvals (request_changes_workflow) is enabled. Kept
947+
# separate from the adequacy check below so that a missing `## refs` or
948+
# a vague field stays a warning.
949+
- name: "AI Use Without Disclosure"
950+
mode: error
951+
instructions: |
952+
coreruleset's AI-CONTRIBUTIONS.md policy requires every PR where AI
953+
tools materially assisted to disclose it in the PR body. Undisclosed
954+
AI-generated content may be closed without merge.
955+
956+
Return Passed with "not applicable" for a trivial PR (a typo fix, a
957+
version bump) or a Renovate/Dependabot update.
958+
959+
The coreruleset PR template ships an `## AI Disclosure` heading with
960+
the line `**AI usage for this contribution:** None / Used` and a
961+
`| Detail | Response |` table, so the heading alone proves nothing.
962+
The disclosure counts as made only when the PR body (in any heading
963+
case, template table or `**tools used**` bullets) says AI was Used
964+
and names at least the tool. If it does, return Passed with
965+
"disclosure present; adequacy is reviewed by AI Contribution
966+
Disclosure".
967+
968+
It counts as missing when the section was deleted, when the answer
969+
is still the unedited `None / Used` or blank, when it says Used but
970+
the table is still placeholders, or when it says None while a
971+
direct signal below shows AI use. In that case flag ❌ blocker: AI
972+
assistance detected without a disclosure — fill in the
973+
`## AI Disclosure` section (tools used, what was AI-assisted, review
974+
performed) per AI-CONTRIBUTIONS.md. Quote the evidence you relied
975+
on. Flag only when at least one direct signal is present, or at
976+
least two indirect ones:
977+
978+
Direct signals: the PR body, a commit message, or a code comment
979+
names an AI tool or agent (Devin, Copilot, Claude, ChatGPT, Cursor,
980+
Codex, Gemini, ...); a `Co-Authored-By` trailer or "Generated
981+
with" line from an AI tool; the author account or a commit author is
982+
an AI agent or bot that is not Renovate/Dependabot.
983+
984+
Indirect signals: generated-looking docstrings or comments at
985+
uniform density; a large, mechanically uniform batch of tests; a
986+
templated description that restates the diff; a large diff squashed
987+
into one commit with no incremental history; claims of verification
988+
("all tests pass", "no false positives") with no command or output
989+
to back them.
990+
991+
With fewer signals than that, return Passed. Do not guess: a blocker
992+
on a human contributor's PR costs more than a missed one, which the
993+
maintainers can still catch in review.
994+
944995
- name: "AI Contribution Disclosure"
945996
mode: warning
946997
instructions: |
947998
coreruleset's AI-CONTRIBUTIONS.md policy requires every PR where AI
948-
tools materially assisted to disclose it in the PR body.
949-
950-
Flag ⚠️ WARNING: <description> — <fix> when the PR body has no
951-
`## ai disclosure` section but the diff shows signals of AI
952-
assistance: generated-looking docstrings or comments at uniform
953-
density, a large mechanically-uniform test batch, boilerplate
954-
phrasing in the description, or an unusually large diff with no
955-
incremental commit history.
956-
957-
When the section is present, flag it as inadequate if any field is
958-
generic. The template requires three fields, each concrete:
959-
`**tools used**` (model name and version), `**assisted with**` (what
960-
was actually generated — "initial regex pattern for SQLi
961-
detection", "test case scaffolding", "regeneration via
962-
crs-toolchain"), and `**review performed**` (concrete verification —
999+
tools materially assisted to disclose it in the PR body. A missing
1000+
section is the job of the "AI Use Without Disclosure" check; this
1001+
one reviews the section's content and the PR body's format.
1002+
1003+
When the section says AI was Used, flag it as inadequate if any
1004+
field is generic. Accept either the PR template's table (Tool(s)
1005+
used / What was AI-assisted / Review performed) or the bullet form
1006+
(`**tools used**` / `**assisted with**` / `**review performed**`).
1007+
Each of the three fields must be concrete: the tool (model name and
1008+
version), what was actually generated ("initial regex pattern for
1009+
SQLi detection", "test case scaffolding", "regeneration via
1010+
crs-toolchain"), and the review performed (concrete verification —
9631011
"ran the full go-ftw suite", "probed the payload at PL4 and mapped
9641012
firing rule IDs", "checked the pattern for RE2 compatibility").
9651013
"assisted with rule writing" or "reviewed the code" is not a
@@ -972,7 +1020,7 @@ reviews:
9721020
does not use attribution trailers.
9731021
9741022
Do not flag a trivial PR (a typo fix, a version bump, a
975-
Renovate/Dependabot automated update) for a missing disclosure.
1023+
Renovate/Dependabot automated update) for missing sections.
9761024
9771025
# Generic application-security review for the tooling repos (go-ftw,
9781026
# crs-toolchain, albedo, crs-linter) and for anything CRS ships that

‎AGENTS.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,8 @@ constraint on the file.
5353
for cross-file invariants and policy a single line can't express. The CRS-specific ones:
5454
regex-assembly source-of-truth (`.conf` `@rx` edited without its `.ra`), go-ftw test coverage,
5555
ReDoS/RE2 compatibility, false-positive risk & existing coverage, rule metadata & ID
56-
conventions, rule/config breaking changes, AI contribution disclosure. Plus the generic
56+
conventions, rule/config breaking changes, AI use without disclosure (the one `mode: error`
57+
check), AI contribution disclosure (adequacy and PR body format). Plus the generic
5758
supply-chain and appsec checks.
5859

5960
The domain knowledge in these strings is derived from `../coreruleset/CLAUDE.md` (rule ID
@@ -63,11 +64,12 @@ PR conventions). Keep the two in sync when either changes.
6364
## Conventions for checks
6465

6566
- Finding format is fixed file-wide: `⚠️ WARNING: <description> — <fix>`, with `❌ blocker`
66-
reserved for RE2-incompatible regexes, plugin rules in a core ID range, and hand-edited
67-
compiled regex that pre-commit.ci will revert.
68-
- Every check is `mode: warning`. Escalating to `mode: error` requires also setting
69-
`request_changes_workflow: true`, and should follow a validated low false-positive rate on
70-
real CRS PRs.
67+
reserved for RE2-incompatible regexes, plugin rules in a core ID range, hand-edited
68+
compiled regex that pre-commit.ci will revert, and AI assistance without a disclosure.
69+
- Every check is `mode: warning`, except "AI Use Without Disclosure". An error-mode check
70+
blocks merge only while `request_changes_workflow: true` (the deprecated alias of
71+
`auto_approvals.enabled`) is set. Escalate another check only after a validated low
72+
false-positive rate on real CRS PRs.
7173
- Ecosystem- and rule-scoped checks open with a **trigger condition** that returns `Passed` +
7274
"not applicable" when the diff touches nothing relevant — this avoids `Inconclusive` noise.
7375
Match that pattern in new checks.

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ Nine glob-scoped entries. These carry the domain knowledge a generic reviewer do
103103

104104
### Pre-merge checks — whole-PR, cross-file
105105

106-
Thirteen checks, all `mode: warning`. These evaluate the diff as a unit and catch invariants no single line can express.
106+
Fourteen checks, all `mode: warning` except AI Use Without Disclosure (`mode: error`). These evaluate the diff as a unit and catch invariants no single line can express.
107107

108108
**CRS-specific:**
109109

@@ -115,7 +115,8 @@ Thirteen checks, all `mode: warning`. These evaluate the diff as a unit and catc
115115
| False Positive Risk & Existing Coverage | a new pattern with no proof the payload is undetected — the PL4 probe, the sibling baseline, the subset argument, and the libinjection caveat |
116116
| CRS Rule Metadata & ID Conventions | what crs-linter fails on: ID collisions and range violations, unapproved tags, stale `ver:`, ordering |
117117
| Rule & Config Breaking Changes | removed or renumbered rule IDs, changed defaults, removed tags, tooling API changes — plus the documentation counterpart |
118-
| AI Contribution Disclosure | a missing or generic `## ai disclosure`, missing `## what`/`## why`/`## refs`, and attribution trailers this project does not use |
118+
| AI Use Without Disclosure | **blocker, `mode: error`**: signs of AI assistance (a named AI tool or agent, an AI trailer, or two indirect signals) with no `## ai disclosure` section |
119+
| AI Contribution Disclosure | a generic `## ai disclosure`, missing `## what`/`## why`/`## refs`, and attribution trailers this project does not use |
119120

120121
**Supply chain and appsec:**
121122

@@ -171,8 +172,8 @@ Deliberately excluded: `template-plugin` and `modsecurity-crs-docker` — they c
171172

172173
Follow these when adding to `path_instructions` or `custom_checks`; the file is consistent about them and reviews read worse when it isn't.
173174

174-
- **Finding format is fixed:** `⚠️ WARNING: <description> — <fix>`. `❌ blocker` is reserved for RE2-incompatible regexes, plugin rules in a core ID range, and hand-edited compiled regex that pre-commit.ci will revert.
175-
- **Every check is `mode: warning`.** Escalating one to `mode: error` also requires `request_changes_workflow: true`, and should follow a validated low false-positive rate on real CRS PRs.
175+
- **Finding format is fixed:** `⚠️ WARNING: <description> — <fix>`. `❌ blocker` is reserved for RE2-incompatible regexes, plugin rules in a core ID range, hand-edited compiled regex that pre-commit.ci will revert, and AI assistance without a disclosure.
176+
- **Every check is `mode: warning`**, except AI Use Without Disclosure. An error-mode check blocks merge only while `request_changes_workflow: true` (auto approvals) is set. Escalate another one only after a validated low false-positive rate on real CRS PRs.
176177
- **Open ecosystem- and rule-scoped checks with a trigger condition** that returns `Passed` plus "not applicable" when the diff touches nothing relevant. Without it the check returns `Inconclusive` on every unrelated PR.
177178
- **Describe acceptable patterns alongside violations**, so the reviewer has a target state rather than only a complaint.
178179
- **Where does it go?** Use `path_instructions` when the finding needs to land on a specific line. Use `custom_checks` for cross-file invariants and policy a single line can't express.

0 commit comments

Comments
 (0)