Skip to content

Commit 6b973a3

Browse files
committed
test: add real-world regression corpus
1 parent 479b163 commit 6b973a3

15 files changed

Lines changed: 634 additions & 9 deletions

KNOWN_LIMITATIONS.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,9 @@ The rule-contract conformance suite uses author-maintained planted examples. Its
2828
answer whether each named built-in rule recognizes its declared positive example in the expected
2929
evidence state and stays quiet on its declared safe neighbour. They do not measure whole-project
3030
vulnerability precision or recall, production reachability, exploitability, language coverage or
31-
the agent-guided methodology. Historical real-world regressions and ordinary-project review remain
32-
separate evidence sets.
31+
the agent-guided methodology. The [historical real-world regression corpus](docs/regressions/v0.5.4-real-world-regressions.md)
32+
guards named minimized failures, while ordinary-project review remains a separate evidence set;
33+
neither is a representative accuracy benchmark.
3334

3435
## MCP and rule expansion
3536

README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,8 +97,9 @@ evidence-integrity rules and 8 opt-in external-adapter rules, with built-in dept
9797
JavaScript/TypeScript and Python Web code.
9898

9999
Exact support and limits are in the [compatibility matrix](docs/compatibility.md), [stable rule
100-
corpus](docs/stable-rule-corpus.json), [rule-contract conformance](docs/conformance/v0.5.4-rule-contract-conformance.md)
101-
and [ordinary-project review](docs/case-studies/journeys/v0.5.0-review.md). MCP and additional stable
100+
corpus](docs/stable-rule-corpus.json), [rule-contract conformance](docs/conformance/v0.5.4-rule-contract-conformance.md),
101+
[historical real-world regressions](docs/regressions/v0.5.4-real-world-regressions.md) and
102+
[ordinary-project review](docs/case-studies/journeys/v0.5.0-review.md). MCP and additional stable
102103
rules are deferred behind the [documented architecture gates](docs/architecture/mcp-and-rule-expansion.md).
103104
The signed v0.5.3 GitHub assets and provenance, verified installer, public npm package and signed
104105
`v1` Action alias have passed their public checks. The installer below defaults to v0.5.3.

README.zh-CN.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,8 @@ v0.5.0 的解释合同继续保留:每个 v3 源码 finding 同时给出行业
9090
规则,内置深度集中在 JavaScript/TypeScript 与 Python Web 代码。
9191

9292
准确支持范围见[兼容矩阵](docs/compatibility.md)[稳定规则语料](docs/stable-rule-corpus.json)
93-
[规则合同一致性结果](docs/conformance/v0.5.4-rule-contract-conformance.md)
93+
[规则合同一致性结果](docs/conformance/v0.5.4-rule-contract-conformance.md)
94+
[历史真实回归语料](docs/regressions/v0.5.4-real-world-regressions.md)
9495
[普通项目复核](docs/case-studies/journeys/v0.5.0-review.md)。MCP 与新增 stable 规则需要先满足
9596
[架构决策中的门槛](docs/architecture/mcp-and-rule-expansion.md)。v0.5.3 的签名 GitHub 资产与
9697
provenance、可信安装器、公开 npm 包和签名 `v1` Action 别名都已通过各自的公网检查;下面的

docs/V0.5.4_ENGINEERING_PLAN.md

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ opens a lower-friction support channel and submits the one currently eligible cu
3838
|---|---|---|---|
3939
| G0 | Scope, baseline and claim boundary | Plan records current v0.5.3 state, non-goals and release approval point | completed |
4040
| G1 | Rule-contract conformance rename | Runtime, tests, generated JSON/Markdown, npm files and capability surfaces use the new name; no active public surface presents planted fixtures as ground truth | completed |
41-
| G2 | Real-world regression corpus | One deterministic command executes structured cases for report rendering, pnpm workspace inheritance, nested templates, moved-condition retest and benign numeric SVG review; generated JSON/Markdown records provenance and limits | in progress |
42-
| G3 | First-trial and v0.5.4 candidate surfaces | Main-branch first trial is unpinned; CI/release/trusted install examples stay pinned; candidate version surfaces agree on 0.5.4 while published release state remains 0.5.3 | pending |
41+
| G2 | Real-world regression corpus | One deterministic command executes structured cases for report rendering, pnpm workspace inheritance, nested templates, moved-condition retest and benign numeric SVG review; generated JSON/Markdown records provenance and limits | completed |
42+
| G3 | First-trial and v0.5.4 candidate surfaces | Main-branch first trial is unpinned; CI/release/trusted install examples stay pinned; candidate version surfaces agree on 0.5.4 while published release state remains 0.5.3 | in progress |
4343
| G4 | Focused and final verification | New focused tests pass, generated artifacts are byte-current, one final `npm run check` and Skill Creator validation pass | pending |
4444
| G5 | Authorized external actions | Discussions is enabled and verified; awesome-devsecops rules are rechecked and a factual v0.5.3 PR is submitted if still eligible | pending |
4545
| G6 | Release approval checkpoint | Candidate commits and external evidence are pushed; exact release operations and residual risks are presented to the author | pending |
@@ -138,6 +138,19 @@ Focused gates:
138138
- The benign-match case records `expected_benign_match` and the required human closure instead of
139139
changing detector behavior.
140140

141+
Completion record (2026-08-16):
142+
143+
- Added one deterministic runner and generated JSON/Markdown for four minimized v0.5.2 failures
144+
plus one numeric-SVG DOM-sink review case derived from external real-project review noise.
145+
- Each guard calls production modules: source audit/workspace resolution, v3 Markdown/HTML
146+
renderers, JS/TS inspection or v3 baseline comparison. Historical descriptions and fix provenance
147+
are generated from `docs/adoption/regressions.json`.
148+
- The numeric SVG case remains `suspected`, is manually classified `expected_benign_match`, requires
149+
numeric-input review and records `suppressionApplied: false`.
150+
- Focused verification: 5 cases passed, 0 failed; validator mutations prove a failed guard and an
151+
improper suppression are rejected; product/distribution contracts and both generated-evidence
152+
`--check` commands pass.
153+
141154
## G3 - First trial and candidate surfaces
142155

143156
Implementation:

docs/capabilities.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,14 @@
123123
"scope": "Regenerates exact JSON and Markdown pass/fail evidence for 20 built-in risk and two evidence-integrity planted positive/negative rule contracts. It is author-maintained conformance evidence, not a production-vulnerability precision, recall, reachability, exploitability or security-coverage measurement.",
124124
"evidence": ["scripts/run-rule-contract-conformance.mjs", "scripts/lib/rule-contract-conformance.mjs", "test/rule-contract-conformance.test.mjs", "docs/conformance/v0.5.4-rule-contract-conformance.json"]
125125
},
126+
{
127+
"id": "real-world-regression-corpus",
128+
"name": "Historical real-world regression corpus",
129+
"category": "evidence_reporting",
130+
"maturity": "stable",
131+
"scope": "Re-executes four minimized historical correctness failures and one review-visible benign DOM-sink match against product code. It guards named regressions and does not provide representative production-vulnerability accuracy.",
132+
"evidence": ["scripts/run-real-world-regressions.mjs", "scripts/lib/real-world-regressions.mjs", "test/real-world-regressions.test.mjs", "docs/regressions/v0.5.4-real-world-regressions.json"]
133+
},
126134
{
127135
"id": "structured-reports",
128136
"name": "Stable multi-format finding reports",

docs/capabilities.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ context and judgment, and `planned` behavior is unavailable.
3232
| Automatic project discovery and scoped run | `stable` | Detects supported Node/Python and split-stack projects, package managers, lockfiles and config/deployment paths, then creates a versioned network-free security scope. It never establishes deployment ownership. | [`scripts/project-start.mjs`](../scripts/project-start.mjs), [`scripts/lib/project-discovery.mjs`](../scripts/lib/project-discovery.mjs), [`test/project-discovery.test.mjs`](../test/project-discovery.test.mjs), [`docs/security-scope.schema.json`](../docs/security-scope.schema.json) |
3333
| Git diff source-audit selection | `stable` | Reduces built-in source-audit review noise with added-line --since selection or an isolated --staged Git index snapshot. It records the immutable base and selection counts, preserves relevant incomplete evidence, excludes external adapters and cannot establish whole-repository safety or baseline/retest state. | [`scripts/lib/git-diff-scope.mjs`](../scripts/lib/git-diff-scope.mjs), [`scripts/project-audit.mjs`](../scripts/project-audit.mjs), [`test/incremental-source-audit.test.mjs`](../test/incremental-source-audit.test.mjs) |
3434
| Reproducible rule-contract conformance | `stable` | Regenerates exact JSON and Markdown pass/fail evidence for 20 built-in risk and two evidence-integrity planted positive/negative rule contracts. It is author-maintained conformance evidence, not a production-vulnerability precision, recall, reachability, exploitability or security-coverage measurement. | [`scripts/run-rule-contract-conformance.mjs`](../scripts/run-rule-contract-conformance.mjs), [`scripts/lib/rule-contract-conformance.mjs`](../scripts/lib/rule-contract-conformance.mjs), [`test/rule-contract-conformance.test.mjs`](../test/rule-contract-conformance.test.mjs), [`docs/conformance/v0.5.4-rule-contract-conformance.json`](../docs/conformance/v0.5.4-rule-contract-conformance.json) |
35+
| Historical real-world regression corpus | `stable` | Re-executes four minimized historical correctness failures and one review-visible benign DOM-sink match against product code. It guards named regressions and does not provide representative production-vulnerability accuracy. | [`scripts/run-real-world-regressions.mjs`](../scripts/run-real-world-regressions.mjs), [`scripts/lib/real-world-regressions.mjs`](../scripts/lib/real-world-regressions.mjs), [`test/real-world-regressions.test.mjs`](../test/real-world-regressions.test.mjs), [`docs/regressions/v0.5.4-real-world-regressions.json`](../docs/regressions/v0.5.4-real-world-regressions.json) |
3536
| Stable multi-format finding reports | `stable` | Renders source v3 and crawl, demo, crawler-identity, edge and AWS v2 conclusions as JSON, Markdown, escaped HTML, SARIF 2.1.0 and JUnit while preserving domain, evidence state, baseline state and fingerprints. Legacy v1 remains historical input only. | [`docs/finding-v3.schema.json`](../docs/finding-v3.schema.json), [`docs/report-v3.schema.json`](../docs/report-v3.schema.json), [`docs/finding-v2.schema.json`](../docs/finding-v2.schema.json), [`docs/report-v2.schema.json`](../docs/report-v2.schema.json), [`scripts/lib/evidence-v3.mjs`](../scripts/lib/evidence-v3.mjs), [`scripts/lib/evidence-v2.mjs`](../scripts/lib/evidence-v2.mjs), [`test/report-v3-contract.test.mjs`](../test/report-v3-contract.test.mjs), [`test/evidence-loop.test.mjs`](../test/evidence-loop.test.mjs) |
3637
| Patch and baseline retest evidence loop | `stable` | Writes patch-only proposals, explains source findings and compares persisted v2 or v3 baselines with subject/scope/rule identity, per-rule coverage and honest new/fixed/unchanged/regressed/unretested/not_comparable states. | [`scripts/project-audit.mjs`](../scripts/project-audit.mjs), [`scripts/lib/evidence-v3.mjs`](../scripts/lib/evidence-v3.mjs), [`scripts/lib/evidence-v2.mjs`](../scripts/lib/evidence-v2.mjs), [`scripts/explain-finding.mjs`](../scripts/explain-finding.mjs), [`test/evidence-loop.test.mjs`](../test/evidence-loop.test.mjs), [`test/baseline-identity.test.mjs`](../test/baseline-identity.test.mjs) |
3738
| Professional and plain-language finding explanation | `stable` | Source report v3 gives every finding a technical term, ordinary-language meaning, realistic consequence, evidence boundary, reviewable proposal, alternatives, side effects, required decisions, separate security and functional retests, and rollback criteria. Default display omits raw evidence; the technical view is explicit. | [`scripts/lib/evidence-v3.mjs`](../scripts/lib/evidence-v3.mjs), [`scripts/explain-finding.mjs`](../scripts/explain-finding.mjs), [`test/report-v3-contract.test.mjs`](../test/report-v3-contract.test.mjs), [`test/evidence-loop.test.mjs`](../test/evidence-loop.test.mjs) |

docs/public-contract.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
"The CLI proposes and records repairs but does not edit project files.",
3030
"Diff-scoped reports reduce review noise and do not establish whole-repository safety.",
3131
"The planted rule-contract conformance suite is not production-vulnerability precision or recall.",
32+
"The historical real-world regression corpus guards named failures and is not a representative accuracy benchmark.",
3233
"No authenticated DAST, automatic BOLA/IDOR proof or unattended production patching is claimed."
3334
]
3435
}
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
{
2+
"schemaVersion": 1,
3+
"release": "v0.5.4",
4+
"evidenceType": "historical_real_world_regression_corpus",
5+
"limitation": "Minimized cases derived from observed project failures and review noise; they prove only that the named historical conditions remain guarded. They are not a representative vulnerability benchmark or a production precision/recall measurement.",
6+
"sourceInventory": "docs/adoption/regressions.json",
7+
"summary": {
8+
"cases": 5,
9+
"passed": 5,
10+
"failed": 0,
11+
"resolvedRegressions": 4,
12+
"expectedBenignMatches": 1
13+
},
14+
"cases": [
15+
{
16+
"id": "v3-summary-object-coercion",
17+
"title": "The main risk summary rendered JavaScript objects",
18+
"classification": "resolved_regression",
19+
"source": {
20+
"kind": "historical_regression",
21+
"reference": "docs/adoption/regressions.json#v3-summary-object-coercion",
22+
"fixedVersion": "0.5.2",
23+
"fixCommit": "a0cb518da23043e549b84f6f4c9cc39726d7c602"
24+
},
25+
"observedFailure": "A v3 report with risk-domain state totals rendered [object Object] in Markdown and HTML.",
26+
"protectedContract": "Markdown and HTML risk summaries render numeric state and severity totals.",
27+
"status": "passed",
28+
"evidence": {
29+
"findingCount": 1,
30+
"markdownNumericSummary": true,
31+
"htmlNumericSummary": true,
32+
"objectCoercionObserved": false
33+
},
34+
"remainingBoundary": "Golden rendering tests protect the current fixture shapes; they do not prove every future presentation is understandable."
35+
},
36+
{
37+
"id": "pnpm-workspace-lockfile-evidence",
38+
"title": "A pnpm workspace could receive a false confirmed lockfile absence",
39+
"classification": "resolved_regression",
40+
"source": {
41+
"kind": "historical_regression",
42+
"reference": "docs/adoption/regressions.json#pnpm-workspace-lockfile-evidence",
43+
"fixedVersion": "0.5.2",
44+
"fixCommit": "a0cb518da23043e549b84f6f4c9cc39726d7c602"
45+
},
46+
"observedFailure": "A root pnpm-lock.yaml and pnpm-workspace.yaml covering apps/* still allowed a child package.json to be reported as confirmed missing-lockfile evidence.",
47+
"protectedContract": "A covered pnpm package inherits its workspace lockfile; only proven uncovered manifests may be confirmed.",
48+
"status": "passed",
49+
"evidence": {
50+
"coveredPackage": "apps/web/package.json",
51+
"coveredPackageFindingCount": 0,
52+
"outsidePackage": "tools/outside/package.json",
53+
"outsidePackageState": "confirmed"
54+
},
55+
"remainingBoundary": "The parser supports the documented bounded subset and is not a general YAML evaluator."
56+
},
57+
{
58+
"id": "nested-template-coverage",
59+
"title": "Nested JS/TS templates could make a whole file partial",
60+
"classification": "resolved_regression",
61+
"source": {
62+
"kind": "historical_regression",
63+
"reference": "docs/adoption/regressions.json#nested-template-coverage",
64+
"fixedVersion": "0.5.2",
65+
"fixCommit": "a0cb518da23043e549b84f6f4c9cc39726d7c602"
66+
},
67+
"observedFailure": "A nested template such as const html = `${`</b>`}</div>` produced unterminated_string_literal and stopped normal source-rule coverage for that file.",
68+
"protectedContract": "Nested template literals do not silently remove normal JS/TS source-rule coverage.",
69+
"status": "passed",
70+
"evidence": {
71+
"sourcePath": "src/server.ts",
72+
"browserSinkCoverage": "completed",
73+
"incompleteEvidenceFindingCount": 0
74+
},
75+
"remainingBoundary": "The tokenizer is still bounded and reports unsupported syntax as incomplete evidence rather than silently passing it."
76+
},
77+
{
78+
"id": "path-rename-retest",
79+
"title": "A path-only rename could make a retest exit successfully",
80+
"classification": "resolved_regression",
81+
"source": {
82+
"kind": "historical_regression",
83+
"reference": "docs/adoption/regressions.json#path-rename-retest",
84+
"fixedVersion": "0.5.2",
85+
"fixCommit": "a0cb518da23043e549b84f6f4c9cc39726d7c602"
86+
},
87+
"observedFailure": "Renaming a file without removing its condition produced one fixed and one new finding, allowing the compatible baseline gate to exit successfully.",
88+
"protectedContract": "A unique path-equivalent condition remains unresolved instead of becoming fixed plus new.",
89+
"status": "passed",
90+
"evidence": {
91+
"priorPath": "src/old.js",
92+
"currentPath": "src/renamed.js",
93+
"baselineState": "unchanged",
94+
"reasonCode": "condition_moved",
95+
"fixedCount": 0,
96+
"newCount": 0
97+
},
98+
"remainingBoundary": "condition_moved means an equivalent condition exists elsewhere; it does not prove historical identity or that a git move caused it."
99+
},
100+
{
101+
"id": "numeric-svg-innerhtml-review",
102+
"title": "Numeric SVG innerHTML remains a review-visible expected match",
103+
"classification": "expected_benign_match",
104+
"source": {
105+
"kind": "external_review_minimized",
106+
"reference": "KNOWN_LIMITATIONS.md#recurring-expected-matches",
107+
"note": "Minimized from an external real-project review; original project content is not included."
108+
},
109+
"observedFailure": "A numeric-only SVG string assembled with innerHTML matched the intentionally broad DOM-sink rule.",
110+
"protectedContract": "Context-dependent DOM sinks stay visible for human closure and are not silently suppressed.",
111+
"status": "passed",
112+
"evidence": {
113+
"sourcePath": "src/charts.ts",
114+
"ruleId": "browser-html-injection-sink",
115+
"evidenceState": "suspected",
116+
"manualClassification": "expected_benign_match",
117+
"requiredClosure": "Confirm that every interpolated value remains numeric and no untrusted markup reaches the sink.",
118+
"suppressionApplied": false
119+
},
120+
"remainingBoundary": "This fixture is benign only because its constructed values are numeric. The detector does not trace arbitrary input or prove other innerHTML assignments safe."
121+
}
122+
]
123+
}

0 commit comments

Comments
 (0)