Skip to content

Commit e8e5052

Browse files
bx33661bx33661
committed
Strengthen omv-find scoring and add golden evals for radar scenarios
Adds diff-alert, duplicate-risk, and portfolio-radar golden outputs. Expands eval scenarios for intelligence lifecycle coverage. Updates omv-critic, omv-dedup, and omv-disclose contract references.
1 parent f25ce03 commit e8e5052

16 files changed

Lines changed: 724 additions & 8 deletions

File tree

‎skills/omv-critic/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ Challenge the finding before report submission.
1616

1717
## Workflow
1818

19-
1. Read `.omv/findings/<id>.yaml`.
20-
2. If present, read `.omv/threatmaps/<id>.yaml`.
19+
1. Read `.omv/findings/<id>.yaml` using `contracts/evidence.v1.yaml` as the local schema reference.
20+
2. If present, read `.omv/threatmaps/<id>.yaml` using `contracts/threat-map.v1.yaml` as the local schema reference.
2121
3. Run or inspect `omv findings validate <id>` for deterministic structure status.
2222
4. Evaluate argument quality separately from structural validation.
2323
5. Output `reject_risk: low|medium|high`.

‎skills/omv-critic/contracts/evidence.v1.yaml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Evidence.v1 — Finding object contract
2+
# Produced by: omv-find (on confirmed/explicit handoff request)
3+
# Consumed by: omv-report
4+
#
5+
# Status values:
6+
# candidate — promising but not yet proven; omv-report produces triage draft only
7+
# confirmed — tested version + source->sink->guard + local reproducer known
8+
# blocked — missing required evidence; omv-report must explain blockers, not write a submission-ready report
9+
#
10+
# Rules:
11+
# - Use "unknown" for unverified fields; never fill from memory.
12+
# - List unverified fields under provenance.unverified_fields.
13+
# - omv-report must preserve all unverified markers; never silently upgrade them.
14+
# - CVE readiness score 0-100 based on field completeness; threshold >=75 is necessary but not sufficient.
15+
# - CLI validation is the machine gate for status transitions; confirmed findings must satisfy field-level checks.
16+
# - Unknown values in important fields must be listed in provenance.unverified_fields until verified.
17+
18+
schema_version: "1"
19+
20+
# ── Identity ──────────────────────────────────────────────────────────────────
21+
handoff_version: "1.0"
22+
status: candidate # candidate | confirmed | blocked
23+
researcher_goal: VulDB # VulDB | CVE | GHSA | OSV | advisory | triage
24+
25+
package:
26+
ecosystem: npm # npm | python | go | rust | java | ruby | php | csharp | swift | dart | elixir | perl | r | lua
27+
registry_name: "" # exact package name in registry
28+
repository_url: "" # verified GitHub URL
29+
vendor: "" # project name / org — NOT the registry name
30+
product: "" # package or product name
31+
32+
# ── Versions ──────────────────────────────────────────────────────────────────
33+
versions:
34+
tested: "" # exact version tested (required for confirmed/submission-ready)
35+
affected_range: unknown # e.g. "up to and including 2.4.1" | "before 2.4.2"
36+
fixed: unknown # first safe version, or "none"
37+
38+
# ── Vulnerability ─────────────────────────────────────────────────────────────
39+
vulnerability:
40+
class: "" # plain English class name (not CWE number)
41+
cwe: "" # e.g. CWE-22
42+
affected_component: "" # file path
43+
affected_function: "" # function name
44+
45+
# ── Evidence ──────────────────────────────────────────────────────────────────
46+
evidence:
47+
source: unknown # attacker-controlled input entry point; confirmed findings include file:line
48+
sink: unknown # dangerous operation reached; confirmed findings include file:line
49+
guard: unknown # check that is missing/bypassable; include file:line or explicit absence
50+
reproducer: unknown # local PoC command or description; "none" if not yet written
51+
observed_result: unknown # user-reported local observation; do not infer or fabricate
52+
repro_artifacts: [] # optional .omv/repro/<id>/ files: commands.sh, observed.txt, screenshots/, docker-compose.yml
53+
54+
# ── CVSS ──────────────────────────────────────────────────────────────────────
55+
cvss:
56+
vector: unknown # e.g. CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
57+
score: unknown # numeric score
58+
severity: unknown # Critical | High | Medium | Low
59+
60+
# ── Impact ────────────────────────────────────────────────────────────────────
61+
impact:
62+
attack_vector: unknown # Network | Local | Physical | Adjacent
63+
authentication_required: unknown # true | false | unknown
64+
user_interaction_required: unknown # true | false | unknown
65+
scope_changed: unknown # true | false | unknown
66+
confidentiality: unknown # High | Low | None | unknown
67+
integrity: unknown # High | Low | None | unknown
68+
availability: unknown # High | Low | None | unknown
69+
70+
# ── Dedup ─────────────────────────────────────────────────────────────────────
71+
dedup:
72+
nvd_searched: false
73+
ghsa_searched: false
74+
ecosystem_db_searched: false
75+
existing_cve: unknown # CVE-YYYY-NNNNN if found, "none" if clean, "unknown" if not searched
76+
notes: ""
77+
78+
# ── Disclosure ────────────────────────────────────────────────────────────────
79+
disclosure:
80+
vendor_contacted: false
81+
contact_date: unknown
82+
vendor_response: unknown
83+
planned_disclosure_date: unknown
84+
85+
# ── Blockers ──────────────────────────────────────────────────────────────────
86+
blockers: []
87+
# Common values:
88+
# - "no tested version"
89+
# - "source to sink not proven"
90+
# - "likely duplicate CVE"
91+
# - "developer-controlled trigger only"
92+
# - "no local reproducer"
93+
94+
# ── Audit Verdict ─────────────────────────────────────────────────────────────
95+
verdict:
96+
exploitability: unknown # proven | plausible | blocked | disproven | unknown
97+
confidence: unknown # high | medium | low | unknown
98+
reason: "" # concise reason for the current verdict
99+
100+
# ── Provenance ────────────────────────────────────────────────────────────────
101+
provenance:
102+
verification_date: ""
103+
researcher: ""
104+
unverified_fields: [] # list field paths that carry "unknown" or unconfirmed data; checked by CLI validation
105+
tool_versions: {} # e.g. {omv-find: "1.0"}
106+
107+
# ── Evidence and Submission Scores (computed, not stored) ─────────────────────
108+
# omv computes evidence_score from field completeness.
109+
# omv computes submission_score by penalizing unresolved blockers, unknown
110+
# observed_result, unknown affected range, incomplete dedup, and blocked/disproven
111+
# verdicts. A high evidence_score is not enough for submission-ready output.
112+
#
113+
# Evidence scoring guide (total 100):
114+
# tested version present +20
115+
# source identified +10
116+
# sink identified +10
117+
# guard missing confirmed +10
118+
# local reproducer written +15
119+
# observed result documented +10
120+
# cvss vector present +10
121+
# dedup search completed +10
122+
# vendor contacted +5
123+
#
124+
# Threshold: submission_score >= 75 plus validation OK is required before
125+
# suggesting promotion to confirmed.
126+
cve_readiness_scoring:
127+
tested_version: 20
128+
source_identified: 10
129+
sink_identified: 10
130+
guard_missing_confirmed: 10
131+
local_reproducer: 15
132+
observed_result: 10
133+
cvss_vector: 10
134+
dedup_searched: 10
135+
vendor_contacted: 5
136+
total: 100
137+
threshold: 75

‎skills/omv-critic/contracts/threat-map.v1.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# ThreatMap.v1 — optional dataflow threat map sidecar
2+
# Produced by: omv-audit
3+
# Consumed by: omv findings show, omv-critic, omv-report
4+
#
5+
# The full threat map for a single finding lives in a sidecar file:
6+
# .omv/threatmaps/<id>.yaml
7+
# The parent `.omv/findings/<id>.yaml` Evidence.v1 file remains valid without
8+
# this sidecar. Evidence.v1 source/sink/guard fields stay as compatibility
9+
# summaries; this graph is used when a richer source -> transform -> sink path
10+
# is useful for audit review or report criticism.
11+
12+
schema_version: "1"
13+
14+
# ── Identity ──────────────────────────────────────────────────────────────────
15+
finding_id: "" # id matching `.omv/findings/<id>.yaml`
16+
package:
17+
ecosystem: ""
18+
registry_name: ""
19+
repository_url: ""
20+
version_analyzed: ""
21+
22+
# ── Dataflow Paths ────────────────────────────────────────────────────────────
23+
# One entry per discovered source-to-sink path.
24+
paths: []
25+
# Each path entry:
26+
# - id: 1
27+
# source:
28+
# type: user_input | file | network | env | config
29+
# location: "src/routes/upload.js:42"
30+
# description: "req.body.filename passed directly"
31+
# transforms:
32+
# - type: parse | decode | normalize | validate | authorize | other
33+
# location: "src/routes/upload.js:45"
34+
# description: "filename is joined without canonical path check"
35+
# sink:
36+
# type: fs_write | exec | eval | html_render | sql | network_req
37+
# location: "src/utils/file.js:88"
38+
# description: "path.join(baseDir, filename) without normalization"
39+
# guard:
40+
# present: false
41+
# description: "no path traversal check before join"
42+
# bypassable: true
43+
# confidence: high # high | medium | low
44+
# notes: ""
45+
46+
# ── Summary ───────────────────────────────────────────────────────────────────
47+
# Digest fields written into the parent Evidence.v1 object.
48+
summary:
49+
path_count: 0
50+
confirmed_paths: 0
51+
highest_confidence: unknown # high | medium | low | unknown
52+
vuln_classes: [] # list of vulnerability class names identified
53+
notes: ""
54+
55+
# ── Provenance ────────────────────────────────────────────────────────────────
56+
provenance:
57+
analysis_date: ""
58+
tool: manual # manual | semgrep | codeql | other
59+
tool_version: unknown
60+
analyst: ""

‎skills/omv-dedup/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Assess duplicate disclosure risk for a local finding.
1616

1717
## Workflow
1818

19-
1. Read `.omv/findings/<id>.yaml`.
19+
1. Read `.omv/findings/<id>.yaml` using `contracts/evidence.v1.yaml` as the local schema reference.
2020
2. Run `omv dedup <id>` to produce deterministic NVD, GHSA, OSV, and ecosystem advisory queries.
2121
3. Show the queries before conclusions.
2222
4. Compare package, ecosystem, affected range, vulnerability class, CWE, source, sink, guard, and fixed version signals.

‎skills/omv-dedup/contracts/evidence.v1.yaml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Evidence.v1 — Finding object contract
2+
# Produced by: omv-find (on confirmed/explicit handoff request)
3+
# Consumed by: omv-report
4+
#
5+
# Status values:
6+
# candidate — promising but not yet proven; omv-report produces triage draft only
7+
# confirmed — tested version + source->sink->guard + local reproducer known
8+
# blocked — missing required evidence; omv-report must explain blockers, not write a submission-ready report
9+
#
10+
# Rules:
11+
# - Use "unknown" for unverified fields; never fill from memory.
12+
# - List unverified fields under provenance.unverified_fields.
13+
# - omv-report must preserve all unverified markers; never silently upgrade them.
14+
# - CVE readiness score 0-100 based on field completeness; threshold >=75 is necessary but not sufficient.
15+
# - CLI validation is the machine gate for status transitions; confirmed findings must satisfy field-level checks.
16+
# - Unknown values in important fields must be listed in provenance.unverified_fields until verified.
17+
18+
schema_version: "1"
19+
20+
# ── Identity ──────────────────────────────────────────────────────────────────
21+
handoff_version: "1.0"
22+
status: candidate # candidate | confirmed | blocked
23+
researcher_goal: VulDB # VulDB | CVE | GHSA | OSV | advisory | triage
24+
25+
package:
26+
ecosystem: npm # npm | python | go | rust | java | ruby | php | csharp | swift | dart | elixir | perl | r | lua
27+
registry_name: "" # exact package name in registry
28+
repository_url: "" # verified GitHub URL
29+
vendor: "" # project name / org — NOT the registry name
30+
product: "" # package or product name
31+
32+
# ── Versions ──────────────────────────────────────────────────────────────────
33+
versions:
34+
tested: "" # exact version tested (required for confirmed/submission-ready)
35+
affected_range: unknown # e.g. "up to and including 2.4.1" | "before 2.4.2"
36+
fixed: unknown # first safe version, or "none"
37+
38+
# ── Vulnerability ─────────────────────────────────────────────────────────────
39+
vulnerability:
40+
class: "" # plain English class name (not CWE number)
41+
cwe: "" # e.g. CWE-22
42+
affected_component: "" # file path
43+
affected_function: "" # function name
44+
45+
# ── Evidence ──────────────────────────────────────────────────────────────────
46+
evidence:
47+
source: unknown # attacker-controlled input entry point; confirmed findings include file:line
48+
sink: unknown # dangerous operation reached; confirmed findings include file:line
49+
guard: unknown # check that is missing/bypassable; include file:line or explicit absence
50+
reproducer: unknown # local PoC command or description; "none" if not yet written
51+
observed_result: unknown # user-reported local observation; do not infer or fabricate
52+
repro_artifacts: [] # optional .omv/repro/<id>/ files: commands.sh, observed.txt, screenshots/, docker-compose.yml
53+
54+
# ── CVSS ──────────────────────────────────────────────────────────────────────
55+
cvss:
56+
vector: unknown # e.g. CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
57+
score: unknown # numeric score
58+
severity: unknown # Critical | High | Medium | Low
59+
60+
# ── Impact ────────────────────────────────────────────────────────────────────
61+
impact:
62+
attack_vector: unknown # Network | Local | Physical | Adjacent
63+
authentication_required: unknown # true | false | unknown
64+
user_interaction_required: unknown # true | false | unknown
65+
scope_changed: unknown # true | false | unknown
66+
confidentiality: unknown # High | Low | None | unknown
67+
integrity: unknown # High | Low | None | unknown
68+
availability: unknown # High | Low | None | unknown
69+
70+
# ── Dedup ─────────────────────────────────────────────────────────────────────
71+
dedup:
72+
nvd_searched: false
73+
ghsa_searched: false
74+
ecosystem_db_searched: false
75+
existing_cve: unknown # CVE-YYYY-NNNNN if found, "none" if clean, "unknown" if not searched
76+
notes: ""
77+
78+
# ── Disclosure ────────────────────────────────────────────────────────────────
79+
disclosure:
80+
vendor_contacted: false
81+
contact_date: unknown
82+
vendor_response: unknown
83+
planned_disclosure_date: unknown
84+
85+
# ── Blockers ──────────────────────────────────────────────────────────────────
86+
blockers: []
87+
# Common values:
88+
# - "no tested version"
89+
# - "source to sink not proven"
90+
# - "likely duplicate CVE"
91+
# - "developer-controlled trigger only"
92+
# - "no local reproducer"
93+
94+
# ── Audit Verdict ─────────────────────────────────────────────────────────────
95+
verdict:
96+
exploitability: unknown # proven | plausible | blocked | disproven | unknown
97+
confidence: unknown # high | medium | low | unknown
98+
reason: "" # concise reason for the current verdict
99+
100+
# ── Provenance ────────────────────────────────────────────────────────────────
101+
provenance:
102+
verification_date: ""
103+
researcher: ""
104+
unverified_fields: [] # list field paths that carry "unknown" or unconfirmed data; checked by CLI validation
105+
tool_versions: {} # e.g. {omv-find: "1.0"}
106+
107+
# ── Evidence and Submission Scores (computed, not stored) ─────────────────────
108+
# omv computes evidence_score from field completeness.
109+
# omv computes submission_score by penalizing unresolved blockers, unknown
110+
# observed_result, unknown affected range, incomplete dedup, and blocked/disproven
111+
# verdicts. A high evidence_score is not enough for submission-ready output.
112+
#
113+
# Evidence scoring guide (total 100):
114+
# tested version present +20
115+
# source identified +10
116+
# sink identified +10
117+
# guard missing confirmed +10
118+
# local reproducer written +15
119+
# observed result documented +10
120+
# cvss vector present +10
121+
# dedup search completed +10
122+
# vendor contacted +5
123+
#
124+
# Threshold: submission_score >= 75 plus validation OK is required before
125+
# suggesting promotion to confirmed.
126+
cve_readiness_scoring:
127+
tested_version: 20
128+
source_identified: 10
129+
sink_identified: 10
130+
guard_missing_confirmed: 10
131+
local_reproducer: 15
132+
observed_result: 10
133+
cvss_vector: 10
134+
dedup_searched: 10
135+
vendor_contacted: 5
136+
total: 100
137+
threshold: 75

0 commit comments

Comments
 (0)