Skip to content

refactor(contracts): single source of truth for evidence/submission scoring - #19

Merged
bx33661 merged 1 commit into
mainfrom
refactor/scoring-single-source-of-truth
Jul 6, 2026
Merged

refactor(contracts): single source of truth for evidence/submission scoring#19
bx33661 merged 1 commit into
mainfrom
refactor/scoring-single-source-of-truth

Conversation

@bx33661

@bx33661 bx33661 commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Context

The project had three overlapping scoring concepts (evidence_score in CLI, submission_score in CLI, cve_readiness_score only in SPEC/comments), three naming conventions, and a Python renderer that silently disagreed with the TypeScript CLI on four deduction values plus three missing deductions and one spurious one. The same YAML produced different numbers depending on the path.

This PR makes the CLI the single canonical source and propagates it everywhere. No CLI logic is changed — this is a cleanup of the contract, SPEC, skills, and Python renderer to mirror what the CLI already does.

Changes

contracts/evidence.v1.yaml

  • Rename the cve_readiness_scoring: block → evidence_score_weights: (it already matched computeEvidenceScore in findings.ts exactly).
  • Add a new submission_score: block listing every deduction id + points + cap, plus cvss_confidence_penalty rules, mirroring SUBMISSION_DEDUCTIONS and cvssConfidencePenalty in findings.ts.
  • Update the line-14 comment to refer to submission_score (the ≥75 gate) rather than the ambiguous 'CVE readiness score'.

SPEC.md

  • Replace all 10 references to cve_readiness_score with evidence_score / submission_score as appropriate (the ≥75 gate is always on submission_score).
  • Replace the divergent cve_readiness_scoring block at line 247 with the canonical one matching the contract. Remove the never-implemented impact_fields_complete: 10 field, fix tested_version: 20 (was incorrectly 15 in SPEC), local_reproducer: 15 (was 20), and rename duplicate_search_donededup_searched.
  • Note: SPEC.md is gitignored locally so this change is visible in the file but is informational; the contract YAML is the tracked canonical source.

skills/omv-report/SKILL.md

  • Inline-define submissionScore as submission_score per the contract when it appears in the preflight block.

skills/omv-report/scripts/render_template.py

  • Re-implement _compute_scores() to mirror src/cli/findings.ts computeSubmissionScore exactly. Fixed values:
    Deduction was now
    missingObservedResult -20 -25
    unresolvedBlockers min(30, n*15) -30 flat
    unknownAffectedRange -15 -10
    incompleteDedup -10 -15
    blockedOrDisproven -30 -50
    plausibleExploitability missing -10
    confirmedBelowThreshold missing -10
    missingReproArtifacts missing -10
    'no tested version' -20 removed (no such CLI deduction)
    cvss_confidence_penalty missing 3/unverified cap 20 + confidence bumps
    blocked → 0 missing added
  • Add unverified_fields to Finding dataclass and load it from provenance.unverified_fields so the penalty has a real input.

Verification

  • npm run build — passes
  • npm test — all 37 tests pass
  • python3 skills/omv-report/scripts/render_template.py --finding .omv/findings/go-webhookx-ssrf.yaml --format vuldb --force — renders evidence: 95/100 | submission: 50/100 matching CLI math: 95 (10 evidence fields set, no vendor_contacted) − 10 (unknown affected_range) − 15 (5 unverified fields × 3, capped at 20) − 20 (confidence unknown) − 10 (confirmed-below-threshold bonus) = 50. ✓
  • grep -rn cve_readiness_score contracts/ skills/ SPEC.md — zero hits in tracked files

Follow-up PR (Phase 2, not in this one)

Split src/cli/findings.ts (1603 lines) into validate.ts + workflow.ts + threatmap.ts + repro.ts, and consolidate rendering in render.ts. The contract now references src/cli/findings.ts computeSubmissionScore deliberately so Phase 1 is self-consistent before Phase 2 moves things around.

Checklist

  • Tests pass
  • Build passes
  • Smoke-tested renderer on real finding
  • Zero stranded cve_readiness_score references in tracked files

…s + add submission_score section

Break the single ambiguous 'CVE readiness score' concept into two
explicitly documented scores:

- evidence_score (field completeness, 0-100, never penalized)
- submission_score (evidence_score minus deductions/clamping, 0-100,
  the actual gate at >=75)

Changes:
- contracts/evidence.v1.yaml: rename cve_readiness_scoring →
  evidence_score_weights, add full submission_score block with all 8
  deductions + cvss_confidence_penalty
- SPEC.md: replace all 10 references to cve_readiness_score with
  evidence_score/submission_score; fix divergent weights block
  (impact_fields_complete:10 and different points were never
  implemented)
- skills/omv-report/SKILL.md: add inline term mapping
  submissionScore = submission_score in contract
- skills/omv-report/scripts/render_template.py: re-implement
  _compute_scores() to exactly mirror CLI submissionsDeductions
  logic (was off by 4 deduction values + missing 3 deductions +
  unused 'missing tested version' deduction)
@bx33661
bx33661 merged commit c031042 into main Jul 6, 2026
0 of 2 checks passed
@bx33661
bx33661 deleted the refactor/scoring-single-source-of-truth branch July 26, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant