Skip to content

cnvd-severity-improvements.md

Latest commit

 

History

History
107 lines (69 loc) · 6.57 KB

cnvd-severity-improvements.md

File metadata and controls

107 lines (69 loc) · 6.57 KB
Raw

CNVD Severity Classifier: Improvements Report

Technical report on changes made to the CNVD severity trainer (classify_severity_cnvd.py) following the independent analysis in VulnTrain#19.

Context

An external review of the CIRCL/vulnerability-severity-classification-chinese-macbert-base model identified several issues: data leakage in the train/test split, poor Low-class recall, keyword dependency, and a suboptimal published checkpoint. The reported headline accuracy of 77.83% was inflated by ~1.7pp due to leakage.

Changes

Data leakage fix

Problem: CNVD reuses boilerplate descriptions across different vulnerability IDs. The original train_test_split split by row index, allowing 1,587 identical descriptions (15.6% of the test set) to appear in both splits.

Fix: New deduplicate_split() function groups all entries by description text and assigns entire groups to one split. No description appears in both train and test.

Impact: The old model evaluated on the deduplicated test set scores 85.2% (inflated — it was trained on data overlapping this test set). A model retrained on the deduplicated split scores 76.8%, matching the independently measured unleaked accuracy of 76.6%.

Class weighting experiments

Problem: The Low class (~9% of data) had only ~41% recall on unleaked data, with 60% of Low entries misclassified as Medium.

Four loss strategies were tested on the deduplicated split:

Mode Low recall Medium recall High recall Overall acc
Uniform (none) 0.4099 0.8165 0.7809 0.7677
Sqrt-dampened 0.4902 0.7481 0.8056 0.7457
Balanced 0.6084 0.7024 0.8099 0.7323
Focal (gamma=2) 0.6328 0.6441 0.8349 0.7110

Conclusion: Every weighting strategy that improved Low recall caused disproportionate Medium recall loss. The Low/Medium vocabulary overlap in CNVD descriptions makes this a data-level limitation, not a loss-function problem. The trainer defaults to uniform loss.

A --class-weights flag (none, sqrt, balanced, focal) was added for future experimentation.

Per-class metrics

compute_metrics now reports precision, recall, and F1 per class (Low/Medium/High) alongside overall accuracy and macro F1 at each evaluation epoch.

Best model checkpoint selection

  • metric_for_best_model set to accuracy (was defaulting to eval_loss)
  • save_total_limit increased from 2 to 3 to prevent the best checkpoint from being pruned

CodeCarbon tracker scoping

The @track_emissions decorator wrapped the entire train() function, including push_to_hub(). The codecarbon background thread never stopped during the upload. Replaced with an explicit EmissionsTracker start/stop scoped to trainer.train() only. Also removed push_to_hub=True from TrainingArguments (it caused trainer.train() to upload internally before returning). The same fix was applied to classify_severity.py.

Dynamic model card

The model card is now a template (model_card_cnvd_severity.md) populated with actual eval metrics from trainer.evaluate() after each training run. Documents per-class metrics, training configuration, and known limitations.

Known limitations documented

The model card now documents:

  • Low severity recall (~41%): ~60% of Low entries are misclassified as Medium due to vocabulary overlap. All weighting strategies degrade Medium recall disproportionately.
  • Keyword dependency: the model biases toward a vulnerability type's typical severity. Accuracy drops from ~89% to ~55% on atypical-severity entries.
  • Negation blindness: "does NOT allow RCE" still predicts High with high confidence.
  • CVE overlap: 81% of CNVD entries have a CVE equivalent. The model primarily adds value for the ~19% CNVD-only entries.

These findings align with independent results from CyberScale Phase 1 (Point 29), which plateaued at ~62% band accuracy on a 4-class CVSS classifier using ModernBERT-base with similar approaches (CWE enrichment, multi-task heads, CPE features — none moving the needle beyond ~2pp).

Dataset improvements

CVE cross-references

The extract_cnvd function now extracts the cve_id field from cves.cve.cveNumber in the raw CNVD JSON. This enables users to cross-reference CNVD entries with their CVE equivalents and filter CNVD-only entries.

Dataset card

A dataset card (dataset_card_cnvd.md) was added documenting:

  • Field descriptions including the new cve_id column
  • CVE overlap rate: 81% overall (68-69% in 2020-2021, 91-97% after 2022)
  • Severity distribution: High ~36%, Medium ~55%, Low ~9%
  • Coverage decline: 94% of reserved IDs published in 2015 → 4% in 2023 (post-RMSV regulations, September 2021)
  • Warning about duplicate descriptions and train/test split leakage

Comparison validator

A dedicated validator (validators/severity_cnvd.py) was added to evaluate the old and new models side by side on the same deduplicated test set. It reports per-class precision/recall/F1, confusion matrices, and a summary delta table.

Commits

Commit Description
6352273 Data leakage fix, class-weighted loss, per-class metrics, best model selection
b1679fb CNVD severity model comparison validator
1fdee05 Sqrt-dampened class weights
a81f90d Scoped codecarbon tracker (CNVD trainer)
65d3d88 Removed push_to_hub from TrainingArguments
b7a2d6d Scoped codecarbon tracker (severity trainer)
ed2c230 --class-weights flag (none/sqrt/balanced)
f1cd426 Focal loss option
9fa0f86 Default to uniform loss
7920361 Model card
5b2866c Dynamic model card from eval metrics
30f1872 CVE cross-references and dataset card

References