|
1 | 1 | # Changelog |
2 | 2 |
|
| 3 | +## Release 3.0.0 (2026-04-03) |
| 4 | + |
| 5 | +### What's New |
| 6 | + |
| 7 | +#### CNVD Severity Trainer |
| 8 | + |
| 9 | +- **Data leakage fix**: new `deduplicate_split()` function groups entries by description text before splitting, preventing identical descriptions from appearing in both train and test sets. CNVD reuses boilerplate descriptions across different vulnerability IDs, which previously contaminated 15.6% of the test set. |
| 10 | +- **Per-class metrics**: `compute_metrics` now reports precision, recall, and F1 per class (Low/Medium/High) alongside overall accuracy and macro F1 at each evaluation epoch. |
| 11 | +- **Class weighting options**: new `--class-weights` flag (`none`, `sqrt`, `balanced`, `focal`) for experimenting with class imbalance strategies. Includes a `FocalLossTrainer` implementation (Lin et al., 2017). Defaults to uniform loss after experiments showed all weighting strategies degrade Medium recall disproportionately. |
| 12 | +- **Best model checkpoint selection**: `metric_for_best_model` set to `accuracy` (was defaulting to `eval_loss`), `save_total_limit` increased from 2 to 3. |
| 13 | +- **Dynamic model card**: model card is now a template populated with actual eval metrics from `trainer.evaluate()` after each training run. Documents per-class metrics, training configuration, and known limitations. |
| 14 | + |
| 15 | +#### CNVD Dataset |
| 16 | + |
| 17 | +- **CVE cross-references**: `extract_cnvd` now extracts the `cve_id` field from `cves.cve.cveNumber`, enabling cross-referencing with CVE equivalents (~81% of entries). |
| 18 | +- **Dataset card**: new dataset card documenting severity distribution, CVE overlap rates, coverage decline post-RMSV (94% published in 2015 to 4% in 2023), and duplicate description caveat. |
| 19 | + |
| 20 | +#### Validation |
| 21 | + |
| 22 | +- **CNVD severity model comparison validator**: new `validators/severity_cnvd.py` script to evaluate old and new models side by side on the same deduplicated test set, reporting per-class metrics, confusion matrices, and summary deltas. |
| 23 | + |
| 24 | +### Fixes |
| 25 | + |
| 26 | +- **CodeCarbon tracker hang**: replaced `@track_emissions` decorator with explicit `EmissionsTracker` scoped to `trainer.train()` only. Removed `push_to_hub=True` from `TrainingArguments` which caused the trainer to upload internally before returning. Applied to both `classify_severity_cnvd.py` and `classify_severity.py`. |
| 27 | + |
| 28 | +### Documentation |
| 29 | + |
| 30 | +- Added technical report: `docs/cnvd-severity-improvements.md`. |
| 31 | +- Improved `docs/index.md` with configuration section, CNVD-specific details, and validator usage. |
| 32 | + |
| 33 | +### Acknowledgments |
| 34 | + |
| 35 | +Thanks to [Eric Romang](https://github.com/eromang) for his thorough independent analysis ([VulnTrain#19](https://github.com/vulnerability-lookup/VulnTrain/issues/19)) that prompted these improvements. |
| 36 | + |
| 37 | + |
3 | 38 | ## Release 2.2.0 (2026-02-19) |
4 | 39 |
|
5 | 40 | ### What's New |
|
0 commit comments