Skip to content

Commit 12f8661

Browse files
chg: Release 3.0.0
CNVD severity trainer: data leakage fix, per-class metrics, class weighting options, focal loss, dynamic model card, best checkpoint selection. CNVD dataset: CVE cross-references, dataset card. New severity model comparison validator. CodeCarbon tracker fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 9cf2f8e commit 12f8661

2 files changed

Lines changed: 36 additions & 1 deletion

File tree

CHANGELOG.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,40 @@
11
# Changelog
22

3+
## Release 3.0.0 (2026-04-03)
4+
5+
### What's New
6+
7+
#### CNVD Severity Trainer
8+
9+
- **Data leakage fix**: new `deduplicate_split()` function groups entries by description text before splitting, preventing identical descriptions from appearing in both train and test sets. CNVD reuses boilerplate descriptions across different vulnerability IDs, which previously contaminated 15.6% of the test set.
10+
- **Per-class metrics**: `compute_metrics` now reports precision, recall, and F1 per class (Low/Medium/High) alongside overall accuracy and macro F1 at each evaluation epoch.
11+
- **Class weighting options**: new `--class-weights` flag (`none`, `sqrt`, `balanced`, `focal`) for experimenting with class imbalance strategies. Includes a `FocalLossTrainer` implementation (Lin et al., 2017). Defaults to uniform loss after experiments showed all weighting strategies degrade Medium recall disproportionately.
12+
- **Best model checkpoint selection**: `metric_for_best_model` set to `accuracy` (was defaulting to `eval_loss`), `save_total_limit` increased from 2 to 3.
13+
- **Dynamic model card**: model card is now a template populated with actual eval metrics from `trainer.evaluate()` after each training run. Documents per-class metrics, training configuration, and known limitations.
14+
15+
#### CNVD Dataset
16+
17+
- **CVE cross-references**: `extract_cnvd` now extracts the `cve_id` field from `cves.cve.cveNumber`, enabling cross-referencing with CVE equivalents (~81% of entries).
18+
- **Dataset card**: new dataset card documenting severity distribution, CVE overlap rates, coverage decline post-RMSV (94% published in 2015 to 4% in 2023), and duplicate description caveat.
19+
20+
#### Validation
21+
22+
- **CNVD severity model comparison validator**: new `validators/severity_cnvd.py` script to evaluate old and new models side by side on the same deduplicated test set, reporting per-class metrics, confusion matrices, and summary deltas.
23+
24+
### Fixes
25+
26+
- **CodeCarbon tracker hang**: replaced `@track_emissions` decorator with explicit `EmissionsTracker` scoped to `trainer.train()` only. Removed `push_to_hub=True` from `TrainingArguments` which caused the trainer to upload internally before returning. Applied to both `classify_severity_cnvd.py` and `classify_severity.py`.
27+
28+
### Documentation
29+
30+
- Added technical report: `docs/cnvd-severity-improvements.md`.
31+
- Improved `docs/index.md` with configuration section, CNVD-specific details, and validator usage.
32+
33+
### Acknowledgments
34+
35+
Thanks to [Eric Romang](https://github.com/eromang) for his thorough independent analysis ([VulnTrain#19](https://github.com/vulnerability-lookup/VulnTrain/issues/19)) that prompted these improvements.
36+
37+
338
## Release 2.2.0 (2026-02-19)
439

540
### What's New

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ build-backend = "poetry.core.masonry.api"
55

66
[project]
77
name = "VulnTrain"
8-
version = "2.2.0"
8+
version = "3.0.0"
99
description = "Generate datasets amd models based on vulnerabilities data from Vulnerability-Lookup."
1010
authors = [
1111
{name = "Cédric Bonhomme", email = "cedric.bonhomme@circl.lu"},

0 commit comments

Comments
 (0)