Skip to content

Add CVE v5 NDJSON rank importer and CVEListV5Handler#24

Merged
adulau merged 2 commits into
mainfrom
codex/add-cpe-extraction-component-in-cpe-guesser
Mar 22, 2026
Merged

Add CVE v5 NDJSON rank importer and CVEListV5Handler#24
adulau merged 2 commits into
mainfrom
codex/add-cpe-extraction-component-in-cpe-guesser

Conversation

@adulau

@adulau adulau commented Mar 22, 2026

Copy link
Copy Markdown
Member

Motivation

  • Provide a way to build vendor:product ranking (ZRANK) from the CIRCL cvelistv5.ndjson CVE v5 dump so common combined vendor:product tuples can be discovered.
  • Ensure extraction picks up CPEs found both in CVE metadata and in vulnerable configuration blocks, and normalize legacy cpe:/… entries to the cpe:2.3: vendor:product form.
  • Keep downloads idempotent by default and offer a force refresh option when re-downloading the NDJSON.

Description

  • Add lib/cpeimport/cvelistv5.py implementing CVEListV5Handler that parses NDJSON line-by-line, recursively extracts CPE strings, normalizes legacy cpe:/ values, deduplicates tuples per record, and emits rank-only batches.
  • Add CLI bin/import_cvelistv5.py to download ./data/cvelistv5.ndjson (only when missing unless --download), optionally clear rank keys with --replace-rank, and drive CVEListV5Handler.
  • Extend lib/cpeimport/base.py so CPEExtractor handles legacy cpe:/ syntax, add rank:vendor_product updates and a process_rank_batch helper used by the new handler.
  • Make downloader create destination directories before saving files by updating lib/cpeimport/downloader.py, export the new handler in lib/cpeimport/__init__.py, add config defaults in config/settings.yaml, and document the workflow in README.md.
  • Add unit tests in tests/test_cvelistv5.py covering recursive extraction, legacy normalization, per-record deduplication, and invalid-NDJSON handling.

Testing

  • Ran python -m unittest tests.test_nvd_json tests.test_cvelistv5 and the test suite completed successfully (all tests passed).
  • Ran python -m py_compile bin/import.py bin/import_cvelistv5.py lib/cpeimport/*.py lib/cpeguesser.py and byte-compilation checks passed.

Codex Task

@adulau adulau merged commit ce76596 into main Mar 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant