Skip to content

Refactor CPE imports and add NVDCPEHandler for JSON feeds#19

Merged
adulau merged 8 commits into
vulnerability-lookup:mainfrom
oh2fih:nvdcpe-json-import
Sep 16, 2025
Merged

Refactor CPE imports and add NVDCPEHandler for JSON feeds#19
adulau merged 8 commits into
vulnerability-lookup:mainfrom
oh2fih:nvdcpe-json-import

Conversation

@oh2fih

@oh2fih oh2fih commented Sep 8, 2025

Copy link
Copy Markdown
Contributor

This PR refactors the legacy CPE dictionary import logic to be more extendable and adds support for the new NVD CPE Dictionary 2.0 JSON feeds.

  • Refactor legacy XML CPE imports
    • Introduced abstract base class CPEImportHandler.
    • Added XMLCPEHandler for legacy XML parsing.
    • Added CPEDownloader for downloading and uncompressing CPE dictionaries.
  • Support for NVD CPE Dictionary 2.0 JSON
    • Added NVDCPEHandler to handle the new JSON format.
    • Supports:
      • Tar archives containing multiple JSON files (current standard).
      • Single JSON files (for flexibility).
    • Deprecated CPEs are skipped to reduce false positives (a behavior not present in XML import).
    • Logs each JSON file inside tar archives for easier debugging.
    • Add an example of the new format in README.md.
  • Updates to import logic (import.py)
    • Import logic moved into the library for cleaner structure and reuse.
    • File-type detection based on extension; wrong content types will raise exceptions.
    • Simplified CLI flags:
      • Removed redundant --update.
      • Converted toggle flag --download to boolean.
  • Configuration
    • Default configuration updated to use CPE Dictionary 2.0 feed.
    • Fix inconsistency between default configuration and documentation (in favor of documentation).

Resolves #18

Refactored the legacy XML CPE dictionary import logic into dedicated
handler classes, introducing an abstract base class and an XML-specific
implementation. Also added a downloader class, preparing the codebase
for cleaner separation of responsibilities and future format support.
Reworked import.py to use the new cpeimport classes, moving the import
logic into the library for better structure and reuse. Simplified CLI
flags by removing the redundant --update option (inverse of --replace
and previously mis-handled) and converting toggle flags to booleans.
Add a handler class for the new NVD CPE Dictionary 2.0 format, laying
the foundation for importing JSON feeds alongside the legacy imports.
Use NVDCPEHandler for CPE data provided as tar archives or single JSON
files, while XML files continue to use XMLCPEHandler. File-type
detection is based naively on the file extension; providing the wrong
content type will raise exceptions in the handlers.
@oh2fih

oh2fih commented Sep 8, 2025

Copy link
Copy Markdown
Contributor Author

Testing legacy XMLCPEHandler (existing file only)

cpe:
  path: './data/official-cpe-dictionary_v2.3.xml'
  source: 'https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz'
$ python3 ./bin/import.py --replace
Using existing file ./data/official-cpe-dictionary_v2.3.xml ...
Flushing 165859 keys from the database...
Populating the database (please be patient)...
Using XMLCPEHandler to parse file ./data/official-cpe-dictionary_v2.3.xml...
... 5000 items processed (13519 words) in 2 seconds
. . .
... 1455000 items processed (4337016 words) in 686 seconds
Finished XMLCPEHandler[official-cpe-dictionary_v2.3.xml]: 1456315 items (4341134 words) in 687 seconds.
Done! 166263 keys inserted.

Testing new NVDCPEHandler together with CPEDownloader

cpe:
  path: './data/nvdcpe-2.0.tar'
  source: 'https://nvd.nist.gov/feeds/json/cpe/2.0/nvdcpe-2.0.tar.gz'
$ python3 ./bin/import.py --replace --download
Downloading CPE data from https://nvd.nist.gov/feeds/json/cpe/2.0/nvdcpe-2.0.tar.gz ...
Uncompressing ./data/nvdcpe-2.0.tar.gz ...
Flushing 166263 keys from the database...
Populating the database (please be patient)...
Using NVDCPEHandler to parse file ./data/nvdcpe-2.0.tar...
NVDCPEHandler parsing nvdcpe-2.0-chunks/nvdcpe-2.0-chunk-00001.json...
... 5000 items processed (14121 words) in 3 seconds
. . .
NVDCPEHandler parsing nvdcpe-2.0-chunks/nvdcpe-2.0-chunk-00014.json...
... 1345000 items processed (4009090 words) in 653 seconds
... 1350000 items processed (4023734 words) in 656 seconds
... 1355000 items processed (4040000 words) in 659 seconds
... 1360000 items processed (4054575 words) in 661 seconds
... 1365000 items processed (4068814 words) in 662 seconds
... 1370000 items processed (4082728 words) in 664 seconds
... 1375000 items processed (4095072 words) in 666 seconds
... 1380000 items processed (4109651 words) in 669 seconds
Finished NVDCPEHandler[nvdcpe-2.0.tar]: 1382942 items (4115900 words), 86664 skipped in 670 seconds.
Done! 165859 keys inserted.
$ python3 ./bin/import.py --replace
Using existing file ./data/nvdcpe-2.0.tar ...
Flushing 165859 keys from the database...
Populating the database (please be patient)...
Using NVDCPEHandler to parse file ./data/nvdcpe-2.0.tar...
NVDCPEHandler parsing nvdcpe-2.0-chunks/nvdcpe-2.0-chunk-00001.json...
... 5000 items processed (14121 words) in 4 seconds
. . .
... 1380000 items processed (4109651 words) in 676 seconds
Finished NVDCPEHandler[nvdcpe-2.0.tar]: 1382942 items (4115900 words), 86664 skipped in 677 seconds.
Done! 165859 keys inserted.

Lookup after JSON import

$ python3 lookup.py microsoft teams | jq .
[
  [
    118171,
    "cpe:2.3:a:microsoft:teams"
  ],
  [
    113340,
    "cpe:2.3:o:hp:elite_slice_g2_-_partner_ready_with_microsoft_teams_rooms_firmware"
  ],
  [
    111235,
    "cpe:2.3:h:hp:elite_slice_g2_-_partner_ready_with_microsoft_teams_rooms"
  ],
  [
    108091,
    "cpe:2.3:o:hp:elite_slice_g2_with_microsoft_teams_rooms_firmware"
  ],
  [
    103192,
    "cpe:2.3:h:hp:elite_slice_g2_with_microsoft_teams_rooms"
  ]
]

Also make the relative path compatible with README.md that
suggests running 'python3 ./bin/import.py'.
@oh2fih oh2fih force-pushed the nvdcpe-json-import branch 3 times, most recently from 10762f1 to 04f1824 Compare September 8, 2025 08:08
All scripts in bin/ already have proper shebangs; add executable
permissions so they can be run directly.
Keep the example of the legacy format as we still support it.
Update copyright years.
@oh2fih oh2fih force-pushed the nvdcpe-json-import branch from 04f1824 to 037795f Compare September 8, 2025 08:11
@adulau

adulau commented Sep 11, 2025

Copy link
Copy Markdown
Member

Awesome! I'll do some tests and merge it if everything is fine. Thanks a lot. That's a great contribution.

@adulau adulau self-requested a review September 12, 2025 09:54
@adulau adulau merged commit 5bc11c0 into vulnerability-lookup:main Sep 16, 2025
2 checks passed
@oh2fih oh2fih deleted the nvdcpe-json-import branch September 16, 2025 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Switch from retired XML dictionary to current JSON dictionary

2 participants