-
Notifications
You must be signed in to change notification settings - Fork 0
Migrate to NVD JSON 2.0 feeds #68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: nscuro <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR replaces the legacy NVD API client with direct HTTP downloads of NVD JSON 2.0 feeds, parses them via Jackson’s streaming API, and removes the need for the NVD_TOKEN.
- Introduces
HttpClient‐based download, gzip decompression, and JacksonJsonParserlooping - Removes old
NvdCveClientcode and related scheduler/executor logic - Cleans up documentation and CI workflows to drop
NVD_TOKEN
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/main/java/org/dependencytrack/vulndb/source/nvd/NvdImporter.java | Replaces API client with HTTP + streaming JSON importer; handles per-year feed downloads |
| README.md | Removed reference to NVD_TOKEN environment variable |
| .github/workflows/update-database.yml | Removed NVD_TOKEN from CI workflow |
Comments suppressed due to low confidence (1)
README.md:47
- Since the importer no longer requires an
NVD_TOKEN, update this section to explain that no NVD-specific credentials are needed or remove the placeholder entirely.
- -e 'NVD_TOKEN=<your_nvd_token>' \
| this.objectMapper = new ObjectMapper() | ||
| .registerModule(new JavaTimeModule()) | ||
| .configure(JsonReadFeature.ALLOW_TRAILING_COMMA.mappedFeature(), true); | ||
| this.feedYears = IntStream.range(2002, LocalDate.now().getYear()).boxed().toList(); |
Copilot
AI
Jun 16, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IntStream.range(2002, LocalDate.now().getYear()) excludes the current year; use rangeClosed(start, LocalDate.now().getYear()) or adjust the end to include this year if you intend to fetch the latest feed.
| this.feedYears = IntStream.range(2002, LocalDate.now().getYear()).boxed().toList(); | |
| this.feedYears = IntStream.rangeClosed(2002, LocalDate.now().getYear()).boxed().toList(); |
| .map(instant -> OffsetDateTime.ofInstant(instant, ZoneOffset.UTC)) | ||
| .orElse(null); | ||
| if (savedLastModified != null | ||
| && (savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified))) { |
Copilot
AI
Jun 16, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The skip condition savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified) is inverted: it prevents downloads when the remote feed is newer. It should only skip when savedLastModified is after or equal to lastModified.
| && (savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified))) { | |
| && (savedLastModified.isAfter(lastModified) || savedLastModified.isEqual(lastModified))) { |
| while (jsonParser.nextToken() != JsonToken.END_ARRAY) { | ||
| final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class); | ||
| final Vulnerability vuln = convert(defCveItem.getCve()); | ||
|
|
||
| // TODO: Batching | ||
| database.storeVulnerabilities(List.of(vuln)); | ||
| } |
Copilot
AI
Jun 16, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling storeVulnerabilities per individual vulnerability can be inefficient. Consider batching several Vulnerability objects into a single call to reduce database round trips.
| while (jsonParser.nextToken() != JsonToken.END_ARRAY) { | |
| final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class); | |
| final Vulnerability vuln = convert(defCveItem.getCve()); | |
| // TODO: Batching | |
| database.storeVulnerabilities(List.of(vuln)); | |
| } | |
| final List<Vulnerability> vulnerabilities = new ArrayList<>(); | |
| while (jsonParser.nextToken() != JsonToken.END_ARRAY) { | |
| final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class); | |
| final Vulnerability vuln = convert(defCveItem.getCve()); | |
| vulnerabilities.add(vuln); | |
| } | |
| database.storeVulnerabilities(vulnerabilities); |
No description provided.