Skip to content

Conversation

@nscuro
Copy link
Member

@nscuro nscuro commented Jun 16, 2025

No description provided.

Copilot AI review requested due to automatic review settings June 16, 2025 13:09
@nscuro nscuro added the enhancement New feature or request label Jun 16, 2025
@nscuro nscuro merged commit 2f7728e into main Jun 16, 2025
3 checks passed
@nscuro nscuro deleted the nvd-json-feed-2.0 branch June 16, 2025 13:10
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR replaces the legacy NVD API client with direct HTTP downloads of NVD JSON 2.0 feeds, parses them via Jackson’s streaming API, and removes the need for the NVD_TOKEN.

  • Introduces HttpClient‐based download, gzip decompression, and Jackson JsonParser looping
  • Removes old NvdCveClient code and related scheduler/executor logic
  • Cleans up documentation and CI workflows to drop NVD_TOKEN

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/main/java/org/dependencytrack/vulndb/source/nvd/NvdImporter.java Replaces API client with HTTP + streaming JSON importer; handles per-year feed downloads
README.md Removed reference to NVD_TOKEN environment variable
.github/workflows/update-database.yml Removed NVD_TOKEN from CI workflow
Comments suppressed due to low confidence (1)

README.md:47

  • Since the importer no longer requires an NVD_TOKEN, update this section to explain that no NVD-specific credentials are needed or remove the placeholder entirely.
-  -e 'NVD_TOKEN=<your_nvd_token>' \

this.objectMapper = new ObjectMapper()
.registerModule(new JavaTimeModule())
.configure(JsonReadFeature.ALLOW_TRAILING_COMMA.mappedFeature(), true);
this.feedYears = IntStream.range(2002, LocalDate.now().getYear()).boxed().toList();
Copy link

Copilot AI Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IntStream.range(2002, LocalDate.now().getYear()) excludes the current year; use rangeClosed(start, LocalDate.now().getYear()) or adjust the end to include this year if you intend to fetch the latest feed.

Suggested change
this.feedYears = IntStream.range(2002, LocalDate.now().getYear()).boxed().toList();
this.feedYears = IntStream.rangeClosed(2002, LocalDate.now().getYear()).boxed().toList();

Copilot uses AI. Check for mistakes.
.map(instant -> OffsetDateTime.ofInstant(instant, ZoneOffset.UTC))
.orElse(null);
if (savedLastModified != null
&& (savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified))) {
Copy link

Copilot AI Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skip condition savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified) is inverted: it prevents downloads when the remote feed is newer. It should only skip when savedLastModified is after or equal to lastModified.

Suggested change
&& (savedLastModified.isBefore(lastModified) || savedLastModified.isEqual(lastModified))) {
&& (savedLastModified.isAfter(lastModified) || savedLastModified.isEqual(lastModified))) {

Copilot uses AI. Check for mistakes.
Comment on lines +113 to 119
while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class);
final Vulnerability vuln = convert(defCveItem.getCve());

// TODO: Batching
database.storeVulnerabilities(List.of(vuln));
}
Copy link

Copilot AI Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling storeVulnerabilities per individual vulnerability can be inefficient. Consider batching several Vulnerability objects into a single call to reduce database round trips.

Suggested change
while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class);
final Vulnerability vuln = convert(defCveItem.getCve());
// TODO: Batching
database.storeVulnerabilities(List.of(vuln));
}
final List<Vulnerability> vulnerabilities = new ArrayList<>();
while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
final var defCveItem = objectMapper.readValue(jsonParser, DefCveItem.class);
final Vulnerability vuln = convert(defCveItem.getCve());
vulnerabilities.add(vuln);
}
database.storeVulnerabilities(vulnerabilities);

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants