Skip to content

Handle incomplete/multiline NDJSON records in CVEListV5Handler and add tests#25

Merged
adulau merged 1 commit into
mainfrom
codex/fix-ndjson-parsing-errors-in-import_cvelistv5.py
Mar 22, 2026
Merged

Handle incomplete/multiline NDJSON records in CVEListV5Handler and add tests#25
adulau merged 1 commit into
mainfrom
codex/fix-ndjson-parsing-errors-in-import_cvelistv5.py

Conversation

@adulau

@adulau adulau commented Mar 22, 2026

Copy link
Copy Markdown
Member

Motivation

  • Improve robustness when ingesting NDJSON exports by correctly handling records that are split across lines or end up as incomplete JSON fragments so subsequent records are not lost.

Description

  • Update process_ndjson_file to buffer consecutive lines into pending_lines and attempt to parse a combined payload before skipping.
  • Add helper methods �_load_record�, �_is_incomplete_record�, �_looks_like_record_start�, �_process_record�, and �_skip_invalid_record� and the INCOMPLETE_ERROR_PREFIXES constant to centralize parsing and error handling.
  • Preserve existing behavior of recording skipped records and progress via record_progress and process_rank_batch while skipping only truly invalid or incomplete records.
  • Add unit test test_skips_single_invalid_multiline_record_and_continues to verify a broken multiline record is skipped and processing continues for following valid records.

Testing

  • Ran the tests/test_cvelistv5.py unit tests which include the new test_skips_single_invalid_multiline_record_and_continues test.
  • All unit tests passed locally with the updated handler and test additions.

Codex Task

@adulau adulau merged commit 72eb6f6 into main Mar 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant