Skip to content

Normalize CPE guessing/indexing for dashed and underscored names#30

Open
adulau wants to merge 1 commit into
mainfrom
codex/review-guesser-for-vendor-name-handling
Open

Normalize CPE guessing/indexing for dashed and underscored names#30
adulau wants to merge 1 commit into
mainfrom
codex/review-guesser-for-vendor-name-handling

Conversation

@adulau

@adulau adulau commented Mar 31, 2026

Copy link
Copy Markdown
Member

Motivation

  • Hyphenated, underscored, or space-separated vendor/product names were not consistently tokenized, causing lookups like rocket-launcher, rocket_launcher, and rocket launcher to miss matches.
  • The importer and guesser must use the same tokenization rules so indexing and searching behave consistently for names with -, _, or spaces.

Description

  • Add _normalize_keywords to CPEGuesser which lowercases and splits input keywords on spaces, underscores, and dashes using re.split(r"[\s_-]+", ...), and use the normalized tokens for set intersection and scoring.
  • Change canonize in lib/cpeimport/base.py to split vendor/product strings with the same re.split(r"[\s_-]+", ...) rule so hyphenated names are indexed as component words.
  • Add regression tests in tests/test_cpeguesser.py to verify guessing works for rocket-launcher, rocket_launcher, and rocket launcher.
  • Add importer test coverage in tests/test_nvd_json.py to verify hyphenated vendor/product values are split and indexed into component words.

Testing

  • Ran unit subsets with PYTHONPATH=. pytest -q tests/test_cpeguesser.py tests/test_nvd_json.py and the tests passed (both files green).
  • Ran full test suite with PYTHONPATH=. pytest -q and observed all tests passing (12 passed).
  • A plain pytest -q run without PYTHONPATH=. failed to import the lib package in this environment, so test runs shown above use the correct PYTHONPATH to validate changes.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant