Fix(PypiMetaAnalyzer): Compliant with the Simple Repository API (PEP 503|691) [DependencyTrack#5735]

[ch8matt]

Description

Refactor PypiMetaAnalyzer to comply with the Python Simple Repository API (PEP 503 & PEP 691) instead of relying on the legacy /pypi/<project>/json endpoint.

The analyzer now retrieves metadata from /simple/<normalized-project-name>/ using HTTP content negotiation and supports both:

• JSON responses (application/vnd.pypi.simple.v1+json, PEP 691)
• HTML responses (PEP 503)

latestVersion is derived from standardized Simple API data, preferring non-yanked releases unless all releases are yanked.
publishedTimestamp is extracted from JSON upload-time fields when available.

This change enables compatibility with PyPI-compatible registries (e.g., GitLab PyPI Registry) that do not implement the legacy PyPI JSON endpoint.

Addressed Issue

Fixes #5735

Additional Details

Previous Behavior

PypiMetaAnalyzer queried: /pypi/<project>/json
This endpoint is specific to PyPI and not part of the official Python packaging specification.
As a result, Dependency-Track failed to resolve metadata when using PyPI-compatible registries that only implement the Simple Repository API.

New Behavior

The analyzer now:

• Normalizes project names according to PEP 503
• Requests: /simple/<normalized-name>/
• Uses an Accept header to prefer JSON (PEP 691) while falling back to HTML (PEP 503)
• Selects parsing logic strictly based on Content-Type
• Computes:
• latestVersion from available versions (preferring non-yanked releases)
• publishedTimestamp from JSON upload-time when available

Additional Changes

• Introduced HTML parsing via jsoup
• Refactored AbstractMetaAnalyzer to support configurable Accept headers
• Added comprehensive tests covering:
• JSON responses
• HTML responses
• Yanked-only releases

Checklist

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have added corresponding update logic
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[owasp-dt-bot]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.01% (target: -1.00%)	✅ 82.61% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (33f6e8b)	24558	19987	81.39%
Head commit (d8bf4a2)	24660 (+102)	20072 (+85)	81.39% (+0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#5778)	138	114	82.61%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences