feat: support pyproject.toml and lockfiles in pypi verify, fail cleanly on wrong file types (#781) - #812
Conversation
…ly on wrong file types (DataDog#781) pypi verify now understands three TOML dependency formats in addition to requirements.txt: - pyproject.toml with PEP 621 [project] dependencies and optional-dependencies - pyproject.toml with Poetry [tool.poetry.dependencies], group dependencies, and legacy dev-dependencies; caret/tilde constraints are translated to PEP 440 ranges, and git/path/url dependencies plus the python constraint are skipped - poetry.lock / uv.lock, using the exact pinned versions from [[package]] The format is sniffed from content, so both file and directory targets work; find_requirements also discovers pyproject.toml, poetry.lock, and uv.lock. Version resolution now uses SpecifierSet instead of Specifier, which also fixes comma-separated ranges (e.g. "flask>=2,<3") that previously fell back to the raw string even in requirements.txt. Wrong file types now fail nicely instead of crashing: npm verify on a non-package.json raises a clear ValueError surfaced as a one-line error (previously an unhandled JSONDecodeError traceback), pypi verify on non-requirements content logs a warning instead of silently scanning nothing, and the verify command exits 1 on scan errors without a traceback. tomli is added as a dependency for Python 3.10 only; 3.11+ uses the stdlib tomllib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 980fdd2145
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| yara-python = "^4.5.4" | ||
| nono-py = ">=0.11.0" | ||
| packaging = ">=21.0" | ||
| tomli = { version = ">=2.0.1", python = "<3.11" } |
There was a problem hiding this comment.
This new dependency changes pyproject.toml but the root poetry.lock is not updated. The checked workflow installs with Poetry at .github/workflows/checks.yml:49 and :147; poetry install --dry-run --only main on this commit fails with pyproject.toml changed significantly since poetry.lock was last generated, so CI will stop before tests until the lockfile is regenerated.
Useful? React with 👍 / 👎.
| constraint = self._poetry_constraint_to_pep440(spec) | ||
| return [f"{name}{constraint}"] | ||
| if isinstance(spec, dict): | ||
| if any(key in spec for key in ("git", "path", "url")): |
There was a problem hiding this comment.
When a Poetry dependency is selected from a named private source, e.g. foo = { version = "1.2.3", source = "internal" }, this condition does not treat it like the other non-PyPI forms. The code then emits foo==1.2.3 and _resolve_requirement_lines queries/scans public PyPI, so projects using private sources can get findings for an unrelated public package; skip or otherwise handle source entries here.
Useful? React with 👍 / 👎.
| requirement_lines = [ | ||
| f"{p['name']}=={p['version']}" | ||
| for p in data["package"] | ||
| if isinstance(p, dict) and "name" in p and "version" in p |
There was a problem hiding this comment.
When a poetry.lock/uv.lock contains non-PyPI packages with a version (for example source = { git = ... } or a private registry source), this comprehension still emits name==version for them. _resolve_requirement_lines then resolves that name through https://pypi.org/pypi/..., so verify can scan an unrelated public package or report the locked dependency missing; filter lockfile packages by their recorded source before adding them.
Useful? React with 👍 / 👎.
- Regenerate poetry.lock for the tomli main-group dependency so poetry install passes again (tomli was already locked as a dev dependency; the diff is the group change plus content-hash) - Skip Poetry dependencies that select a named source (source = "internal"): resolving them against public PyPI could scan an unrelated package sharing the name - Filter lockfile [[package]] entries by recorded source: keep poetry.lock entries without a source table and uv.lock entries whose registry is pypi.org; skip git/path/private-registry packages - Add tests for all three cases Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed all three: regenerated poetry.lock (minimal diff, tomli was already locked as a dev dep), and both pyproject and lockfile parsing now skip non-PyPI sources, with tests for each. Good catches on the private-source cases. |
2 participants
Closes #781. Covers both parts of the issue, plus one adjacent resolution fix surfaced along the way.
pyproject.toml and lockfile support
pypi verifynow understands three TOML dependency formats in addition to requirements.txt, sniffed from content so both file and directory targets work:[project] dependenciesand[project.optional-dependencies][tool.poetry.dependencies], group dependencies, and legacy dev-dependencies. Caret/tilde constraints are translated to PEP 440 ranges (^2.2→>=2.2,<3); git/path/url dependencies and thepythonconstraint are skipped since they can't be verified against PyPI[[package]]entriesfind_requirementsalso discoverspyproject.toml,poetry.lock, anduv.lockduring directory scans.Fail nicely on wrong file types
npm verifyon a non-package.json now raises a clearValueErrorsurfaced as a one-line error; previously it dumped an unhandledJSONDecodeErrortracebackpypi verifyon non-requirements content (e.g. a package.json) logs a warning ("No valid Python requirements found...") instead of silently scanning nothing_verifyexits 1 on scan errors without a tracebackAdjacent fix
Version resolution used
packaging.Specifier, which only accepts a single constraint, so comma-separated ranges (e.g.flask>=2,<3in a plain requirements.txt) fell into the raw-string fallback and never resolved. Switched toSpecifierSet, which handles both and preserves the raw fallback for git/URL references (the #88 regression tests still pass).tomliis added as a dependency for Python 3.10 only; 3.11+ uses the stdlibtomllib.Testing
flake8with repo flags: 0 issues;black --check: clean);mypyclean on the changed scannerpypi verifyon a pyproject.toml and a poetry.lock (both resolve and scan),npm verifyon a text file (one-line error, exit 1, no traceback),pypi verifyon a package.json (warning, no crash)🤖 Generated with Claude Code