Skip to content

Improve typosquatting top-packages sources for npm and PyPI - #799

Open
sobregosodd wants to merge 4 commits into
v3from
s.obregoso/improve_npm_toplist
Open

Improve typosquatting top-packages sources for npm and PyPI#799
sobregosodd wants to merge 4 commits into
v3from
s.obregoso/improve_npm_toplist

Conversation

@sobregosodd

@sobregosodd sobregosodd commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • npm: replaced dependency on github.com/LeoDog896/npm-rank (a third-party GitHub release that had already been renamed/redirected once) with direct pagination of the npms.io search API — the same source used by evals/benchmark.py
  • PyPI: replaced the hugovk.github.io URL (which 301-redirects to hugovk.dev) with a direct bulk query to the public ClickHouse instance that hosts PyPI's official download data (sql-clickhouse.clickhouse.com, table pypi.pypi_downloads_per_month) — the same upstream source hugovk uses
  • Both ecosystems now produce 10,000 packages (previously npm was capped at 8k, PyPI at 5k)
  • Cache logic is unchanged: 30-day TTL, falls back to stale data on network failure

Test plan

  • poetry run pytest tests/analyzer/metadata/test_typosquatting.py — all 39 tests pass
  • guarddog npm scan expresss flags as typosquat of express
  • guarddog pypi scan reqeusts flags as typosquat of requests

🤖 Generated with Claude Code

@sobregosodd
sobregosodd requested a review from a team as a code owner July 3, 2026 13:02
@datadog-official

This comment has been minimized.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread guarddog/analyzer/metadata/npm/typosquatting.py Outdated
@tesnim5hamdouni

Copy link
Copy Markdown
Contributor

nit: in the PR description- guarddog pypi scan reqeusts flags as typosquat of requests, the command actually retruns 404 because reqeusts doesn't exist

tesnim5hamdouni
tesnim5hamdouni previously approved these changes Jul 3, 2026

@tesnim5hamdouni tesnim5hamdouni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, left 2 comments

@christophetd

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a00f386cbe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread guarddog/analyzer/metadata/npm/typosquatting.py
try:
resp = requests.get(
_NPMS_URL.format(offset=offset),
headers={"User-Agent": "guarddog/1.0"},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably be guarddog 3.0? or just a dynamic version number?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants