fix(scan): bound crawl breadth with a page budget - #269
Conversation
Crawl only capped recursion depth, not breadth per level. A link-heavy page (pagination, faceted search) drives fetch count toward branching^depth with no ceiling: a depth-3 crawl of a page linking 40 fresh urls issued 1641 real fetches. Cap total fetches at a page budget via an atomic counter in an OnRequest hook that aborts once exceeded, and add a Truncated flag on CrawlResult so callers can tell a run was cut short.
github-actions
Bot
added
scan
pr summary2 files changed (+101 -2)
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #269 +/- ##
=======================================
Coverage ? 54.78%
=======================================
Files ? 81
Lines ? 6882
Branches ? 0
=======================================
Hits ? 3770
Misses ? 2842
Partials ? 270 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vmfunc
left a comment
vmfunc
left a comment
There was a problem hiding this comment.
counting in OnRequest before the dial is the right spot, abort lands before any fetch. counter is 1..500 pass and 501 aborts, so exactly maxCrawlPages get fetched, clean. Truncated read after Wait() so no race with the callbacks. scoping the context deadline out to its own PR is the right call. in.
3 participants
MaxDepthbounds recursion depth but nothing bounds breadth, so fetch count grows as branching^depth. Repro before the fix: a depth-3 crawl of one page linking 40 fresh urls issued 1641 real fetches; after, it stops at the 500-page budget and setsTruncated.Verified by
crawl_budget_test.go: a server linking past the budget capsvisitedatmaxCrawlPagesand flagsTruncated; an under-budget crawl leaves it false. Race-clean, the counter is atomic since colly fires callbacks across goroutines.Scope: only the runaway-breadth ceiling. A crawl-wide context deadline is a separate root cause and its own PR.