Drop language-scope filtering from crawler config - #86
Conversation
Removes ScopeConfig.languages and include_url_patterns, and the code paths that read them: scope evaluation, gap-crawl URL filtering, the discovery scope-reason map, and the crawler's derived-language manifest field. Sites are now crawled comprehensively, filtered only on allowed_domains/exclude_url_patterns/allowed_content_types, matching the 2026-08-05 discovery review's finding that per-site language structure is too inconsistent to scope-filter reliably, and that Finland's actual official languages (Finnish/Swedish) make English-only scoping wrong in the first place. Language-variant handling moves to the retrieval layer (#68). Also drops the now-stale /en path from tyomarkkinatori's gap_crawl seed URL and vero's base_url, matching the other sites' language-neutral config. ManifestRecord.language stays in the schema (harmless, forward-compatible with real future language detection) but is no longer derived from a forced single-language scope config, so it's always None until then. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 59 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
CI Results
All checks passed. 🎉 |
1 participant
Summary
ScopeConfig.languagesandinclude_url_patterns, and every code path that read them: scope evaluation (discovery/scope.py), gap-crawl URL filtering (discovery/gap_crawl.py), the discovery scope-reason map (discovery/runner.py), and the crawler's derived-language manifest field (crawler/crawler.py).allowed_domains/exclude_url_patterns/allowed_content_types./enpath fromtyomarkkinatori'sgap_crawl.seed_urlsand fromvero'sbase_url, matching the other sites' language-neutral config.ManifestRecord.languagestays in the schema (harmless, forward-compatible with real future language detection) but is no longer derived from a forced single-language scope config, so it's alwaysNonefor now.Motivated by the 2026-08-05 discovery review: per-site language path structure is too inconsistent to scope-filter reliably (kela.fi has no language signal at all; dvv.fi uses
/se/for Swedish, not the expected/sv/), and Finland's actual official languages are Finnish/Swedish, not English, so English-only scoping was wrong to begin with. Language-variant handling moves to the retrieval layer (#68).Closes #83. Companion spec update: #85.
Test plan
uv run pytest -q— 108 passeduv run ruff check ./uv run ruff format --check .— cleanuv run mypy tapio_crawler— no issues