Fix 403 errors on CDN-protected sites and auto-recover broken cache states - #67
Open
fakhavan wants to merge 1 commit into
Open
Fix 403 errors on CDN-protected sites and auto-recover broken cache states #67fakhavan wants to merge 1 commit into
fakhavan wants to merge 1 commit into
Conversation
… cache states Implement a method in DatabaseManager to reset a link's visited status if it has no corresponding page content. Update Scraper to utilize this method when a URL is processed without valid content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem 1: 403 Forbidden on child pages
Some sites (particularly those behind CDN/WAF providers like WPX Cloud, LiteSpeed, Cloudflare) block requests that lack standard browser headers. The scraper was sending requests with no
User-Agentand noReferer, causing the homepage to return200while all child pages returned403— resulting in a crawl of only 1 page.Fix: Set a realistic
User-AgentandReferer(base URL) on the session at initialization.Problem 2: Broken cache state causes silent re-run failures
If a scraping run was interrupted or failed after a URL was marked
visitedbut before its page content was stored, re-running the scraper would silently exit with no output. The progress bar would show1/1complete, but the exported files would be empty. There was no way to detect or recover from this without manually deleting the.sqlitecache file.Fix: Added
reset_if_no_content(url)toDatabaseManager, called on the starting URL before the crawl loop. It resetsvisited = FALSEfor any URL that has no corresponding page content, allowing the scraper to automatically recover on the next run without user intervention and without affecting successfully scraped pages.Files changed:
crawler_to_md/scraper.py— browser headers on session init; callreset_if_no_contenton startcrawler_to_md/database_manager.py— newreset_if_no_content(url)method