Add apify-blocked-scrape-triage skill

[mikhail-koviazin]

Skill

• Name: apify-blocked-scrape-triage
• What it does: Works out why a scrape came back blocked or empty, and fixes it in cost order, so that a 403 doesn't turn straight into residential proxies plus a browser.
• Author: Mikhail Koviazin (@mikhail-koviazin)

Checklist

• PR adds one skill in skills/apify-<name>/ and changes nothing outside that directory
• SKILL.md frontmatter has name (matches the folder, apify- prefix), description (≤ 1024 chars) and metadata.keywords
• I read CONTRIBUTING.md (agents: agents/AGENTS.md)
• Tested the skill end-to-end at least once

Notes

Why I wrote this one. The skills here answer "how do I get this data", and they do that well. None of them answers "the run came back with nothing, now what", and that is where most of my scraping time actually goes. I went and checked apify/agent-skills as well, and apify-ultimate-scraper doesn't mention 403, 429, challenges, proxies or fingerprints anywhere either. So an agent that hits a block has nothing to go on, and it reaches for the most expensive thing available.

What it argues against. The reflex answer to a 403 is residential proxies and a browser. That is one to two orders of magnitude more per page, and half the time it is not even the problem. This skill makes you classify first and then escalate in cost order: rate and IP spread, request fidelity, session tokens, the site's own internal API, browser last.

The diagnostic was blind on the one case it exists for, and that is fixed here. Crawlee treats 401, 403 and 429 as blocking and throws before your page function runs, so the item lands in the dataset carrying none of your fields: no status, no headers, nothing to classify with. Six identical requests to a challenged site, three of them back as empty error items, and no way to tell what they were. A one-line hook in preNavigationHooks retires the blocked session but stops the throw, and the same six came back six for six with status and headers on them. The flag people reach for first, throwHttpErrors, changes nothing here, and dropping the retire call unblinds the run while letting nothing through. Browser Actors have the identical problem.

A 403 is two different situations, and the header that separates them belongs to one vendor only. A hard block and a managed challenge arrive as the same status. On Cloudflare the challenge carries cf-mitigated and an interstitial body, the hard block carries neither, and they route in opposite directions: one needs a different address, the other needs a better client and a few seconds. Without the fix above you cannot see either header from inside the Actor, which is why the two were one row in my earlier draft. The trap I walked into next was treating that header as universal. It cannot appear on any other edge, so on an Akamai 403 its absence is guaranteed and carries no information, while the rule that reads it happily returns "your address was judged" for every non-Cloudflare refusal and is right only by luck. The skill now names the edge first, from server and cf-ray, or from ak_p and X-Reference-Error, and applies that vendor's discriminator rather than one vendor's header everywhere.

A refusal is a measurement, not a property of the site. That same Akamai 403 arrived with Server-Timing: cdn-cache; desc=HIT on a response whose own Cache-Control said max-age=0, so the error object came out of the edge cache and never reached the origin. Thirty-eight minutes later the same URL from the same address answered 200 with 81 KB, three times running, with nothing changed on my side. Firewall rules by ASN do sit there for years, so this is not a promise that refusals dissolve; it is an argument about cost. One repeat request either turns a single observation into a fact or stops you buying proxies against a refusal that had already expired.

The cheapest step in the whole document is the one nobody takes: look at your own egress before you look at the site. One free request gives you the address, the network and whether it is a hosting range, and that single fact explains most "this site blocks me" reports. Skip it and the reflex on a 403 from a US county site is to buy residential proxies, which is the purchase this skill exists to prevent. The same goes for DNS: on a machine running a proxy client in fake-ip mode every UDP lookup answers out of 198.18.0.0/15 whichever resolver you address, so a reachability check that trusts DNS will report live hosts as dead. I had that exact setup and did not know until a clean host disagreed with me.

Two egresses of the same class disagreeing is a diagnosis, not noise. One county refused my hosting address three different ways on three of its hosts, a silent timeout, a CDN block and a 399-byte Access Denied, and answered a datacenter proxy with 200 on the same hosts minutes later. That means the rule is about my specific address rather than about datacenter ranges, and the fix is another datacenter egress I already have rather than residential traffic I would have to buy.

Escalating to a browser made it worse, measured. Same URL, same proxy pool, one minute apart: the browser Actor was answered with a hard block page while the plain HTTP Actor got a challenge twice and a clean 200 on the third request. A Crawlee browser Actor also aborts on the challenge's 403 before the page can clear, so it never reaches the point where a challenge solves itself; a desktop Chrome on the same platform sat through "Verifying you are human" and loaded the page. Three requests cost 0.0011 compute units through the HTTP Actor and 0.0131 through the browser one.

I tested it. Every worked example in the skill is a measurement rather than something I remembered, and this is the one the ordering came from. Same URL, a large US county assessor site behind a CDN, four requests:

Egress	Client	Result
hosting provider, non-US	default HTTP client	403, CDN error page
same IP	live desktop Chrome	403, identical
Apify Proxy, datacenter	apify/cheerio-scraper	200, 92 KB, real page, retryCount 0
Apify Proxy, residential, US	apify/cheerio-scraper	200, same byte length

So the site is not refusing automation: the request that got through ran no JavaScript and was not a browser. It is not refusing datacenter addresses either, because the ones that worked came from a datacenter. And residential, the reflex, gave me the same page byte for byte and cost more. That experiment is the reason the skill leads with classification instead of escalation.

The same run gave me one of the gotchas for free. It returned a 200 titled Page not found, with the full site navigation on it. A soft 404 looks like success in the status, in the headers and in the size, only the content gives it away, so a crawler counts it as a page and a diff calls it a change instead of a dead link.

What I checked instead of assuming. Actor ids come from the Store API. Every input field named in references/actor-index.md was read from that Actor's own input schema. The proxyRotation part comes from the enum and its documented behaviour; the bit I added is which value you actually want when the flow carries state, because that is the one people set backwards.

What I left out on purpose, and the skill says so out loud: antidetect browsers, account warming, CAPTCHA solving. TLS level impersonation is named as the right class of tool when the evidence points below the header layer, but it is not routed here, since those aren't Actors. The skill also opens with when not to use it at all: login walls, paywalls, and sources that publish a real API.

[mikhail-koviazin]

Rebased onto current main and adapted to the rules that came in with #85. The hand-written .claude-plugin/marketplace.json entry is out of the diff, and its keywords and category now sit in the SKILL.md frontmatter, so this PR touches nothing outside skills/apify-blocked-scrape-triage/. That also settles the recurring conflict on the shared catalog file rather than postponing it.

I updated the checklist above to the current template, since two of its boxes described the old marketplace.json flow. Everything below it is unchanged.

Checks run locally against the branch, all green:

bash scripts/lint_telemetry.sh skills/apify-blocked-scrape-triage
uv run scripts/lint_references.py skills/apify-blocked-scrape-triage --check-actors skills/apify-blocked-scrape-triage
uv run scripts/generate_agents.py