Show and tell: LinkedIn Jobs Scraper built with Crawlee — 45K results, no auth

[MarcinDudekDev]

Hey Crawlee folks 👋

Wanted to share a scraper I built on top of Crawlee that's been holding up surprisingly well: a LinkedIn Jobs search extractor that pulls full job listings at scale without requiring a LinkedIn API key or logged-in session.

What it does
Given a search query + location, it walks LinkedIn's public jobs search and returns structured results: title, company, location, posted date, description, apply URL, seniority, employment type. I've pulled ~45K results in a single run without tripping the usual guest-session walls.

How it's built

• CheerioCrawler for the listing pages (fast, no browser needed for the guest search endpoint)
• PuppeteerCrawler fallback for detail pages that gate behind JS
• Crawlee's built-in RequestQueue for pagination through the offset-based result pages
• proxyConfiguration with rotating residential proxies + exponential backoff on 429s
• useSessionPool: true — sessions get retired fast when LinkedIn starts shadow-blocking
• Deployed as an Apify actor so the queue, proxy, and storage are all managed

Crawlee's session pool + request retry logic is what makes this viable — hand-rolling that retry state machine would be painful.

Use cases

• Job board aggregators
• Market research on hiring trends by region/role
• Recruiter tooling (sourcing pipelines)
• Salary/skill demand datasets

Trigger via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~linkedin-jobs-scraper/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "searchQuery": "python developer",
    "location": "Berlin",
    "maxResults": 500
  }'

Link: https://apify.com/cryptosignals/linkedin-jobs-scraper

Would love feedback on:

• Edge cases where LinkedIn's guest search breaks (geo-specific quirks?)
• Session pool tuning — currently retiring after 2 soft-blocks, might be too aggressive
• Anyone else built job-site scrapers on Crawlee and want to compare notes on anti-bot patterns?

Happy to dig into any part of the implementation if it's useful.