Add per-domain concurrency, retry backoff, internal stats, and stability improvements #557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

decompi wants to merge 1 commit into thomasdondorf:master from decompi:feature/per-domain-concurrency

decompi commented Mar 25, 2025

Overview

This PR improves puppeteer-cluster by adding:

✅ Per-domain concurrency limits via sameDomainConcurrency option.
🔁 Exponential backoff for retries based on retryDelay.
📊 A new getStats() method to expose internal cluster metrics.
🔄 Graceful shutdown logic using idle() in close().
✅ Improved TypeScript typings across internal modules.

Why?

These changes enhance stability and performance in production environments with many dynamic or domain-heavy scraping targets.

Test Plan

Manual testing with ~100 jobs across multiple domains
Verified retry logic & backoff behavior
Confirmed getStats() returns correct runtime metrics
Graceful shutdown confirmed after tasks complete


          Add per-domain concurrency, backoff retries, internal stats, graceful…

… shutdown, and improved typings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet