Improve parallelization ramping, error reporting, and dynamic scaling

[hemanth-asirvatham]

Motivation

• Clarify ramp-up messaging and lower the initial ramp fraction so the worker pool ramps from a smaller starting concurrency by default.
• Ensure the user is informed when ramping is halted and why, and that the first connection and rate-limit errors are surfaced to the user rather than only logged.
• Include rate-limit and connection error counts in periodic status updates and make dynamic scaling more robust in the presence of transient errors.

Description

• Lowered the default ramp_up_start_fraction from 0.25 to 0.15 to start ramping from a smaller concurrency.
• Added rate_limit_errors_since_last_status tracking and include rate_limit_errors and connection_errors in the periodic status line as x/processed with per‑interval deltas.
• Surface first rate-limit and first connection errors to the user via print (when not quiet) in _log_rate_limit_once and _log_connection_once, and clarified the ramp log to say "parallel threads".
• Added worker_manager and ramp_monitor tasks plus a managed workers list so ramp-up can actually spawn workers over time and emit a "Ramp-up complete" status; changed scaling logic to require quiet windows for both recent rate-limit and connection errors before scaling up and ensured successes/resets are handled on errors.

Testing

• Ran python -m pytest tests/test_parallelization_tuning.py tests/test_timeout_logic.py and all tests passed.
• Test result: 12 passed (run completed successfully).

---

Codex Task

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}