Skip to content

Use bounded concurrent collector fetches - #824

Merged
chubes4 merged 6 commits into
mainfrom
feat/814-concurrent-collector
Aug 3, 2026
Merged

Use bounded concurrent collector fetches#824
chubes4 merged 6 commits into
mainfrom
feat/814-concurrent-collector

Conversation

@chubes4

@chubes4 chubes4 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Fixes #814

Depends on #822 and is intentionally stacked on feat/820-concurrent-http-transport. Merge #822 first.

Summary

  • Integrates Bound URL fetch retry pacing #818 retry-only request pacing commits (92163bc4, ccef5e2b).
  • Acquires admitted initial page and asset queues through Static_Site_Importer_URL_Fetcher::fetch_many() with a global limit of 4 and per-origin limit of 2.
  • Commits responses through the existing queue order, preserving deterministic artifacts, hashes, diagnostics, limits, and injected-fetcher/cache behavior.
  • Propagates collector transport deadlines and cancellation through the bounded transport path.

Evidence

  • php tests/smoke-url-site-collector.php passed 53 assertions, including bounded global/per-origin in-flight acquisition, deliberately out-of-order completions, serial-equivalent snapshot hashes/diagnostics, and a transport-backed wall-clock reduction against serial delays.
  • php tests/smoke-url-batch-import.php passed.
  • php tests/smoke-url-concurrent-fetcher.php passed.
  • PHP syntax checks and git diff --check passed.
  • Homeboy lint passed with zero findings: run ecdffaff-414c-4395-a73b-09fb7c9c32df.
  • Homeboy test passed 12/12: run 97b18760-8659-4f80-bb05-37a1630d4a73.

AI Assistance

OpenAI gpt-5.6-sol via OpenCode integrated retry pacing, implemented deterministic bounded collector acquisition, and added transport-backed tests. Chris Huber reviewed and remains responsible for every line.

chubes4 added 5 commits August 3, 2026 10:41
OpenAI gpt-5.6-sol via OpenCode implemented the bounded concurrent validated-IP transport and deterministic tests. Chris Huber reviewed and remains responsible for every line.
OpenAI gpt-5.6-sol via OpenCode restored focused WordPress formatting and verified the concurrent URL transport. Chris Huber reviewed and remains responsible for every line.
OpenAI gpt-5.6-sol via OpenCode was used to trace the collector scheduling path and draft the implementation and deterministic tests. Chris Huber reviewed and remains responsible for the change.
OpenAI gpt-5.6-sol via OpenCode was used to repair scoped WordPress formatting and PHPStan findings. Chris Huber reviewed and remains responsible for the change.
OpenAI gpt-5.6-sol via OpenCode was used to integrate retry pacing, implement deterministic bounded collector acquisition, and add transport-backed tests. Chris Huber reviewed and remains responsible for every line.
@chubes4
chubes4 changed the base branch from feat/820-concurrent-http-transport to main August 3, 2026 16:51
Resolve collector integration conflicts after the concurrent transport and retry pacing squash merges.

AI assistance: OpenAI gpt-5.6-sol via OpenCode reconciled the merge, resolved conflicts, and ran validation. Chris Huber remains responsible for every line.
@chubes4
chubes4 merged commit d0f8fe3 into main Aug 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace serial delayed URL fetching with bounded per-origin scheduling

1 participant