Skip to content

Bound URL fetch retry pacing - #818

Merged
chubes4 merged 2 commits into
mainfrom
perf/814-bounded-fetch-scheduling
Aug 3, 2026
Merged

Bound URL fetch retry pacing#818
chubes4 merged 2 commits into
mainfrom
perf/814-bounded-fetch-scheduling

Conversation

@chubes4

@chubes4 chubes4 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Refs #814

Current Scope

  • Removes the unconditional 100 ms post-fetch sleep; successful uncached requests are immediately eligible by default.
  • Retains bounded retries and applies explicit request_delay_ms pacing only between retry attempts.
  • Records the blocking transport's effective same-origin and cross-origin concurrency limits (1 each) in collection metadata.

Evidence

  • php tests/smoke-url-site-collector.php passed 50 assertions, including deterministic clock-driven default throughput, retry pacing, cache-bypass, and canonical snapshot checks.
  • php tests/smoke-url-batch-import.php passed.
  • homeboy review test static-site-importer --extension wordpress --changed-since origin/main --placement local passed 12/12 WordPress tests: run 1ac1306a-6441-481e-b3b4-4f161de168fa.
  • homeboy review lint static-site-importer --extension wordpress --changed-since origin/main --placement local passed with zero findings: run 8001f26a-7079-48d3-88bb-35ac683bfb6c.
  • npm run test:inventory, PHP syntax checks, and git diff --check passed.

Follow-up Required For #814

The hardened URL transport is synchronous, per-IP socket I/O. Actual concurrent acquisition requires an owning transport primitive that preserves validated-IP connection pinning, TLS/SNI verification, response caps, per-hop redirect validation, retry/cache behavior, and deterministic completion-independent collection. This PR deliberately does not claim that concurrency criterion is complete.

Compatibility

  • Existing fetcher, SSR validation, redirect handling, retry count, response/cache counters, and canonical artifact construction remain on their existing paths.
  • request_delay_ms remains accepted but now paces only retries; its omitted default is 0.

AI Assistance

OpenAI gpt-5.6-sol via OpenCode traced the collector scheduling path, drafted the implementation and deterministic tests, and repaired the scoped lint findings. Chris Huber reviewed and remains responsible for every line.

OpenAI gpt-5.6-sol via OpenCode was used to trace the collector scheduling path and draft the implementation and deterministic tests. Chris Huber reviewed and remains responsible for the change.
OpenAI gpt-5.6-sol via OpenCode was used to repair scoped WordPress formatting and PHPStan findings. Chris Huber reviewed and remains responsible for the change.
@chubes4
chubes4 merged commit 3380526 into main Aug 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant