Skip to content

feat: shared HTTP client pool with per-host budgets - #732

Closed
Killerjunior wants to merge 1 commit into
Stellabill:mainfrom
Killerjunior:feat/http-client-pool
Closed

Killerjunior wants to merge 1 commit into
Stellabill:mainfrom
Killerjunior:feat/http-client-pool

Conversation

@Killerjunior

Copy link
Copy Markdown

Description

Introduces a shared, resilient HTTP client pool in internal/httpx with per-host connection budgets, per-host circuit breakers, and periodic DNS refresh to avoid stale A-record connection storms during failovers.

Changes Included

1. Core HTTP Client Pool (internal/httpx)

  • Per-Host Connection Budgeting: Enforces strict MaxConnsPerHost and MaxIdleConnsPerHost limits per remote target host to prevent noisy neighbors from starving outbound slots.
  • Per-Host Circuit Breaker: Integrates gobreaker state per host to fast-fail requests with ErrCircuitOpen when upstream failure thresholds are breached.
  • DNS Refresh & TTL Support: Uses net.Resolver with a pluggable Clock interface. Automatically re-resolves target hosts when DNSTTL expires (or on every dial when DNSTTL = 0). Recycles idle connections whenever host address changes occur.
  • Prometheus Metrics: Instrument http_client_conn_reuse_ratio gauge labeled by host to monitor connection reuse efficiency.

2. Outbox & Integrations Wiring

  • Outbox System: Wired outbox HTTPPublisher and SlackPublisher through httpx.Pool via PooledHTTPClient and PooledSlackClient.
  • Service Configuration: Updated outbox.NewService to accept optional custom HTTPPool while defaulting to the package-level shared pool.
  • Integrations: Integrated PagerDuty client (internal/integrations/pagerduty) to back outbound event delivery using the shared httpx.Pool.

Verification

  • Added test suite in internal/httpx/client_test.go covering:
    • Connection reuse ratio metric tracking
    • Per-host max connection limits under high concurrency
    • Circuit breaker trips on consecutive transport failures
    • DNS TTL zero refresh on every call vs cached TTL behavior
    • Automatic recycling of idle connections when DNS addresses change
    • Resolver failure fallbacks and malformed address handling

Closes #687

@drips-wave

drips-wave Bot commented Aug 4, 2026

Copy link
Copy Markdown

@Killerjunior Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Killerjunior

Copy link
Copy Markdown
Author

Review please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HTTP client pool with per-host budgets and DNS refresh

2 participants