Skip to content

Add multi-proxy support & granular proxy selection#160

Open
CorentinB wants to merge 2 commits intomasterfrom
feat/more-proxy-stuff
Open

Add multi-proxy support & granular proxy selection#160
CorentinB wants to merge 2 commits intomasterfrom
feat/more-proxy-stuff

Conversation

@CorentinB
Copy link
Copy Markdown
Collaborator

Implements a granular multi-proxy system with fine-grained control over proxy selection.

Features

  • Multi-proxy support: Configure multiple proxies with automatic round-robin selection
  • Network filtering: Filter proxies by IPv4/IPv6 support via ProxyNetwork enum
  • Type classification: Categorize proxies as Mobile/Residential/Datacenter via ProxyType enum
  • Domain routing: Route specific domains to specific proxies using glob patterns
  • Per-proxy statistics: Track RequestCount, ErrorCount, and LastUsed per proxy
  • Context-based selection: External callers can request specific proxy types via context

Example: Context-based proxy type selection

// Configure proxies with different types
httpClient, _ := warc.NewWARCWritingHTTPClient(warc.HTTPClientSettings{
    Proxies: []warc.ProxyConfig{
        {
            URL:     "socks5://mobile-proxy:1080",
            Network: warc.ProxyNetworkAny,
            Type:    warc.ProxyTypeMobile,
        },
        {
            URL:     "socks5://datacenter-proxy:1080",
            Network: warc.ProxyNetworkAny,
            Type:    warc.ProxyTypeAny,
        },
    },
})

// Request specific proxy type via context
ctx := warc.WithProxyType(req.Context(), warc.ProxyTypeMobile)
req = req.WithContext(ctx)
resp, _ := httpClient.Do(req)

Breaking change

Replaces Proxy string field with Proxies []ProxyConfig in HTTPClientSettings.

- Support multiple proxies with round-robin selection
- Add ProxyNetwork enum (IPv4/IPv6 filtering)
- Add ProxyType enum (Mobile/Residential/Datacenter)
- Add per-domain routing with glob patterns
- Add per-proxy statistics (RequestCount, ErrorCount, LastUsed)
- Add context-based proxy type selection
- Breaking change: replace Proxy string with Proxies []ProxyConfig
@CorentinB CorentinB added the enhancement New feature or request label Nov 20, 2025
@CorentinB CorentinB self-assigned this Nov 20, 2025
Comment thread client.go Outdated
Comment thread client.go
Comment on lines +16 to +28
// ProxyNetwork defines the network layer (IPv4/IPv6) a proxy can support
type ProxyNetwork int

const (
// ProxyNetworkUnset is the zero value and must not be used - forces explicit selection
ProxyNetworkUnset ProxyNetwork = iota
// ProxyNetworkAny means the proxy can be used for both IPv4 and IPv6 connections
ProxyNetworkAny
// ProxyNetworkIPv4 means the proxy should only be used for IPv4 connections
ProxyNetworkIPv4
// ProxyNetworkIPv6 means the proxy should only be used for IPv6 connections
ProxyNetworkIPv6
)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love to see the golang-style enum pattern being used here!

Comment thread dialer.go
Comment on lines +161 to +163
if proxyConfig.URL == "" {
continue
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why continue without error if URL is blank?

Comment thread client.go
Comment on lines +30 to +42
// ProxyType defines the infrastructure type of a proxy
type ProxyType int

const (
// ProxyTypeAny means the proxy can be used for any type of request
ProxyTypeAny ProxyType = iota
// ProxyTypeMobile means the proxy uses mobile network infrastructure
ProxyTypeMobile
// ProxyTypeResidential means the proxy uses residential IP addresses
ProxyTypeResidential
// ProxyTypeDatacenter means the proxy uses datacenter infrastructure
ProxyTypeDatacenter
)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, it seems that ProxyTypeAny represents something different than the other ProxyTypes because it describes the traffic's preference, where as the other types describe the kind of infrastructure the proxy uses
I think part of my confusion is that I understand you might want to send a URL to a proxy with residential IPs, or you might want to send a URL to a proxy with datacenter IPs, but ProxyTypeAny isn't a kind of proxy but is actually a decision that you don't care which proxy type you send the data to
So I'm confused under what condition you would define a proxy as ProxyTypeAny?

@willmhowes
Copy link
Copy Markdown
Collaborator

I didn't finish submitting my review back when this was fresh, so concluded my thoughts and pushed the review today. That being said, perhaps this issue should be closed given the discussion in #186, specifically #186 (comment). @NGTmeaty ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants