Multiple Client Support Per Manager

[plutopulp]

Multiple Client Support Per Manager

Problem

Users cannot easily use different HTTP clients (with different auth headers, timeouts, or other configuration) for different downloads within a single DownloadManager instance.

Current Limitation

The manager accepts a single client that all workers share:

async with DownloadManager(
    client=my_auth_client,  # ← All downloads use THIS client
    download_dir=Path("./downloads")
) as manager:
    await manager.add_to_queue([
        FileConfig(url="https://api.example.com/private/file1.zip"),  # uses auth client
        FileConfig(url="https://public.example.com/file2.zip"),       # also uses auth client (unnecessary)
    ])

Use Cases

1. Mixed public/authenticated downloads - Some files require auth headers, others don't
2. Per-host timeouts - Different servers need different timeout configurations
3. Rate limiting - Different API keys/clients for different rate limit buckets
4. Proxy routing - Some downloads via proxy, others direct

Current Workaround

Create separate managers for each client type:

async with DownloadManager(client=auth_client) as auth_manager:
    async with DownloadManager(client=public_client) as public_manager:
        await auth_manager.add_to_queue(auth_files)
        await public_manager.add_to_queue(public_files)
        await asyncio.gather(
            auth_manager.queue.join(),
            public_manager.queue.join()
        )

Limitations:

• Duplicates worker pools, trackers
• Separate queue for each pool
• More resource overhead

Possible Solution

Option 1: Per-FileConfig Client Selection

Add client selection to FileConfig:

manager = DownloadManager(
    clients={
        "auth": auth_client,
        "public": public_client,
    },
    default_client="public",
    ...
)

files = [
    FileConfig(url="https://api.example.com/file1.zip", client_key="auth"),
    FileConfig(url="https://public.example.com/file2.zip"),  # uses default
]

Option 2: Client Resolver Callback

Allow users to provide a function that selects client based on FileConfig:

def resolve_client(file_config: FileConfig) -> ClientSession:
    if "api.example.com" in file_config.url:
        return auth_client
    return public_client

manager = DownloadManager(
    client_resolver=resolve_client,
    ...
)

Option 3: Think of something better :p