[Bug] Customer ID Header Incorrectly Sent in Refit-Based Microservices Communication

[Moghaddm]

Hi guys!

We’ve built a production microservices-based software recently released for customers. The software is architected as SaaS and follows a layered design approach for resolving customer identifiers:

BFF Layer: Customer identifier is resolved via claims in tokens.
Back-End Layer: Customer identifier is resolved via headers added to the HTTP request for internal service calls.

We use Refit for our HTTP client communication between services and rely on custom handlers to append the customer ID header based on an abstract ICustomerResolver.

However, we have encountered a critical issue in production (and sometimes stage environments):

Issue: Occasionally, the CustomerId header is either not sent or, worse, contains the customer ID of a different request/customer.
For example:
A request from Customer ID 1004 may end up sending 1002 (a different customer ID).
This mismatch results in a potential data leak between customers.

Analysis

We suspect the issue is related to how Refit handles HTTP requests through custom message handlers. Specifically, the problem seems to stem from the CustomerIdHeaderHandler implementation or the way handlers are interacting with Refit's request lifecycle.
Code Snippets for Reference
Configuration for Services

Here's the configuration snippet used in our services for setting up the Refit clients:

/// <summary>
/// Configures and sets up a proxy client for HTTP communication with retry and circuit breaker policies,
/// adding authentication headers, metrics handlers, and base configurations.
/// </summary>
public static void SetupProxyClient<TProxy>(this IServiceCollection services, ClientUri clientUri)
    where TProxy : class, IProxy
{
    var serviceName = EnvironmentVariableHelpers.GetServiceName();
    var environmentName = EnvironmentVariableHelpers.GetEnvironmentName();
    services.AddTransient<ProxyMetricsHandler>();
    services.AddTransient<ProxyLoggingHandler>();
    services.AddTransient<CustomerIdHeaderHandler>();
    services.AddRefitClient<TProxy>(new RefitSettings
        {
            ContentSerializer = new CustomContentSerializer(),
        })
        .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
        {
            KeepAlivePingDelay = TimeSpan.FromMinutes(1),
            KeepAlivePingTimeout = TimeSpan.FromSeconds(30),
            PooledConnectionLifetime = TimeSpan.FromMinutes(10),
            PooledConnectionIdleTimeout = TimeSpan.FromMinutes(2),
            MaxConnectionsPerServer = 1024,
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
            EnableMultipleHttp2Connections = true
        })
        .ConfigureHttpClient((_, c) =>
        {
            c.BaseAddress = new Uri(clientUri.CalculateUri());
            c.DefaultRequestVersion = HttpVersion.Version20;
            c.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrHigher;
            c.Timeout = TimeSpan.FromSeconds(300);
            c.DefaultRequestHeaders.Add(HttpRequestHeaderConstants.FromService, serviceName);
            c.DefaultRequestHeaders.Add(HttpRequestHeaderConstants.FromServiceEnvironment, environmentName);
            c.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
        })
        .AddPolicyHandler((_, _) => Policy<HttpResponseMessage>
            .HandleResult(CheckCanRetry)
            .WaitAndRetryAsync(
                HttpClientConstants.RetryCount,
                retryAttempt =>
                    TimeSpan.FromMilliseconds(HttpClientConstants.BaseRetryDelayInMillisecond * retryAttempt))
        )
        .AddPolicyHandler((_, _) => GetCircuitPolicy())
        .AddHttpMessageHandler<CustomerIdHeaderHandler>()
        .AddHttpMessageHandler<ProxyLoggingHandler>()
        .AddHttpMessageHandler<ProxyMetricsHandler>();
}

Customer ID Header Handler Implementation
Here’s the implementation of the problematic CustomerIdHeaderHandler:

internal class CustomerIdHeaderHandler : DelegatingHandler
{
    private readonly ICustomerResolver _customerResolver;

    public CustomerIdHeaderHandler(ICustomerResolver customerResolver)
    {
        _customerResolver = customerResolver;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var customerId = _customerResolver.CustomerId;
        if (customerId is not null)
            request.Headers.Add(HttpRequestHeaderConstants.CustomerId, customerId.ToString());

        return await base.SendAsync(request, cancellationToken);
    }
}

Observed Issue

• The handler appears to retrieve incorrect customer IDs under some conditions.
• This may be due to shared state or improper resolution in the ICustomerResolver.
• There's potential for CustomerId leakage between concurrent HTTP requests via Refit.
• The issue is intermittent, making it harder to diagnose.

Questions

• Could this bug stem from improper usage of Refit in combination with custom handlers (DelegatingHandler)?
• Does Refit cause unintended concurrency issues when resolving headers, especially with custom implementations like ICustomerResolver?
• Is there a race condition in CustomerIdHeaderHandler or incorrect lifecycle handling?

Recommended Actions

• Conduct a thorough review of how Refit initializes handlers.
• Validate how ICustomerResolver behaves across threads (ensure it’s not introducing shared state issues).
• Debug handlers in stage/production environments to identify where mismatches occur.
• Test with higher load to check if Refit's concurrency model contributes to the issue.

Reproduction repository

https://github.com/reactiveui/refit

Expected behavior

The CustomerIdHeaderHandler should correctly append the appropriate CustomerId to the HTTP request headers for every outgoing request:

• Each HTTP request should include the CustomerId corresponding to the specific context of the client/request.
• No cross-request or cross-customer leakage of CustomerId data should occur:
• For Request A (Customer ID: 1004), the header must contain CustomerId: 1004.
• For Request B (Customer ID: 1002), the header must contain CustomerId: 1002.

In any concurrent or high-load scenario:

• CustomerId headers should always match the request-specific context.
• No CustomerId should be skipped, missing, or incorrectly carried over from other requests.

Screenshots 🖼️

No response

IDE

Rider MacOS

Operating system

MacOS

Version

8.0.0

Device

No response

Refit Version

8.0.0

Additional information ℹ️

No response