`Auth.Client.ScopeManager` grows unbounded per registry

[ridhoq]

What happened in your code base?

Auth.Client retains a per-registry ScopeManager whose internal SortedSet<Scope> is append-only. Every IRepository operation that triggers a 401 challenge calls ScopeManager.SetScopeForRegistry(...), which merges the challenged scope into the per-registry set via ConcurrentDictionary.AddOrUpdate (ScopeManager.cs#L108–L115). There is no eviction, no public Reset/Clear, and the ScopeManager property on Auth.Client has no setter, so callers cannot reset accumulated state on a long-lived client.

On the next 401 challenge, Client.SendCoreAsync requests an OAuth2 token covering all accumulated scopes via repeated scope= query parameters (Client.cs token-fetch path). The auth service returns a JWT whose access claim mirrors every granted scope. The JWT therefore grows linearly with the count of distinct (repository, actionSet) tuples touched by that Auth.Client instance.

For long-lived Auth.Client instances that touch many distinct repositories of the same registry, the base64-encoded bearer token plus the "Bearer " prefix eventually exceeds the per-header size limit of some HTTP component on the path — typically a reverse proxy or gateway in front of the registry, but the same shape applies to any intermediary or server with a bounded request-header buffer. Once the limit is exceeded, the request is rejected pre-parse with HTTP/1.1 400 Bad Request and an empty body — the registry data plane never sees the request. Client.SendCoreAsync only retries on 401, never 400, so the failure surfaces as a per-request error even though the root cause is cumulative client state.

The exact threshold is environment-specific (depends on the front-end's configured header-buffer size and on average repository-name length), but is reachable by any consumer that issues operations across a few hundred distinct repositories on a single Auth.Client. I reproduced this against Azure Container Registry; I have not tested against other registries, but the failure mode is generic enough that other registries with a similarly tuned front-end are likely to reproduce it as well.

For an ACR user writing an oras-dotnet application, this failure mode is essentially undiagnosable from the client side, and I think that is a major reason this bug has not been reported sooner. The symptom is a 400 BadRequest with no WWW-Authenticate, no JSON error body, no Docker-distribution error code, and no callable API on Auth.Client to inspect the current scope-set size — and no obvious connection between "I touched a lot of repositories on this client" and "individual requests started returning 400". The registry side shows no log entry either; the request was rejected by a front-end before the data plane ever saw it. I was only able to root-cause this because I have access to internal ACR service telemetry that let me correlate the client-side 400s with what the registry data plane was (or was not) seeing. A typical ACR user does not have that affordance, so for them this bug manifests as an unsolvable wall.

How we worked around it

In our own code we worked around the bug by constructing a short-lived Auth.Client per logical operation, while continuing to share the HttpClient, CredentialProvider, and token ICache across all of those short-lived clients (so we keep connection pooling and the token cache intact). Each fresh Auth.Client starts with an empty ScopeManager, so the JWT never grows past the scopes needed for a single operation. This works but is awkward: Auth.Client does not expose a first-class "share supporting infra, fresh scope state" construction path, so the workaround ends up reaching into shapes the API does not really intend to expose. Proposal (4) below codifies this as a supported pattern.

Why the accumulating-scope design exists

I want to call this out explicitly because it is load-bearing for the proposals below: the current "accumulate every scope ever challenged on this registry" behavior is not accidental — it is a steady-state optimization for the token ICache. The cache is keyed by the set of granted scopes, so a token minted for {A, B, C} satisfies any future request whose required scope is a subset of {A, B, C}. Requesting all accumulated scopes on each fresh mint means a workload that touches a bounded set of N distinct repositories converges to ~N token mints total (one per first-encounter of a new repo), regardless of how many manifest/blob operations are issued. Single-scope-per-mint would defeat the cache and cost one mint per operation.

That optimization is correct and valuable for the workloads it was designed for (build pipelines, replication, mirroring across a small bounded repo set). It backfires only for fan-out workloads — batch tooling, scanners, long-lived daemons — that touch an unbounded set of distinct repositories on a singleton client. The fix space below tries to preserve the steady-state optimization for the former while keeping the latter on the road.

What did you expect to happen?

I would expect one or more of the following so that Auth.Client is safe to use as a long-lived client against a registry, regardless of the front-end's request-header size limit:

1. (highest leverage) Bound ScopeManager by default with LRU eviction. A configurable cap on retained scopes per registry — e.g. default of 64, settable via ClientOptions or a constructor parameter — would close the failure mode for everyone while preserving the steady-state token-cache-reuse optimization that motivates the current "accumulate everything" design. Workloads that touch fewer than the cap continue to see one-mint-per-registry behavior after warmup. Pathological fan-out workloads pay one extra 401 + mint per evicted scope on next use, which is graceful degradation rather than a hard wire-level failure. Existing behavior can be preserved by setting the cap to int.MaxValue.

2. Public Auth.Client.ResetScopes(string? registry = null). An explicit escape hatch for callers who know their own operation boundaries (batch tooling, long-lived daemons, scanners) and want to clear scopes for a registry — or all registries — without dropping the underlying HttpClient, CredentialProvider, or token ICache. Useful in conjunction with (1) for callers who want hard isolation rather than LRU heuristics.

3. Documentation on Auth.Client / ScopeManager. Call out the growth, the request-header-size failure mode it can produce against intermediaries, and — for consumers stuck on a pre-fix version — the workaround of constructing a short-lived Auth.Client per operation while sharing the HttpClient, CredentialProvider, and ICache to preserve connection pooling and token reuse. The current XML docs on ScopeManager.SetScopeForRegistry describe the merge semantics but do not mention that the set is unbounded or that it has a wire-level consequence.

4. (optional follow-up) Auth.Client.CreateChild() or an Auth.ClientFactory. A first-class way to construct a derived Auth.Client that shares the parent's HttpClient + CredentialProvider + ICache but starts with a fresh ScopeManager. This codifies the "short-lived per-operation client" workaround as a supported pattern. Less critical if (1) lands.

I think (1) + (3) is the minimum that fixes the bug for all consumers by default. (2) and (4) are nice-to-haves for callers with explicit control needs.

How can we reproduce it?

Minimal reproducer against an Azure Container Registry (replace with any OCI registry — the bug is not vendor-specific, any registry whose front-end caps request-header size is reachable):

using OrasProject.Oras.Registry;
using OrasProject.Oras.Registry.Remote;
using OrasProject.Oras.Registry.Remote.Auth;

var httpClient = new HttpClient();
var authClient = new Client(
    httpClient: httpClient,
    credentialProvider: /* your ICredentialProvider for myregistry.azurecr.io */ null);

const string registryHost = "myregistry.azurecr.io";
var registry = new Registry(registryHost) { Client = authClient };

for (int i = 0; i < 300; i++)
{
    var repoName = $"test/sample-repo-{i:D4}";
    var repo = await registry.GetRepositoryAsync(repoName, default);

    try
    {
        // Any operation that triggers a 401 challenge works here.
        // ResolveAsync against a non-existent tag is sufficient; we only need the challenge.
        _ = await repo.Manifests.ResolveAsync("latest", default);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"[i={i}] repo={repoName} ex={ex.GetType().Name}: {ex.Message}");
        // After enough distinct repos have been touched on this Auth.Client, you will start to
        // see HttpRequestException wrapping a 400 BadRequest with an empty body. The exact
        // threshold depends on the registry front-end's header-buffer size and on repo-name
        // length. Before that point you will see expected NotFound/Unauthorized from the
        // registry itself.
        break;
    }
}

Observable symptoms once the threshold is crossed:

• HttpRequestException (or ResponseException) wrapping 400 BadRequest.
• Empty response body, no WWW-Authenticate header on the response, no Docker-* headers.
• The same request issued from a fresh Auth.Client succeeds (or fails with the expected registry-level error, e.g. NotFound).

A side-by-side comparison run that constructs a fresh Auth.Client per loop iteration completes all 300 iterations without the cliff — the bearer-token length stays flat because the ScopeManager is empty on each call.

If you want a length probe instead of waiting for the wire-level failure, you can subclass HttpClientHandler and log request.Headers.Authorization!.Parameter.Length on each outbound request — it grows monotonically per repo encountered on the singleton-client path.

What is the version or commit of the ORAS .NET library?

Reproduced against v0.5.0. The relevant code (Auth/ScopeManager.cs::SetScopeForRegistry) is unchanged on main as of writing — the growth path is identical.

What is the version or commit of the ORAS .NET library?

Reproduced against v0.5.0. The relevant code (Auth/ScopeManager.cs::SetScopeForRegistry) is unchanged on main as of writing — the growth path is identical.

What are your OS and Runtime environments?

Linux containers, .NET 10.0.

Are you willing to submit PRs to fix it?

• Yes, I am willing to fix it.