Skip to content

Partition ScopeManager by partitionId (scope state currently shared across partitions per host) #402

Description

@akashsinghal

Summary

ScopeManager stores requested scopes keyed by registry host only, while the token ICache is keyed by (partitionId, registry). As a result, scope state is not isolated per partition: in a multi-tenant / shared-client deployment (one Client + one ScopeManager serving many partitions against the same registry host), scopes accumulated on behalf of one partition are merged into, and replayed in, the token requests of every other partition for that host.

We should partition the ScopeManager by partitionId the same way Cache already is, so scope state is isolated per (partitionId, registry).

Current behavior

Two separate structures back auth, and only one of them is partitioned:

  1. Token cache (Cache / ICache) — keyed by partition + registry:

    • GetCacheKey(registry, partitionId)"ORAS_AUTH_{partitionId}|{registry}" (src/OrasProject.Oras/Registry/Remote/Auth/Cache.cs).
    • Value is a CacheEntry whose Tokens is a Dictionary<scopeKey, token>.
  2. Scope accumulator (ScopeManager) — keyed by registry only:

    • ConcurrentDictionary<string, SortedSet<Scope>> Scopes keyed by registry (src/OrasProject.Oras/Registry/Remote/Auth/ScopeManager.cs).
    • GetScopesForHost(string registry) / GetScopesStringForHost(string registry) take only registry.
    • SetActionsForRepository / SetScopeForRegistry write under reference.Registry with no partitionId.

The token request path in Client.SendCoreAsync builds the scope key from the unpartitioned accumulator and then looks up the partitioned cache:

scopeKey = string.Join(" ", ScopeManager.GetScopesStringForHost(host));   // unpartitioned, host-wide
token    = Cache.TryGetToken(host, scheme, scopeKey, ..., partitionId);   // partitioned

So partitionId isolates the tokens, but the scopeKey that indexes them is derived from host-wide scope state shared across all partitions.

Impact

For a shared single-client deployment (e.g. a proxy or gateway that makes requests on behalf of many downstream callers to a common upstream registry, including large public registries):

  • Cross-partition scope bleed (isolation): repo scopes requested for partition A are merged into the shared per-host SortedSet<Scope> and then included in token requests issued for partition B against the same host. Tenant request scope sets cross-pollinate even though their tokens are partitioned.
  • Unbounded scope growth → oversized token requests: because the set is host-wide and never scoped down per partition/operation, it grows with every distinct repo touched on that host. The full set is replayed on every token fetch (FetchBearerAuthAsync emits one scope= query param per entry), which can blow past token-endpoint URL/header limits and fail auth.
  • Cache thrash: the host-wide scopeKey changes every time any partition touches a new repo, so the per-partition cache key keeps changing → misses → refetch with an ever-larger scope set. (Note MemoryCache SizeLimit/Size=1 counts the outer CacheEntry as 1 and does not bound the inner Tokens dictionary, which also grows per distinct scopeKey.)

Proposed change

Partition ScopeManager state by partitionId, mirroring Cache.GetCacheKey:

  • Key scope storage by (partitionId, registry) instead of registry alone (e.g. reuse the same {partitionId}|{registry} composite key scheme, or nest partitionId -> registry -> SortedSet<Scope>).
  • Thread partitionId through the scope APIs:
    • GetScopesForHost / GetScopesStringForHost
    • SetActionsForRepository / SetScopeForRegistry (instance + static helpers)
    • the call sites in BlobStore, ManifestStore, Repository, Registry (they already pass Repository.Options.PartitionId to the cache; pass it to the scope manager too)
    • the read/merge in Client.SendCoreAsync (attemptedKey, GetScopesForHost)
  • Keep partitionId == null behaving exactly as today (default partition) for backwards compatibility.

API note

SetActionsForRepository(IClient, Reference, params Scope.Action[]), SetScopeForRegistry(IClient, string, Scope), and the ScopeManager instance methods are public, so adding a partitionId parameter is a public API surface change. Suggest adding optional string? partitionId = null overloads to stay non-breaking.

Out of scope / related

This issue is specifically about partitioning the scope manager. Separately worth considering as follow-ups (not required here):

  • Bounding/evicting per-(partition, registry) scope sets, and/or scoping token requests to the current operation's repository rather than replaying the full accumulated set (closer to oras-go's context-scoped GetAllScopesForHost), to cap token-request size.
  • Bounding the inner CacheEntry.Tokens dictionary growth.

Acceptance criteria

  • Scope state is isolated per (partitionId, registry); scopes set under one partition are not returned/merged for another partition on the same host.
  • partitionId == null preserves current behavior.
  • Public scope APIs gain non-breaking optional partitionId overloads, and internal call sites forward PartitionId.
  • Tests covering: isolation between two partitions on the same host, and equivalence with existing behavior when partitionId is null.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions