Cache sandbox-to-node routing in the AENV gateway

[yingdi-shan]

Affected component

Gateway / scheduler

Problem statement

The gateway calls the scheduler LookupNode RPC for every request containing a sandbox ID. This adds scheduler load and latency.

Use case

Large scale deployments handling frequent data-plane and control-plane requests for long-lived sandboxes. The same sandbox may receive many requests through different gateway replicas, while deletes and node changes may be handled by any replica.

Current behavior and workarounds

Every sandbox-routed request performs a scheduler lookup. There is no gateway cache. The scheduler has no explicit binding-delete RPC, and ReportSandboxEvent currently logs delete events without mutating the binding store. A local-only delete eviction would therefore leave other gateways with stale entries until their process restarts or a scheduler binding expires.

Desired behavior

• Repeated requests for an active sandbox should avoid scheduler lookups while a valid route-cache entry exists.
• Each gateway may keep a bounded, process-local positive TTL cache keyed by sandbox ID and storing the node ID and endpoint.
• The scheduler remains the source of truth. Successful lookups may be cached; NotFound and scheduler errors must not be negative-cached.
• After a successful delete, the handling gateway evicts its local entry and the shared scheduler binding is removed, so other gateways converge after cache expiry or stale-route detection.
• If a cached route receives a reliable sandbox-not-found response, the gateway evicts the entry and performs one uncached scheduler lookup. If the scheduler also returns NotFound, return 404. If it returns a new node, retry once only for a replay-safe request.
• Streaming, WebSocket, non-replayable, and ambiguous data-plane 404 requests must not be retried.
• Cache hits, misses, evictions, and refetches should be observable, and concurrent misses for one sandbox should be coalesced.

Proposed approach

Add a bounded process-local cache in the gateway. Query the configured scheduler on cache misses, and populate or refresh entries after successful LookupNode calls and successful create or fork assignment recording.

Compatibility and operational impact

API/config changes:

No client-facing API change is required.

Snapshot or storage format changes:

None.

New host/runtime requirements:

None. The cache is in process memory.

Alternatives considered

• Keep doing a scheduler lookup for every request: correct but adds avoidable scheduler traffic and latency.
• Use a distributed cache or pub/sub invalidation for every gateway: provides faster invalidation but adds infrastructure and another consistency path.
• Evict only the gateway that handled delete: insufficient because other gateways retain stale mappings.

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

[DAOZHENREN]

Hi, I'd like to work on this issue.

My proposed implementation plan is:

1. Add a bounded, process-local positive TTL route cache in the gateway,
   keyed by sandbox ID and storing the resolved scheduler node.
2. Coalesce concurrent cache misses for the same sandbox so that only one
   LookupNode RPC is issued.
3. Populate or refresh the cache after successful LookupNode calls and
   successful create/fork assignment recording. Scheduler errors and
   NotFound responses will not be cached.
4. Evict the local entry after a successful sandbox deletion.
5. Add conditional binding removal to the scheduler binding store and process
   delete lifecycle events in ReportSandboxEvent. The removal would only
   delete the binding if it still points to the reporting node, preventing a
   delayed delete event from removing a newer assignment.
6. On a reliable control-plane sandbox-not-found response from a cached route,
   evict and perform one uncached scheduler lookup. Retry at most once and only
   for explicitly replay-safe control-plane requests. WebSocket, streaming,
   non-replayable, and data-plane 404 responses will not be retried.
7. Add bounded-cardinality Prometheus metrics for cache hits, misses,
   evictions, and refetches, together with unit tests for expiry, capacity,
   concurrency, invalidation, and retry behavior.

Before I start the implementation, could you confirm:

• Is extending ReportSandboxEvent with conditional delete handling preferred,
  or would you rather introduce an explicit scheduler binding-delete RPC?
• Should the initial replay-safe whitelist be limited to control-plane
  GET, HEAD, and DELETE requests?
• Do you have preferred defaults or names for the cache capacity and TTL
  configuration?

If this direction looks good, I can prepare the implementation and tests.