Skip to content

Allow callers to control prefix-cache partitioning #83

Description

@h4x3rotab

Problem

Private AI Gateway does not provide consistent caller control over prefix-cache partitioning.

For the active Kimi-K2.6 providers:

  • Chutes forwards a caller's optional cache_salt to vLLM.
  • Tinfoil strips cache_salt and derives its own value. It accepts the router-only user_cache_secret as the caller-controlled input, but the gateway does not currently translate the public field.

As a result, the same cache_salt request has different behavior across providers.

Desired behavior

Keep cache_salt caller-controlled across providers:

  • Forward cache_salt unchanged to Chutes and direct vLLM providers.
  • Translate cache_salt to user_cache_secret for Tinfoil, which derives the final salt.
  • Do not derive, overwrite, or bind the partition to authenticated Redpill tenant identity.
  • Preserve current provider behavior when the caller omits cache_salt.

Callers may choose distinct values for isolation or reuse a value to share a cache namespace. This is user-controlled partitioning, not a gateway-enforced tenant boundary.

Acceptance criteria

  • A supplied cache_salt reaches Chutes and direct vLLM providers unchanged.
  • The same public field reaches Tinfoil as user_cache_secret.
  • Omitting cache_salt does not add a gateway-generated partition.
  • Tests cover provider-specific request transformation and receipt consistency.
  • Documentation explains the caller-controlled semantics and provider translation.

Evidence

Tinfoil's attested router strips cache_salt, consumes user_cache_secret, and derives the final salt: https://github.com/tinfoilsh/confidential-model-router/blob/v0.0.118/cache_salt.go

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions