[RFC]: Read-only NIXL GDS connector for filesystem KV-cache loads

[maheshrbapatu]

Motivation.

vLLM's native OffloadingConnector can move KV-cache blocks from GPU memory to CPU DRAM. The Tiering connector extends this with secondary storage such as a filesystem.

The existing data path is:

Store: GPU → CPU → filesystem
Load:  filesystem → CPU → GPU

When KV data is available only on the filesystem, it must first be promoted into the CPU cache and then copied to the GPU. For storage supporting GPUDirect Storage, the filesystem data can instead be read directly into GPU memory, avoiding the intermediate filesystem-to-CPU transfer.

This RFC explores an optional, read-only NIXL GDS connector for that load path. The existing Tiering connector remains responsible for CPU caching and filesystem stores.

The goals are:

• Load existing filesystem-backed KV data directly into GPU memory.
• Preserve the current filesystem store path and file format.
• Keep live CPU-cache hits ahead of filesystem loads.
• Keep scheduler metadata independent of process-local GPU pointers.
• Compose the functionality using the existing MultiConnector.

The initial scope is the merged Tiering filesystem layout with matching store and load configurations.

Proposed Change.

Configure two ordered OffloadingConnector children under MultiConnector:

1. A TieringOffloadingSpec child that continues to manage the CPU primary cache and filesystem stores.
2. A read-only NixlGDSOffloadingSpec child that looks up and loads the files produced by the Tiering connector.

flowchart LR
    subgraph Store["Existing store path"]
        SGPU["GPU KV cache"] --> CPU["CPU primary cache"]
        CPU -->|"Existing Tiering store"| FS[("Filesystem")]
    end

    subgraph Load["Load selection"]
        MC["MultiConnector"]
        T["Tiering connector"]
        G["Read-only GDS connector"]
        STAGE["GPU staging"]
        LGPU["GPU KV cache"]

        MC --> T
        MC --> G
        T -->|"Live CPU hit"| LGPU
        G -->|"Filesystem hit"| FS
        FS -->|"NIXL GDS_MT"| STAGE
        STAGE --> LGPU
    end

Loading

Connector responsibilities

The Tiering connector continues to perform:

• GPU-to-CPU stores.
• CPU-to-filesystem stores.
• Live CPU-cache loads.
• Filesystem naming, creation, commitment, and lifecycle.

The GDS connector performs only filesystem lookup and filesystem-to-GPU loads. It does not produce store jobs or modify filesystem objects.

Load selection

The connectors are considered in this order:

1. Check the live CPU primary cache.
2. If the block is not resident in CPU, check the filesystem through the GDS connector.
3. On a filesystem hit, load the required range into GPU staging and copy it into the destination KV-cache blocks.

The filesystem secondary tier needs a read-side option such as load_enabled: false. This leaves its existing stores active while preventing it from starting a filesystem-to-CPU promotion for a load that should use GDS.

Existing filesystem layout

The GDS connector consumes the merged Tiering filesystem format. It does not introduce another format.

For a tensor-parallel configuration, one filesystem block contains the existing per-rank regions:

┌────────────────┬────────────────┬────────────────┐
│ rank 0 region  │ rank 1 region  │ ... rank N     │
└────────────────┴────────────────┴────────────────┘

Every worker receives the same file path. Worker rank N determines its range using the existing CPU offload layout and reads only that range:

existing file + rank-local offset + rank-local length
    → NIXL GDS_MT
    → rank-local GPU staging
    → destination KV blocks

The scheduler passes file information and logical destination block IDs. Each worker resolves its local GPU addresses. A load is reported complete only after the storage read and GPU copy have completed; failed or partial loads must not be exposed as valid KV-cache hits.

Scope

The initial implementation covers:

• CUDA and NIXL GDS_MT.
• Read-only filesystem-to-GPU loads.
• The existing merged Tiering filesystem format.
• Single-node tensor parallelism.
• Matching model, dtype, block-size, and parallel configurations between store and load.
• Rank-local filesystem extents large enough to benefit from GDS.

It does not change filesystem stores, file ownership, eviction policy, parallel-layout compatibility, or model-layout support provided by the existing offloading implementation.

Questions for feedback

1. Is a separate read-only connector the right boundary for direct GDS loads?
2. Should MultiConnector prefer a live CPU hit before consulting GDS?
3. Is a read-side filesystem-tier switch appropriate for keeping existing stores active while bypassing filesystem-to-CPU promotion?
4. Should the GDS connector derive rank ranges from the existing CPU offload layout, or should those ranges be included explicitly in scheduler metadata?

Feedback Period.

Two weeks.

CC List.

@orozery @dannyharnik @ronensc @rarepepi

Any Other Things.

This proposal builds on the Tiering architecture introduced in #38260 and implemented in #40020.

Draft PR #38863 explored filesystem and GDS offloading before the Tiering framework was implemented. That earlier exploration motivated revisiting direct filesystem-to-GPU loads using the architecture now available in vLLM.

In the later #38260 discussion, it was noted that GDS, if explored, could remain separate from the Tiering connector and be composed through MultiConnector. This RFC explores that arrangement.