[EC Connector] CPU Offloading EC Connector

[omerpaz95]

Add ECCPUConnector — CPU encoder-cache offloading connector

Purpose

This PR adds ECCPUConnector, a self-contained encoder-cache (EC) connector that offloads encoder outputs to a shared CPU mmap region on /dev/shm so a single ec_role=both vLLM instance can reuse them on a later request instead of re-running the vision encoder.

The connector is intentionally minimal. Everything happens within one instance — encode once, save the encoder output to the CPU region, and on a subsequent cache hit reload mmap→GPU rather than recomputing.

This is a self-contained, minimal version of the bigger, p2p-enabled EC CPU NIXL Connector.

Logic

• Scheduler delegate (ECCPUScheduler) — owns the mmap region and the
  shared offload bookkeeping (_local_encodings, _blocks), and routes
  scheduler-side calls to a producer and/or consumer sub-delegate. It exposes
  has_cache_item, ensure_cache_available, update_state_after_alloc, and
  build_connector_meta.
  • ECCPUProducer — the encode/offload pipeline. ensure_cache_available performs a speculative region-capacity check;
    update_state_after_alloc records a pending save;
    build_saves allocates region blocks (FIFO eviction of unpinned entries on exhaustion) and promotes the mm_hash into the shared _local_encodings cache.
  • ECCPUConsumer — the local-reload path. has_cache_item reports a hit when an mm_hash is present in _local_encodings;
    update_state_after_alloc pins the producer-allocated blocks;
    build_loads re-serves them for an mmap→GPU copy and unpins.
• Worker delegate (ECCPUWorker) — a separate OS process, pure torch/mmap.
  save_caches copies GPU→mmap (synchronizing the save stream so the copy is CPU-complete before the block is served); start_load_caches copies mmap→GPU.

End-to-End Architecture

                 single vLLM instance (ec_role=both)
  ┌───────────────────────────────────────────────────────────────┐
  │  Scheduler process                                             │
  │  ┌──────────────────────── ECCPUScheduler ───────────────────┐ │
  │  │  ECCPUProducer            ECCPUConsumer                    │ │
  │  │    build_saves()            has_cache_item()               │ │
  │  │    ensure_cache_available   update_state_after_alloc (pin) │ │
  │  │                             build_loads() (re-serve)       │ │
  │  │  shared: _local_encodings, _blocks, _shared_lock           │ │
  │  └────────────────────────────────────────────────────────────┘│
  │                         │ saves / loads metadata (per step)      │
  │                         ▼                                        │
  │  Worker process                                                 │
  │  ┌──────────────────────── ECCPUWorker ──────────────────────┐ │
  │  │  save_caches:  GPU ──────► mmap block   (/dev/shm)         │ │
  │  │  start_load:   mmap block ─────► GPU                       │ │
  │  └────────────────────────────────────────────────────────────┘│
  └───────────────────────────────────────────────────────────────┘

Lifecycle (offload-reuse cycle)

1. A request with mm_features arrives. has_cache_item(mm_hash) returns False (first sight), so the encoder input is scheduled for compute.
2. The worker encodes the item; save_caches copies the encoder output GPU→mmap (with a blocking stream sync, so the bytes are durable before the step's model execution returns).
3. In the same step, build_saves allocates the region blocks, records mm_hash → block_indices in _blocks, and promotes mm_hash into _local_encodings (the shared local cache) — so the item is reusable from the next step.
4. A later request for the same mm_hash: has_cache_item now returns True, so the scheduler routes it to external_load_encoder_input instead of recomputing.
5. update_state_after_alloc pins the cached blocks; build_loads re-serves the same block indices as a load and unpins them.
6. start_load_caches copies mmap→GPU. The encoder is not re-invoked.

Configuration

Selected by name through the transfer config, with the ec_both role:

from vllm import LLM

llm = LLM(
    model="your-vision-model",
    kv_transfer_config={
        "ec_connector": "ECCPUConnector",
        "ec_role": "ec_both",
    },
)

ECTransferConfig is unchanged — connector selection is purely by name. All
three roles (ec_producer, ec_consumer, ec_both) are accepted, but only
ec_both delivers reuse in the CPU-only connector (a producer-only instance
saves but never reloads; a consumer-only instance finds nothing locally). The
full role plumbing is kept so the NIXL subclasses inherit role handling
untouched.

Files

All new files live under vllm/distributed/ec_transfer/ec_connector/:

File	Purpose
ec_shared_region.py	ECSharedRegion — mmap lifecycle, alloc/free/pin/unpin, cudaHostRegister
cpu/__init__.py	Package marker
cpu/common.py	ECRegionContext, ECCPUConnectorMetadata, setup_ec_region
cpu/worker.py	ECCPUWorker — GPU↔mmap copies
cpu/connector.py	ECCPUConnector — role router + _make_* seams
cpu/scheduler/__init__.py	ECCPUScheduler — scheduler delegate + _build_* seams
cpu/scheduler/common.py	evict_and_alloc — FIFO eviction helper
cpu/scheduler/producer.py	ECCPUProducer — encode/offload pipeline
cpu/scheduler/consumer.py	ECCPUConsumer — local mmap reload

Plus one line in factory.py registering ECCPUConnector.

Test Plan

Unit tests live under tests/v1/ec_connector/unit/ and run in full with:

python -m pytest tests/v1/ec_connector/unit/ -v

Coverage:

• test_ec_shared_region.py — region alloc/free/try_free/pin/unpin
  refcount semantics, FIFO eviction skipping pinned blocks, exhaustion raising AllocationError.
• test_metadata.py — ECCPUConnectorMetadata defaults and independence of saves/loads.
• test_producer.py — build_saves allocate-and-promote, _fifo_alloc eviction (oldest-unpinned / skip-pinned), ensure_cache_available deferral on a full region.
• test_consumer.py — has_cache_item, update_state_after_alloc pin, build_loads re-serve + unpin.
• test_worker.py — GPU↔mmap round-trip reproduces the source tensor bit-for-bit (CUDA-gated tests skip on hosts without a GPU).
• test_scheduler_cpu.py — delegate wiring and the full offload-reuse cycle (save → promote → hit → reload of the same blocks).
• test_connector_cpu.py — role dispatch (scheduler vs worker) and factory registration.
• test_no_nixl_imports.py — AST guard asserting no nixl/zmq/msgspec imports (and no NIXL-only submodules) are reachable from the cpu/ package.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[orozery]

remove file?