Skip to content

add vLLM-side LMCache EC connector entrypoint #38668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
benyebai wants to merge 1 commit into
base: main
Choose a base branch
from
+59 −0
Draft

add vLLM-side LMCache EC connector entrypoint #38668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions vllm/distributed/ec_transfer/ec_connector/factory.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,9 @@ def get_connector_class(
"vllm.distributed.ec_transfer.ec_connector.example_connector",
"ECExampleConnector",
)

ECConnectorFactory.register_connector(
"LMCacheECConnector",
"vllm.distributed.ec_transfer.ec_connector.lmcache_connector",
"LMCacheECConnector",
)
53 changes: 53 additions & 0 deletions vllm/distributed/ec_transfer/ec_connector/lmcache_connector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from typing import TYPE_CHECKING, Any

import torch

from vllm.config import VllmConfig
from vllm.distributed.ec_transfer.ec_connector.base import (
ECConnectorBase,
ECConnectorMetadata,
ECConnectorRole,
)
from vllm.v1.core.sched.output import SchedulerOutput

from lmcache.integration.vllm.vllm_ec_adapter import LMCacheECConnectorImpl
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The top-level import of lmcache creates a hard dependency on an external package that is not a mandatory dependency of vLLM. If lmcache is not installed, this module will fail to import, which can break module discovery or static analysis tools. It is recommended to handle this optional dependency gracefully.

Suggested change
from lmcache.integration.vllm.vllm_ec_adapter import LMCacheECConnectorImpl
try:
from lmcache.integration.vllm.vllm_ec_adapter import LMCacheECConnectorImpl
except ImportError:
LMCacheECConnectorImpl = None


if TYPE_CHECKING:
from vllm.v1.request import Request


class LMCacheECConnector(ECConnectorBase):
def __init__(self, vllm_config: VllmConfig, role: ECConnectorRole):
super().__init__(vllm_config=vllm_config, role=role)
self._impl = LMCacheECConnectorImpl(
vllm_config=vllm_config,
role=role,
parent=self,
)
Comment on lines +23 to +29
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since lmcache is an optional dependency, you should verify its presence before attempting to instantiate the implementation class. This provides a much clearer error message to the user if the package is missing.

Suggested change
def __init__(self, vllm_config: VllmConfig, role: ECConnectorRole):
super().__init__(vllm_config=vllm_config, role=role)
self._impl = LMCacheECConnectorImpl(
vllm_config=vllm_config,
role=role,
parent=self,
)
def __init__(self, vllm_config: VllmConfig, role: ECConnectorRole):
super().__init__(vllm_config=vllm_config, role=role)
if LMCacheECConnectorImpl is None:
raise ImportError(
"LMCacheECConnector requires the 'lmcache' package. "
"Please install it with `pip install lmcache`.")
self._impl = LMCacheECConnectorImpl(
vllm_config=vllm_config,
role=role,
parent=self,
)


def start_load_caches(
self, encoder_cache: dict[str, torch.Tensor], **kwargs: Any
) -> None:
return self._impl.start_load_caches(encoder_cache, **kwargs)

def save_caches(
self,
encoder_cache: dict[str, torch.Tensor],
mm_hash: str,
**kwargs: Any,
) -> None:
return self._impl.save_caches(encoder_cache, mm_hash, **kwargs)

def has_cache_item(self, identifier: str) -> bool:
return self._impl.has_cache_item(identifier)

def update_state_after_alloc(self, request: "Request", index: int) -> None:
return self._impl.update_state_after_alloc(request, index)

def build_connector_meta(
self, scheduler_output: SchedulerOutput
) -> ECConnectorMetadata:
return self._impl.build_connector_meta(scheduler_output)
Comment on lines +22 to +53
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The LMCacheECConnector class is missing delegations for several lifecycle methods defined in ECConnectorBase, including register_caches, get_finished, update_connector_output, and request_finished.

Specifically, request_finished is critical for signaling when an asynchronous transfer is complete and the cache can be safely freed. By not overriding it, the connector defaults to the base implementation which returns False, potentially leading to race conditions or resource leaks if LMCache expects to manage the cache lifecycle. Ensure all methods from the base class are properly delegated to self._impl.

Loading