Skip to content

Port of #1558 Adapt multi_model_api_server to vLLM ServingTokenizatio… - #1559

Open
iboiko-habana wants to merge 6 commits into
mainfrom
iboiko/pre_releases/v0.24.0
Open

Port of #1558 Adapt multi_model_api_server to vLLM ServingTokenizatio…#1559
iboiko-habana wants to merge 6 commits into
mainfrom
iboiko/pre_releases/v0.24.0

Conversation

@iboiko-habana

Copy link
Copy Markdown
Collaborator

…n refactor

…n refactor

Signed-off-by: Iryna Boiko <iboiko@habana.ai>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Gaudi v1 integration to track upstream vLLM API changes around serving tokenization/state initialization and KV offload metrics, while also improving HPU KV-cache registration compatibility with upstream NIXL probing.

Changes:

  • Update multi-model OpenAI server initialization to use ServingTokenization and the new speech-to-text / pooling factory init entrypoints.
  • Extend TensorTuple to expose untyped_storage() and data_ptr() for upstream NIXL KV-cache registration logic, and update LoRA manager creation call signature.
  • Refresh KV offloading connector metrics unit tests to match upstream “labeled metrics” payload structure.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
vllm_gaudi/v1/worker/hpu_model_runner.py Pass vllm_config into LoRA manager creation and add TensorTuple storage/pointer delegation helpers for NIXL KV-cache registration.
vllm_gaudi/entrypoints/openai/multi_model_api_server.py Port multi-model server state setup to upstream ServingTokenization refactor and new factory-based state initializers.
tests/unit_tests/kv_offload/offloading_connector/test_metrics.py Update tests for upstream labeled-metrics contract (values nested under label-tuples).

Comment on lines +6758 to +6767
def _first_tensor(self) -> torch.Tensor:
"""Return the first contained torch.Tensor.

Raises:
ValueError: If the tuple holds no tensors.
"""
for item in self:
if isinstance(item, torch.Tensor):
return item
raise ValueError("TensorTuple contains no torch.Tensor")
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
Signed-off-by: Radoslaw Smyrek <radoslawx.smyrek@intel.com>
…preserving) #1546

Signed-off-by: Iryna Boiko <iboiko@habana.ai>
…41184) #1566

Signed-off-by: Iryna Boiko <iboiko@habana.ai>
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
@iboiko-habana
iboiko-habana temporarily deployed to approved-workflow June 29, 2026 09:23 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants