Port of #1558 Adapt multi_model_api_server to vLLM ServingTokenizatio… - #1559
Open
iboiko-habana wants to merge 6 commits into
Open
Port of #1558 Adapt multi_model_api_server to vLLM ServingTokenizatio…#1559iboiko-habana wants to merge 6 commits into
iboiko-habana wants to merge 6 commits into
Conversation
…n refactor Signed-off-by: Iryna Boiko <iboiko@habana.ai>
iboiko-habana
requested review from
PatrykWo,
adobrzyn,
afierka-intel,
jbyczkow,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
June 24, 2026 08:50
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 24, 2026 08:50 — with
GitHub Actions
Error
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Gaudi v1 integration to track upstream vLLM API changes around serving tokenization/state initialization and KV offload metrics, while also improving HPU KV-cache registration compatibility with upstream NIXL probing.
Changes:
- Update multi-model OpenAI server initialization to use
ServingTokenizationand the new speech-to-text / pooling factory init entrypoints. - Extend
TensorTupleto exposeuntyped_storage()anddata_ptr()for upstream NIXL KV-cache registration logic, and update LoRA manager creation call signature. - Refresh KV offloading connector metrics unit tests to match upstream “labeled metrics” payload structure.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
vllm_gaudi/v1/worker/hpu_model_runner.py |
Pass vllm_config into LoRA manager creation and add TensorTuple storage/pointer delegation helpers for NIXL KV-cache registration. |
vllm_gaudi/entrypoints/openai/multi_model_api_server.py |
Port multi-model server state setup to upstream ServingTokenization refactor and new factory-based state initializers. |
tests/unit_tests/kv_offload/offloading_connector/test_metrics.py |
Update tests for upstream labeled-metrics contract (values nested under label-tuples). |
Comment on lines
+6758
to
+6767
| def _first_tensor(self) -> torch.Tensor: | ||
| """Return the first contained torch.Tensor. | ||
|
|
||
| Raises: | ||
| ValueError: If the tuple holds no tensors. | ||
| """ | ||
| for item in self: | ||
| if isinstance(item, torch.Tensor): | ||
| return item | ||
| raise ValueError("TensorTuple contains no torch.Tensor") |
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 24, 2026 09:41 — with
GitHub Actions
Error
Signed-off-by: Iryna Boiko <iboiko@habana.ai>
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 25, 2026 08:33 — with
GitHub Actions
Error
Signed-off-by: Radoslaw Smyrek <radoslawx.smyrek@intel.com>
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 25, 2026 10:11 — with
GitHub Actions
Error
…preserving) #1546 Signed-off-by: Iryna Boiko <iboiko@habana.ai>
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 26, 2026 09:53 — with
GitHub Actions
Error
…41184) #1566 Signed-off-by: Iryna Boiko <iboiko@habana.ai>
iboiko-habana
had a problem deploying
to
pre-merge-approval
June 26, 2026 09:54 — with
GitHub Actions
Error
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
iboiko-habana
temporarily deployed
to
approved-workflow
June 29, 2026 09:23 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…n refactor