[Render][Speculator] Add return_loss_mask to render endpoint for training data generation#46846
Merged
vllm-bot merged 1 commit intoJun 28, 2026
Conversation
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
WindChimeRan
force-pushed
the
speculators/render-loss-mask
branch
from
450bf56 to
8fe6896
Compare
WindChimeRan
requested review from
AndreasKaratzas,
DarkLight1337,
NickLucche,
aarnphm,
chaunceyjiang,
njhill,
robertgshaw2-redhat and
russellb
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
4 tasks
Member
|
Why deviate from the established naming convention of "return_assistant_tokens_mask"? I think this is more aligned with the functionality this service would provide, rather than "return_loss_mask" which just reflects what a training server would want to do with it |
benchislett
reviewed
Jun 26, 2026
benchislett
reviewed
Jun 26, 2026
Member
|
Besides the naming, this generally LGTM. Ideally would like a review from a codeowner here though |
benchislett
reviewed
Jun 26, 2026
benchislett
reviewed
Jun 26, 2026
WindChimeRan
requested review from
luccafong,
patrickvonplaten,
sighingnow,
tjtanaa and
zyongye
as code owners
mergify
Bot
added
deepseek
Add a `return_assistant_tokens_mask` field to ChatCompletionRequest and
an `assistant_tokens_mask` field to GenerateRequest, so the render
endpoint can return a per-token mask marking assistant-generated tokens.
The mask is computed in a single apply_chat_template call when the chat
template uses {% generation %} tags; otherwise assistant_tokens_mask is
null. Downstream consumers (e.g. speculators) use this for training
data generation without duplicating tokenization locally.
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
WindChimeRan
force-pushed
the
speculators/render-loss-mask
branch
from
cc37059 to
27e7c17
Compare
DarkLight1337
approved these changes
Jun 27, 2026
github-actions
Bot
added
the
ready
wincent8
pushed a commit
to wincent8/vllm
that referenced
this pull request
Jun 29, 2026
…ning data generation (vllm-project#46846) Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
10 tasks
rjrock
pushed a commit
to rjrock/vllm
that referenced
this pull request
Jul 1, 2026
…ning data generation (vllm-project#46846) Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
noooop
pushed a commit
to noooop/vllm
that referenced
this pull request
Jul 9, 2026
…ning data generation (vllm-project#46846) Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Adds a
return_loss_maskfield toChatCompletionRequestand aloss_maskfield toGenerateRequest, soPOST /v1/chat/completions/rendercan return a per-token assistant/trainable mask alongsidetoken_ids. This is needed by downstream training frameworks (e.g. [speculators](vllm-project/speculators#652)) that use the render endpoint to delegate tokenization to vLLM instead of duplicating it locally with HFapply_chat_template. The mask is computed in a singleapply_chat_templatecall (no second pass) via HF'sreturn_assistant_tokens_maskwhen the chat template has{% generation %}tags; otherwiseloss_maskisnulland the caller handles the fallback.Test
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.