[RFC]: Logprobs/Logits Semantics and Determinism Across the vLLM Ecosystem

[aoshen02]

Related: vllm#45585 (RL CI Matrix)

Motivation

Several open bugs and in-flight fixes across vLLM, vLLM-Ascend, and VERL point to the same underlying problem: the semantics and determinism of returned logits/logprobs are not specified tightly enough.

This shows up as:

• prompt-side values ignoring logprobs_mode
• request-order-dependent prompt_logprobs under prefix caching
• duplicated sampled tokens in top_logprobs
• backend-specific mode propagation gaps
• rollout/trainer logprob mismatch in VERL
• logprob values changing based on the number of requested log probabilities
• no mechanism for deterministic decode replay (re-evaluating logprobs of a previously generated sequence)

Problem Statement

The same logical request can produce different answers depending on:

• whether prompt-side or decode-side probabilities are returned
• raw vs processed logprobs_mode
• prefix caching being enabled
• streaming delta behavior
• sampled token overlap with top-k/top-p output
• backend implementation details in core vLLM, vLLM-Ascend, or VERL integration paths
• the number of top_logprobs requested (token-string collision in top_logprobs map)
• micro-batch size differences between rollout and actor (causes on-policy pg_clipfrac ≠ 0)
• MoE combine non-determinism under DP+EP without VLLM_BATCH_INVARIANT

That creates user-visible wrong values, non-deterministic scoring, and training signal mismatch.

RL-specific impact: PPO/GRPO training requires logprobs for importance sampling ratio (old_logprobs vs new_logprobs). Non-deterministic or semantically incorrect logprobs directly corrupt the policy gradient signal. This is the root cause behind GRPO collapse reports (e.g., AReaL#1290).

Proposed Change

Define a shared contract for:

• logits
• logprobs
• prompt_logprobs
• top_logprobs

The contract should cover:

• raw vs processed semantics
• deterministic behavior with prefix caching
• streaming delta behavior
• de-duplication rules for sampled token vs top-k list
• backend parity requirements across core vLLM, vLLM-Ascend, and VERL-facing rollout paths

Rollout Plan

1. Finish and land core semantic fixes and regression tests in vLLM.
2. Propagate the same contract into vLLM-Ascend sampler/backend paths.
3. Align VERL rollout/trainer behavior with the same raw/processed definitions.
4. Keep this issue as the single working tracker.

Open Bugs

Semantic — logprobs values themselves are wrong

Bug	Fix PR	Status	Description
#35832	#47680 (V1+V2)	OPEN	prompt_logprobs ignores logprobs_mode — always returns log_softmax even when raw_logits is configured. Corrupts KL divergence in PPO/GRPO. Earlier PRs #36539 and #35885 are superseded.
#44319	#44504	OPEN	Logprobs values change based on the number of requested top_logprobs — token-string collision in top_logprobs map
#36660	#36746	OPEN	Duplicate sampled token with different logprob in top_logprobs
#37037	—	OPEN	FlatLogprobs empty slice crash + delta-mode logprobs[-0:] returns stale data

Determinism — same input produces different logprobs

Bug	Fix PR	Status	Description
#42019	#42245	OPEN	prompt_logprobs depends on request order when prefix caching is enabled — torch.empty uninitialized memory, cache hit region retains stale data
—	#45683	OPEN	MoE combine (reduce_scatterv) non-deterministic under DP+EP — needs VLLM_BATCH_INVARIANT mode
#47425	—	OPEN	Logprob divergence between vLLM and HF transformers on fine-tuned VL model

Streaming / spec-decode edge cases

Bug	Fix PR	Status	Description
—	#44845	OPEN	Discarded speculative rows leaking logprobs
—	#37738	OPEN	Streaming tool-parser drops logprobs when suppressing control tokens
—	#47815	OPEN	Wrong text_offset in streamed completion logprobs with echo=True
—	#44727	OPEN	Shape mismatch crash mixing spec-decode chat + scoring with logprob_token_ids

Replay — missing RL primitives

PR	Status	Description
#46701	OPEN (no reviewer)	trace_decode_token_ids — force sampler to emit predetermined token sequence while computing real logprobs. Core RL primitive: PPO/GRPO needs to re-evaluate logprobs of previous trajectories under current policy.
#43463	OPEN	Expose logprob_token_ids on OpenAI chat/completion endpoints — request logprobs at specific vocab IDs

Downstream (not vLLM bugs)

Issue	Framework	Description
verl#6240	verl	Processed vs raw logprobs_mode mismatch in async RL → trainer inconsistency
verl#6280	verl	Rollout vs actor micro-batch size difference → on-policy pg_clipfrac ≠ 0
SkyRL#877	SkyRL	Top-p/top-k sampler replay — propagate sampling mask to trainer (per DeepSeek v3.2 tech report)
SkyRL#1776	SkyRL	LoRA + MoE inference mismatch for Qwen3.6 MoE and GLM-4.7-Flash

Ascend parity

Bug	Fix PR	Status
vllm-ascend#7218	#8643	OPEN

Performance

PR	Status	Description
#47594	OPEN	Fused generated-token top-logprobs path
#45327	OPEN	Reduce prompt logprobs memory usage
#47623	OPEN	Avoid redundant logprobs list materialization

Acceptance Criteria

• The meaning of returned logits/logprobs is documented and consistent.
• Prompt-side values are mode-aware ([Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode #47680 landed).
• Prefix-cached results are deterministic ([Bugfix] Fix prompt_logprobs non-determinism with prefix caching (issue #42019) #42245 landed).
• top_logprobs never duplicates the sampled token.
• trace_decode_token_ids available for RL trajectory replay ([Core][V1] Support trace_decode_token_ids for deterministic decode replay #46701 landed).
• Ascend and VERL either follow the same contract or document explicit deviations.

Feedback Period

At least one week.