Skip to content

feat(sensenova): add vLLM-Omni rollout - #393

Open
CjhHa1 wants to merge 11 commits into
Tencent-Hunyuan:mainfrom
CjhHa1:feat/sensenova-vllmomni
Open

feat(sensenova): add vLLM-Omni rollout#393
CjhHa1 wants to merge 11 commits into
Tencent-Hunyuan:mainfrom
CjhHa1:feat/sensenova-vllmomni

Conversation

@CjhHa1

@CjhHa1 CjhHa1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a SenseNova-U1.5 vLLM-Omni adapter and RL-aware pixel-flow pipeline with driver-authored noise, FlowGRPO SDE transitions, and replay-ready prefix-cache capture
  • add grouped T2I serving, full-weight IPC synchronization, and a colocated training recipe
  • bound rollout transfer memory by storing only replay-required trajectory positions and sharing same-prompt prefix caches
  • move the vLLM stack to the first stable SenseNova pixel-head release (vllm / vllm-omni 0.22) with an explicit CUDA 12.9 wheel and documented glibc floor

Depends on #389, which adds the SenseNova-U1.5 training-side model and replay implementation. Until #389 merges, this PR intentionally includes those stacked commits; GitHub will drop them from this diff afterward.

Related Issue

N/A

Test Plan

  • python3 -m pytest -q tests/models/test_sensenova_u1.py tests/rollout/test_sensenova_vllm_omni.py (13 passed)
  • Hydra compose: python3 -m unirl.train_diffusion --config-name=diffusion/sensenova_u1_5/sensenova_u1_5_vllmomni_full_ipc --cfg job --resolve
  • pre-commit run --all-files
  • H20 end-to-end smoke using SenseNova-U1.5-8B-MoT-Preview, vLLM-Omni 0.22.0+cu129, 1 GPU, 4 denoise steps, one SDE step, and a synthetic two-sample prompt batch: 2 images, sparse trajectory (2, 3, 16, 3072) with indices [1, 2, 4], SDE log-prob (2, 1), shared CPU/pickle-safe prefix caches

Compatibility / Risk

  • The vllm extra moves from 0.20 to 0.22 and selects the explicit CUDA 12.9 x86_64 wheel.
  • vLLM 0.22's required llguidance wheel raises the effective glibc floor to 2.31; this is documented in INSTALL.md.
  • Existing v0.20-only HI3 cache compatibility code now self-skips safely when its removed classes are absent.
  • Qwen3-Omni's legacy audio-video M-RoPE correction is gated when native v0.22 Thinker support is detected, avoiding a double position shift.
  • SenseNova full-weight sync now fails closed when an input name is neither direct-loadable nor covered by the fused QKV/gate-up mapping.
  • The first SenseNova rollout path is T2I, TP=1, full-weight IPC sync; image editing and TP rollout are not exposed by this recipe.

Reviewer Notes

  • Review the sigma/data-time sign conversion and unit-noise SDE transition in RLSenseNovaU1Pipeline first.
  • Distinct prompts are intentionally split into separate engine calls because upstream SenseNova 0.22 only batches multiple outputs of one prompt.
  • Sparse trajectory_indices are carried through the shared vLLM-Omni response builder into LatentSegment.indices for trainer replay.
  • This PR is stacked on feat(model): add SenseNova U1.5 pixel-flow support #389.

Checklist

  • I reviewed the changed code and removed unrelated/generated artifacts.
  • I updated tests, docs, and configs where needed, or explained why not.

CjhHa1 added 8 commits August 24, 2026 20:36
Vendor the NEO-Unify model definitions and add trainside FlowGRPO rollout/replay so SenseNova U1.5 can be trained with deterministic pixel-space trajectories.
Load the complete interleaved MoT layer in fp32 so frozen understanding and trainable generation parameters satisfy FSDP2's per-group dtype contract while compute remains bf16.
Keep deterministic Euler updates in the BF16 trajectory dtype so 512px CFG rollouts remain tensor-identical to the official 50-step implementation while stochastic log-probability math stays fp32.
Normalize stochastic transitions into unit-noise coordinates, match official CFG reduction semantics, move prefix caches with conditions, and remove redundant per-branch and batch work while expanding contract tests.
Serve SenseNova U1.5 rollouts through the first compatible vLLM-Omni release while preserving driver noise, FlowGRPO trajectories, replay caches, and full-weight synchronization.
The v0.22 runtime removed the old HI3 cache classes, so guard their lookup before installing the legacy compatibility patch.
vLLM-Omni warms the unified pipeline through an image-edit request whose active CFG branches do not always include an unconditional cache.
Document llguidance's effective runtime requirement and keep the wheel pin scoped to the repository's supported x86_64 environment.
@github-actions github-actions Bot added the need review Ready and waiting for review label Aug 25, 2026
CjhHa1 added 3 commits August 25, 2026 17:23
Keep the SenseNova model index entries while adopting the upstream HunyuanVideo-1.0 naming cleanup and rollout fixes.
Store only replay-required trajectory positions, share same-prompt prefix caches, and reject full-weight buckets that the vLLM loader would silently skip.
Avoid applying UniRL's old audio-video position shift on vLLM-Omni 0.22, whose native Thinker already computes interleaved positions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need review Ready and waiting for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant