experiments: DSV4-Flash EAGLE 3/1/4 + symm-mem two-shot patch (inline dockerfile)

[Evrard-Nil]

What

New experiment conf experiments/deepseek-v4-flash-eagle-test.yaml: DeepSeek-V4-Flash served with EAGLE 3/1/4 MTP spec-decode plus the torch symm-mem two-shot allreduce fallback patch, applied as an inline dockerfile on top of the exact v0.5.14 digest prod pins (sha256:5027e95b…). Modeled on experiments/deepseek-v4-flash-int4-autoround-test.yaml (full stack: nginx / vllm-proxy / registrar / dcgm / otel), registering as deepseek-ai/DeepSeek-V4-Flash-EAGLE on SNI dsv4-flash-eagle.completions.near.ai. TP2 on GPUs 0,1.

Why (validated on gpu02 dev CVM, 2026-07-06/07 — full detail in Notion "dsv4 work doc")

EAGLE 3/1/4, deliberately WITHOUT --speculative-adaptive:

• +11–22% decode tok/s at prod 30k-context shape (α 2.27 code / 2.42 rag, same-wave two-port A/B, held at prod settings 0.83/64 with no OOM and ~24 GB headroom); +33–40% short-context.
• --speculative-adaptive is condemned on 0.5.14: 2/2 CUDA illegal-memory-access crashes under c1↔c8 burst soaks (two different GPU pairs); upstream #25715 does not backport (fixes a main-only refactor). Cost of no-adaptive: c4 −26%/req during rare bursts — prod DSV4 averages ~1.08 running requests.
• Otherwise identical to prod flags (marlin, cgbs 64, mf 0.83, mrr 64, queue bound 16, mixed-chunk, cps 16384).

Inline patch = sgl-project/sglang#30364 (submitted upstream from this work): sglang's TorchSymmMemCommunicator hard-disables itself whenever NVLS/multicast is unavailable — which is always on our fleet (the driver zeroes MULTICAST_SUPPORTED under CC modes) — even though its two-shot kernel only needs P2P. The patch gates multicast on the multimem path only and falls back to two-shot; --enable-torch-symm-mem then engages (kernel=two_shot). Anchored-replace with loud assert failures + ast.parse check, GLM-#28305 inline-patch style; drop the patch when the upstream PR ships in a release.

Note on expectations: on DSV4 TP2 the symm-mem path measured perf-neutral (AR is only ~2.2 ms / 12.5% of the decode step; two-shot 6.3 µs vs NCCL-in-graph 25.1 µs drowns in wrapper staging + noise) — the experiment's decode win comes from EAGLE. The patch is included to soak the two-shot path on a CVM as groundwork for TP4/TP8 models, where the AR bucket is larger. Fusion-on-IPC (flashinfer) is explicitly out of scope — op-level validated only, its sglang integration isn't.

Validation

• docker compose config passes (gpu02 host) with the inline dockerfile + heredoc surviving interpolation.
• The heredoc patch was executed against the pristine v0.5.14 file: all three anchors hit exactly once, ast.parse passes, result is semantically identical to the upstream PR diff.
• The patched build (same two changes) served DSV4-Flash TP2 end-to-end on the dev CVM: TorchSymmMemCommunicator: enabled (kernel=two_shot, max_size=67108864) at init, correct outputs, stable generation.
• EAGLE 3/1/4 no-adaptive: clean under the c1↔c8 burst soak that killed adaptive (see work doc).

Deploy notes

• Any 8-GPU experiment CVM; model uses GPUs 0,1 only.
• Watch for Decode batch … accept len: in sglang logs to confirm spec engagement, and the kernel=two_shot init line for the patch.
• Remember MIN_TAG_AGE_HOURS (backdated tag) when deploying via compose-manager.

🤖 Generated with Claude Code

[github-actions]

🔍 OpenCodeReview found 2 issue(s) in this PR.

• ✅ 2 posted as inline comment(s)
• 📝 0 posted as summary

[github-actions]

Spelling error in YAML key: hugginface_cache should be huggingface_cache (missing the second 'g'). Several other compose files in this project (e.g., cleanup-hf-model.yaml, experiments/GLM-5.1-AWQ-4bit.yaml, prod/Qwen3.5-122B.yaml, etc.) use the correctly-spelled huggingface_cache. The cleanup-hf-model.yaml explicitly states the volume name "MUST match the inference compose files" so it resolves to the same <project>_huggingface_cache Docker volume. Because Docker Compose prefixes volume names with the project name, this misspelling creates a separate volume (<project>_hugginface_cache) that the cleanup utility and other services cannot share, leading to cache duplication, wasted disk from orphaned volumes, and ineffective cleanup.

Suggestion:

Suggested change

	- hugginface_cache:/root/.cache/huggingface
	- huggingface_cache:/root/.cache/huggingface

[github-actions]

Spelling error in YAML key: the volume declaration hugginface_cache should be huggingface_cache to match the correct spelling used in other compose files (e.g., cleanup-hf-model.yaml, experiments/GLM-5.1-AWQ-4bit.yaml). This declaration and the volume reference on line 21 must both be renamed together for Docker Compose to pick up the correct volume. See the comment on line 21 for full impact analysis.

Suggestion:

Suggested change

	hugginface_cache:
	huggingface_cache:

[PierreLeGuen]

Additive-only change: one new experiment compose file (739 additions, 0 deletions) touching no existing prod/staging config. docker compose config renders cleanly, the inline dockerfile's Python patch is self-guarding (three assert s.count(anchor) == 1 checks plus a final ast.parse, so any drift in the pinned sglang source fails the build loudly), and model-name/model-path/base-url/SNI references are all internally consistent. Good to merge.

Optional follow-up (non-blocking):

• experiments/deepseek-v4-flash-eagle-test.yaml:665 — the registrar probes the proxy /v1/models without an Authorization header while the proxy runs with TOKEN=${PROXY_TOKEN}; if the proxy enforces that token, readiness will exhaust its retries before registering. Worth confirming the probe matches the prod registrar pattern (PROXY_TOKEN env + -H "Authorization: Bearer $$INFERENCE_TOKEN").

Checks: docker compose config passes with dummy env; registrar script and full YAML diff cleanly against the on-main sibling deepseek-v4-flash-int4-autoround-test.yaml (only expected model/host/domain/engine substitutions); validate_otel_labels.rb, validate_proxy_dependencies.rb, validate_proxy_environment.rb, validate_registrar_auth.rb all pass. Image build / end-to-end serving not run (requires pinned sglang base + GPU CVM).

[evrardnil]

Exact with/without-patch numbers (requested A/B, run 2026-07-08 on the recovered dev CVM): sequential same-GPU-pair A→B→A′ protocol on GPUs 0,1, identical prod-digest image + EAGLE 3/1/4 config, arm B = this PR's inline patch + --enable-torch-symm-mem (verified TorchSymmMemCommunicator: enabled (kernel=two_shot)), stock arms verified no engagement. A′ = stock relaunch to bound run-to-run drift.

metric	A stock	B symm-patch	A′ stock (drift check)
greedy bs1 med tok/s (n=12, temp 0, same prompt)	88.0	88.7	83.9
greedy med TPOT	11.37 ms	11.28 ms	11.92 ms
med TTFT (short prompt)	470 ms	466 ms	486 ms
30k-ctx code, med decode tok/s (n=6, temp 0.7)	78.9	81.7	—
30k-ctx doc/rag, med decode tok/s (n=6)	97.3	98.0	—
conc=4 per-req med tok/s (n=12)	58.7	63.9 (+8.4%)	59.2
conc=4 aggregate tok/s	205.9	220.5 (+7.1%)	204.4

Verdict:

• bs1: neutral — the A↔A′ drift (±5%) exceeds the A/B delta. Matches the earlier op-level analysis (AR is only ~12% of the bs1 decode step).
• c4 concurrency: real +7–9% — A and A′ agree tightly (58.7/59.2, 205.9/204.4) while the patch arm stands apart (63.9/220.5). Mechanism: batched EAGLE rounds multiply all-reduce count/size, so the two-shot kernel (6–15 µs vs NCCL 23–27 µs in-graph) finally materializes end-to-end.
• 30k single-stream: directionally positive (+3.5% code, +0.7% doc), single round — treat as indicative.

This upgrades the patch from "enablement-only" to a measurable concurrency win on DSV4 TP2, and strengthens the TP4/TP8 expectation (more ranks, more AR share). Raw results: /dstack/persistent/claude-bench/dsv4mtp/abres_{stock,symm,stock2}.txt on the dev CVM.

🤖 Generated with Claude Code