Qwen3.8-27B serving line: full-precision adoption, captured block-draft speculation, small-M and state-stash tiers - #175
Open
LiangSu8899 wants to merge 44 commits into
Open
Conversation
…bling of the expert-bank door A checkpoint that ships full-precision linear projections too big for the card packs each one straight to the Hub NVFP4 layout through the same seam binder the pre-quantized door uses - no dequant-requant detour through someone else's grid, and both doors produce the same executable form. Two families are refused with their reasons on the receipt: gated-delta state layers belong to their own scheme's packing and release semantics, and a vocabulary projection is logits-family precision, an explicit binder decision rather than a bulk side effect.
Native mirrors of the three additive kernel tiers delivered to the hub packages, plus the harnesses that judge them, so packaging can verify against this tree directly: - fp4_w4a4_mma_warpsplit_ilv_sm120: interleaved-B warp-split GEMV and its bind-time repack; every block's global B reads become fully sequential, 89-91% of the achievable DRAM read roof on the wide decode shapes, bit-exact against the base kernel. - gated_delta_wy_kkt_mma: the WY K*KT chunk as one block per (chunk, k-head) with wmma accumulation and group-shared Gram tiles; 32.8x over the scalar kernel at S=2048, numerics in the bf16 reduction-order band, same entry surface as the scalar version. - cutlass_nvfp4_gemm_m256_sm120: 256x128x128 cooperative large-M tier; wins every prefill family over the 128-tile baseline. CUTLASS >= 4.5. csrc/kernels/checks/ carries the three judges: bit-exactness for the GEMV (including the device repack against a host reference), the scalar-vs-MMA numeric band and speed for the KKT, and the TFLOPS receipt for the GEMM tier. Build lines are in each file header.
…irst assembly One round = one block-draft forward plus one multi-token verify, replacing the MTP family's per-token draft chain. The draft is a small full-attention infill network conditioned on auxiliary features tapped from five target layers (forward pre-hooks over the loop's own passes); verify and re-advance ride _fwd_full with the GDN snapshot/rollback idiom, and the static-cache row mask makes cropping unnecessary. The geometry that survived measurement: the draft emits one logit row per noise slot including the seed slot - dropping the seed row, as the draft checkpoint's own reference implementation does, halves the acceptance length. The Markov bigram head applies sequentially, anchored at the seed. First-assembly receipts: deterministic repeat, coherent output, acceptance length 3.13 at 48 new tokens. Open case on the record: acceptance thins with generation depth (2.26 at 96) while the text stays coherent and easy - not stream hardness, not monotonic state corruption (rounds recover sporadically); the per-round diagnostic ladder is written up in the working notes. Graph capture of the round is the planned second stage, as with the MTP member.
The draft forward is one captured graph: static context K/V take the block's own keys in place, one masked SDPA covers context and block siblings, and the serial Markov head unrolls on device. The verify captures with the gated-delta snapshot at its head; re-advance drops to the state sublayers alone - the verify already committed the accepted region's KV and carries the bonus logit at the cut - with rollback captured at each graph's head. One host sync per round (the arbiter), verify-input assembly rides inside the draft graph. Real-source coding stream, decode-only, same card and window: 38 -> 86 tok/s at AL 2.9 against the plain loop's 71; the rewrite-heavy stream 95.6 at AL 3.2. Repeat gate green; round profile identical to the whole-model re-advance form.
The 16x8x64 block-scaled MMA atom already produces a 16-row output
tile; the M=1 interleaved GEMV feeds it one live row and fifteen rows
of zero ballast. This tier feeds up to 16 live rows through the same
atom, same shared-memory plan, same accumulator budget - B stays the
only DRAM-bound stream, so a multi-row call lands near single-GEMV
cost. The spec-verify shapes (M = block + 1, and the shorter re-advance
prefixes) are the customers.
Measured on the wide decode shapes at M=8: 74-80% of the 1690 GB/s
read roof, 5.3-6.0x over M sequential GEMV calls, 1.16-1.37x over the
plain-B multi-row variant (the interleave advantage grows with M), and
~1.3x over the production tile GEMM's 59% at these shapes. Bit-exact
per row against the M=1 interleaved kernel across all shapes, M in
{1,2,4,7,8,16}, and warp configs (in-tree check included). SFA reads
the production 512B-block layout (row r at tile*512 + r*16); B shares
the M=1 entry's bind-time interleave.
2<=M<=16 activations - the spec-verify block and its re-advance
prefixes - dispatch to the native build's multi-row warp-split GEMM
where the local tree carries it: the 16x8x64 block-scaled atom
computes a full 16-row tile, so all rows ride one weight stream. The
pointer-style native entry registers as a torch custom op with a fake
shim, so the compiled multi-row passes trace through it and the call
lands on the capture stream. Per-shape launch config: deeper stages
hide the strided-B latency the extra A-row loads expose.
Layout compatibility is proven at the quantizer, not assumed: the
production activation quantizer's multi-row SFA block feeds the tier
bit-exactly against per-row GEMV calls across every projection family
and M in {2,4,7,8}. Absence of the native build changes nothing - the
tiled GEMM keeps serving every shape.
Same card, same real-source coding stream: verify pass 21.4 -> 18.3ms,
re-advance tiers 4.0-6.0ms, decode-only 86 -> 96.6 tok/s at AL 2.9
against the plain loop's 71.5; repeat gate green, round profile
identical to the tiled-GEMM form.
The draft's dual-source attention moves off SDPA onto the vendored fa2 seqused entry: the valid-K length (context + block) lives in a device buffer the kernel reads at run time, so the captured launch replays each round's window with no mask and no materialised attention math - the compiled SDPA path was demoting the masked GQA form to an expand-clone of the whole context window. Registered as a custom op with a fake shim so the compiled draft graph traces through it on the capture stream. Draft projection precision is decided by measurement, not appetite: W4A4 through the adoption door costs acceptance (2.93 -> 2.79 on the real-source coding stream) and W8A8 per-tensor costs the same (2.80) - a 1.4B draft's weight stream is not the binding constraint, its prediction quality is. bf16 stays the default; both arms remain behind FRT_DSPARK_DRAFT_QUANT for streams that judge differently. Same card, same stream: propose graph 6.95 -> 4.12 ms, decode-only 96.6 -> 100.9 tok/s at AL 2.93 against the plain loop's 71.5; draft tokens identical to the SDPA form round for round, repeat gate green.
The from-conv chunk core gains a stash arm: identical recurrence, gating math, and per-row bf16 state requantisation, with the carried state after each row additionally written to a stash slab. The stash write IS the carried requantisation, so row s is bit-equal to the final state a re-advance over rows 0..s would store - proven at the kernel level against the plain chunk entry across prefix lengths. A rejected speculative round then rolls back by selection: pick the stash row at the accepted length, rebuild the conv window from the snapshot tail plus the stashed raw rows (the epilogue's own semantics), and touch no projection at all. The re-advance graphs drop from forty-eight sublayer re-drives to a fixed set of copies: 4.0-6.0ms -> 0.27-0.39ms per shape, while the verify pass pays 0.4ms for the stash writes. The fused layer arms the stash eagerly (before any compiled pass traces) and only where the native build carries the kernel; absent that, rejected rounds keep the re-drive form. Same card, same real-source coding stream: decode-only 100.9 -> 116.4 tok/s at AL 2.97 against the plain loop's 71.5; repeat gate green.
Capability probes now prefer the installed artifact and fall back to the local native build, tiled/scalar entries always the floor: - linear_proj: the multi-row warp-split tier resolves hub-first (fp4_w4a4_gemm_warpsplit_mrows_bf16, a compile-safe torch op) and keeps the native custom-op fallback for local builds. - gated_delta: the per-row stash arm resolves hub-first (gdn_chunk_from_conv_smem_stash_bf16); the whole-prompt WY chain takes the wmma Gram tier (gdn_wy_kkt_b64_mma_bf16) where the artifact carries it - same signature, same A layout, and the measured long-prompt Gram term drops off the profile's top set (42.1ms -> 1.3ms class at 2K). - decode_loop: enable_dspark(draft_dir) attaches the block-draft speculative runner as a loop member, mirroring enable_mtp. Real-source coding stream after the Gram swap: prefill at 5.2K drops ~100ms, decode round unchanged at ~25.5ms (AL moves within band with the prompt-numerics shift), repeat gate green.
M>=512 activations dispatch to nvfp4_gemm_m256_bf16 where the artifact carries it - the tier wins every measured prefill family over the base tile, and the wrapper owns its workspace. Absence changes nothing. With the Gram swap and this tier together, the 2K prefill wall drops 214.5 -> 164.8 ms on the 27B host; the decode round and the token stream are unchanged, repeat gate green.
Sibling projections that read the same normed hidden (the attention q/k/v trio, the MLP gate/up pair) each quantized their input separately; inductor does not CSE the custom quantize op, so the same rows were packed up to three times per site. link_shared_producers hands each sibling group one identity-keyed cell: a seam reuses the stored quantization only when its input is the very tensor that produced it, so a fresh activation can never be served stale data. quantize_on_adopt links the groups after adoption - the checkpoint's default execution form. Bit-identical by construction; the sharing contract (functional activations) is documented on the cell.
A one-shot prompt pass over a deep window overruns the card: the whole-prompt activations, the GQA repeat-expand of the KV heads, and the full-rows lm_head each materialize window-sized transients. The loop (and the block-draft runner) now accept prefill_chunk: the prompt drives the same offset-mask forward in slices, gated-delta state carrying across slices through the continuation branch the verify batches already exercise. Slice attention routes through a compiled flex_attention arm over an offset-causal block mask (eager flex is the math fallback and materializes the score matrix; masked GQA SDPA does the same) - the mask depends only on slice length and start, so a prompt's handful of masks replay across layers, chunks and runs. The runner's slices take the last-row head: nothing in a prompt slice consumes the full-rows logits. Both arms are flag-gated to chunked loops; established short-window forms keep their exact paths. The sliced pass is not bitwise-identical to the one-shot form: two runs of the FP4 pipeline decorrelate at low mantissa bits (measured slice-vs-oneshot logits cosine equals the arbiter band squared), so the equivalence contract is the arbiter gate plus continuation identity, never bit equality.
nvfp4_balance projection format: the same FP4 band with a per-input-channel balance fitted on calibrated activation amax, plus calibrate_gdn_channel_amax - the house Collector observing the two projection inputs over a caller-supplied real forward. A host layer without attached calibration keeps the BF16 band (counted, never raised). Schemes w4a4_balance_decode(_release) select the format; calibration is the caller's precondition. Measured on this host family: the out-projection seat cuts its quantization error 2.5x (the in-projection saturates at 1.3x), but the end-to-end teacher-forced gate is unchanged - the near-tie token flips are the aggregate of the whole pipeline's quantization noise, not the projection grid alone. The arm therefore ships as an explicit opt-in with this receipt, not as a default.
…ill path Four additive kernels with native bindings: a SwiGLU activation + NVFP4 quantize producer (quantize stage transcribed from the production bf16 quantize kernel, bit-exact against the split chain), a (1+w)-form RMSNorm + NVFP4 quantize producer, a batched 64x64 unit-lower-triangular inverse for the WY chain (column-independent forward substitution, identity/tril preparation folded in), and a v2 launch plan for the WY norm/pack + gate-cumsum pair (chunk-parallel cumsum, bit-exact per chunk). Also exposes the interleaved warp-split GEMV tiers through the native module.
The gate and up seams concatenate into one merged seam (one weight stream, one launch per tier) and the activation between them collapses into the fused silu-mul quantize producer, feeding the down projection its packed FP4 input directly. The projection seam grows a tier-dispatched packed-input entry and a measured per-shape launch config for the decode GEMV.
One thread computes 8 consecutive tokens with the taps rolling through registers, cutting the packaged kernel's K-fold DRAM re-read of every input element to (STEPS+K-1)/STEPS; the q/k/v channel split folds into the store. Tap order and fma chain match the packaged kernel.
The triangular solve routes to the native batched unit-lower inverse (same forward-substitution recurrence as the batched cuBLAS solve, identity/tril preparation folded in), the norm/pack + gate-cumsum pair takes the chunk-parallel v2 launch, and the conv update goes through the GQA-split variants - the step-batched native arm where the build carries it, the packaged one otherwise. Projection seams now receive the caller's tensor instead of a view so an upstream producer's identity-keyed quantization handoff survives the call.
The (1+w)-form norm computes exactly the host norm and quantizes the same values in one kernel, publishing packed FP4 + SFA through the identity-keyed share cell its consumer group already honors; the normed BF16 tensor still flows through the host graph, so a cell miss just means the consumer quantizes for itself. Decode-width calls keep the host norm - the group quantize is cheaper there than a standalone producer launch. Scale-atom tails are zero-filled: unwritten bytes in a partial 128-row atom were a measured nondeterminism channel.
…rking Runtime weights in a mixed-precision engine are packed in formats the seam cannot read as rows, so binding takes its material from the source checkpoint the engine loaded (fused modules concatenated in the engine's own order) and gates each seat on a cosine probe against the host module it replaces. Optional tiers are parked by tracing them under a fake mode rather than by name, so an artifact that ships the missing fake impls lights them up with no code change; the gated-delta dependency pins the version whose build matrix covers the engine's runtime.
…P seat A host that compiles a seam forward for a range of row counts hands the dispatch a symbolic M, so a tier branch would bake in whichever side the tracing sample took and run it for every replayed shape. Such callers now take the tiled GEMM, which serves every M; concrete row counts keep their tiers. The vLLM adapter gains a fused SwiGLU seat: the host's MLP is already gate_up -> activation -> down, so the three steps collapse into one seat where the fused activation-quantize producer is available, and it parks the two row-count-thresholded tiers this engine cannot replay safely.
The recurrence gives every value column its own thread, so the columns now spread across blocks instead of tying block width to the 128 columns, and the column state streams in two passes rather than sitting in a per-thread array the hardware spills to local memory — 2x on the step's own shape, output bit-identical. The gated norm gains a variant that also emits its consumer's NVFP4 rows: the output projection is its only consumer and quantizes what it receives, and the quantizer's blocks tile a head's lanes exactly, so the whole step fits in the block that produced the row (bit-identical normed rows, packed bytes and scales alike).
…oducer The recurrent step routes to the V-split entry where the build carries it, and the gated norm hands the output projection packed rows directly instead of writing a row its only consumer re-reads to quantize. Buffers for the producer arm are allocated before any capture so the graph records stable addresses, and the norm weight is copied detached to keep the op off an autograd edge. Kernel resolution also stops trusting a single release: a version range now walks the published tags down to its floor and takes the newest release whose build matrix covers the caller's torch/CUDA pair. A publisher adding variants release by release can drop a pair an older release carried, and a resolver too old to read versions lands on the repo default regardless — either way the range, not one pinned tag, is what the dependency actually asked for.
The measured win in the streaming rewrite was never the column split — it was not spilling the state column to local memory. Splitting the columns across narrower blocks also moved the q/k norms onto a warp reduction, which changes their fp32 summation order for a further 0.03 ms/step; the block width goes back to 128 so the whole kernel is bit-identical to the packaged one, state and output alike, and the step still stops paying DRAM for state it believes is resident.
The band where the fused producer beats the host norm is a property of what the compiler folded around that norm, not a constant: at decode width the host path carries the residual add into the same kernel, and replacing it there costs more than the quantize it saves (measured +0.13 ms/step). The threshold stays where the measurement put it and is now nameable, so the next host can be measured rather than assumed.
A resolver that accepts only an exact major still names the majors it has when it rejects a range, so take the newest one at or above the floor from that report instead of probing tags downward. The blind walk stays as the fallback for a resolver that says nothing.
Each of these three kernels now ships in a published package, and the packaged entries are torch ops carrying fakes — a host that compiles the call traces them unaided, and a serving process that cannot load this build's native extension (different torch/CUDA pair) still gets the fused form instead of falling back to the split chain. The local build stays as the second source, wrapped as before.
Version resolution walks releases over the network, and every step of that walk is an unbounded round trip. A host binding a whole model asks for the same missing package once per seat, so one unpublished build variant used to cost that walk hundreds of times over, with the engine silent throughout. Default the hub timeouts (callers that set their own keep them) and remember resolution failures for the life of the process: absence is as cacheable as presence and far more expensive to re-establish.
A host holding a packed checkpoint does not need the regridding entry and should not pay it: dequantizing someone else's grid and quantizing it again with ours replaces their calibration with our packer's rounding, which is a change of model wearing the costume of an acceleration. bind_proj_seam_packed adopts the e2m1 bytes by reference (the block-scale relayout is a permutation, it loses nothing) and carries a checkpoint's per-tensor factor on each tier's alpha, where the epilogue is already writing the result; the separate pass it replaces cost a full-size read-modify-write per projection. Also let the fp8 no-bias form bind the quantize's return value: on the traced path the persistent out-buffer is None and the returned tensor is the only handle to the result. The fused silu-mul hub entry sizes its outputs at the call site for the same tracing reason: the wrapper's size helper is a custom op without a fake, and a host tracing on Meta tensors dies inside it.
A mixed-precision checkpoint is a per-projection decision made with calibration data, and seating every position at the adapter's default width overrides that decision silently. Read the choice off what the host actually holds and honour it: positions already in the seam's width adopt the host's own tensors by reference (no regrid, no second copy on the card), FP8 positions get a seat on their own storage (the engine keeps its weight transposed for its cutlass entry, so the orientation comes from the layer's declaration, and one transposed swap leaves total memory unchanged), and unquantized positions are left alone -- a draft model a checkpoint excluded from quantization stays excluded, because a speculative host pays for numerics in acceptance length, not in any number a throughput probe reads. The auto tier adds one opinion on top: a W8 position is carried to W4, sourced from the host's own FP8 rows (every FP8 code is exact in BF16, so no external checkpoint is involved), and the attach report says so in plain text -- the precision change is the caller's to validate. Staged release of replaced host weights now actually releases them: the revert closure used to pin the very tensor it was freeing through a default argument, which is why a consume pass reported the same bytes freed whether the staging ran or not.
An air-gapped serving container cannot walk the hub at all: the xet transport ignores the process proxy and hangs, offline resolution insists on a complete snapshot when the cache holds only the variant that was ever downloaded, and pinning the resolver library's version trades one incompatibility for another. FRT_KERNEL_DIR_<REPO> names the build variant directory directly and imports it as the package it already is -- exactly what the resolver would do after its network walk, minus the walk. The arch check still runs; a wrong variant refuses the same way a resolved one would.
The sglang hook now routes to the same tier assembly the vllm adapter runs: the two engines share module lineage down to the quant-method attribute names, so per-position adoption and the auto tier's W8->W4 carry assemble unchanged, and the engine-specific part of that adapter stays what it was -- the sitecustomize carrier. Two host behaviours surfaced by the second engine, fixed where they belong: a projection seat now falls through attribute reads to the module it replaced (an engine that planned a fusion around the projection reads scale attributes off the module object, and a seat answering only forward turned that into a startup crash far from the seam); and a merged projection carrying one global factor per constituent refuses adoption under a single alpha unless the factors agree -- pretending otherwise would scale one half by the other's factor.
The verify-attention seat routes speculative verify steps through the paged BF16-Q/FP8-KV kernel behind a full guard ladder: metadata must be exactly a single-request causal verify batch on a supported layer geometry, anything else falls through to the host. Page counts come from a hook on the prefill wrapper's plan (the device buffers hold the same values, but reading them there is a host sync per layer, which stalls the launch pipeline for more than the kernel saves), K/V claims are as_strided views over the raw cache storage (flattening the permuted cache view silently copies the whole pool), and capture-time calls are declined so replayed graphs never bake a stale shape. The head and draft seats are explicit opt-ins rather than defaults: the seam's row tiers quantize activations to FP4, and on the logits family and the proposal path that measurably moves speculative acceptance — the mechanical step-rate win is paid back with interest. A checkpoint-packed NVFP4 head can be adopted zero-copy behind FRT_HEAD_W4A4; a BF16 draft can be re-gridded via draft_precision. Projection seats now honor the host's return_bias so a bare-tensor call site is answered in kind.
The host forward the verify-attention seat replaces is not just the attention call — it is also where the step's new K/V rows enter the paged cache. Taking the call without that write leaves the tokens' K/V unwritten: the seat's own kernel reads stale slots for them, and every later step is missing them as prefix, so the damage compounds for the rest of the sequence. The seat now performs the host's cache update before running the kernel, and refuses calls that carry fused output-quant arguments it does not implement.
The head's verify-step calls are M<=16 logits projections, served by the host on a tiled kernel sized for prefill. The relay adopts the checkpoint's own NVFP4 pack into the small-M layout at bind time — same grid, same dequantized values — and routes calls inside the band there, host method for everything else. Judged across three boots the step rate moves +7-8% with acceptance unchanged. The bind receipt compares against an E2M1 table-decode of the checkpoint's rows rather than the host's apply: the host quantizes activations on its own path, so an exact relay still reads ~0.92 against it while sitting at 0.999999 against the rows themselves. A 'w8' precision tier joins the projection binder for the draft band: per-channel weight-only INT8 from the module's dense rows, the middle ground between BF16's weight stream and W4's rounding.
The engine's MTP draft shares the target's lm_head, and acceptance is an agreement test between the two. Relaying only the target's head to the small-M layout changes one side of that agreement: judged paired over ten prompts the acceptance length lands lower on all ten (about -0.5) while the step rate gain is smaller than the loss. With a speculative config present the relay now defaults off; without one it remains the head's default, where it is pure step-rate. The paired-by-prompt protocol is the judging change that surfaced this: single-prompt greedy runs drift to different continuations per arm and bury systematic acceptance effects in content variance.
Usage for the engine adapters with every configuration in its measured form: the load-hook window and precision tiers, the spec-decode head-relay rule, the explicit-KV long-context recipe, the SGLang sitecustomize/container bridge with the DSpark memory floor, and the judging protocol the receipts require (paired-by- prompt for speculative arms, multi-boot medians at long context, real-text prompts with difference-based decode measurement).
The bind-or-refuse determinants in one place: SM120 for the sm120a kernel tiers (elsewhere the seats refuse and the host runs unmodified), the vLLM/SGLang series the adapters read, the torch/CUDA build variant the published kernels ship for, and the ModelOpt mixed-precision checkpoint family the adopt tiers expect — with the re-grid tiers as the road for dense checkpoints. Records the measured speculative configuration and the single-stream scope of the receipts.
Same multiplier, second engine: the stock DSpark server's pool is 34,659 tokens and refuses a 60K request; the attached server's pool is 87,128 (2.51x) and decodes an 80K real prompt at 210.8 tok/s. Absolute ceilings differ from vLLM's because the hybrid-state cache and draft KV price each token higher — the released weight memory multiplies whichever budget the engine's accounting allows.
The guide now opens with what it is and is not: measured hot-plug configurations on each engine's documented recipe, not an exhaustive engine tuning and not a community comparison — the ceiling figures reflect the engines' default memory accounting under these settings. Names the measured checkpoints (the ModelOpt mixed-precision quantization of the 27B host, and the DSpark speculator) so the receipts are reproducible from the hub. The SGLang long-context receipt becomes a table with the full curve: pool 34,659 vs 87,128 tokens, the stock refusal at 60K, and 147.7 / 210.8 tok/s decode at 60K / 80K on the attached server.
The comparison now states what was tried before it was written: the draft model card's own serving flags on both arms, a memory-fraction sweep (a higher fraction grows the paper pool to ~56K tokens but 48K requests then fail on runtime headroom, and neither arm boots above it), CUDA-graph trims, and the hybrid-cache knobs — none of which move the pool, whose 2.25 GB intermediate state cache is insensitive to all three. Notes that higher published context figures for this family come from the card's multi-GPU recipe, keeping the single-card table apples-to-apples.
The disclaimer now says plainly what is owed: these adapters exist because both engines are excellent hosts, and better configurations than ours are welcome — the numbers will be updated for them. The SGLang stock-arm tuning narrative becomes a checkable table: every configuration attempted, its pool, and what a long request actually did, with the paper-pool footnote and the stable-envelope conclusion the comparison table rests on.
Retitled from a disclaimer to what it actually is — a note on how to read the numbers fairly: not a community benchmark, an exploration of what hot-plugging adds to each engine as configured. The voice now matches the project (single author), and the invitation stands: better configurations are welcome and the numbers will be updated for them.
The guide gains the zero-code-change path for a stock vllm serve: a sitecustomize on PYTHONPATH installs the load hook behind an env gate, with the measured server-side receipts (code +7.5%, text +9%, TTFT -13% against back-to-back stock boots; the spec-decode head rule carries over unchanged). The example is the demo built on it: the same serve command twice, where the stock boot refuses the model's native 262144 context on a 32 GB card and the attached boot serves it — released weight memory becoming the KV pool — plus a streaming client with a live decode meter for asking questions of a 200K-token codebase.
One conversation that keeps growing beats one giant prompt on camera: with prefix caching on, each turn prefills only its increment, so the demo keeps a conversational pace while the context meter climbs. The stock arm dies at its ceiling with the server's own 400 printed as the crash row; the attached arm walks the same turns to the model's native maximum. Arms are recorded separately (one card cannot hold two copies of the weights) and the client prints the same table for both, so the recordings align turn for turn. The race form runs without speculative decode, which this vLLM series requires for prefix caching; the one-shot scenes keep the speculative speed numbers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Additive serving line for the Qwen3.8-27B host (Qwen3.5 backbone, 48 gated-delta + 16 full-attention layers), in three layers:
Adoption.
quantize_on_adoptgains thelinear_proj_nvfp4format: full-precision dense projections pack to the hub NVFP4 layout on adoption, with structural guards (gated-delta layers stay with their own scheme, vocabulary projections are refused by embedding-signature match) and aretainedreport for every refusal. CPU contract tests included.Speculation.
decode_loopgains a block-draft (DFlash/DSpark family) member, attached viaenable_dspark(draft_dir). The round rides the loop's existing graph families: the draft forward is one captured graph (static context K/V, serial Markov head unrolled on device, verify-input assembly in-graph), the gamma+1 verify captures with the gated-delta state snapshot at its head, and rejection rolls back by selection — the verify pass stashes the carried state after every row, so the re-advance graphs reduce to a fixed set of copies. One host sync per round. The draft's attention runs on the vendored fa2 seqused entry (device-side valid-K length, replay-adaptive); draft weights stay bf16 by measurement — both W4A4 and W8A8 arms cost more acceptance than they save and remain env-gated.Kernel tiers. Two additive native kernels with in-tree bit-exactness checks, mirrored to the kernels repo:
The structures layer resolves every tier capability-first: hub artifact, then the local native build, with the tiled/scalar entries always the floor. The whole-prompt WY chain now takes the wmma Gram tier and M>=512 prefill slabs take the cooperative 256-tile tier where the installed artifact carries them.
Measured — standalone decode loop line (RTX 5090)
This line serves the full-precision BF16 checkpoint
(
Qwen/Qwen3.8-27B, 52 GB) and quantizes at load time — a differentstarting point from the engine adapters below, which adopt the
pre-quantized checkpoint. All rows are from one fresh re-measurement
window, full production form (graphs + compiled step + fused-FFN and
norm-quant producer knives), gates green unless noted:
One open case rides the 2K bucket (teacher-forced agreement 0.936
against a 0.95 gate — the archived balance-band item; it reproduces
exactly, so it is an old open case, not a fresh regression). The
block-draft member's copy-stream and 20K-edit receipts are withheld
until their gate is repaired (it currently fails on a stream-capture
error after the tree moved); re-measured numbers will replace this
note.
Gates: repeat chains bitwise-identical across calls at every stage; the multi-row tier is bit-exact against per-row GEMV across all projection families and M in {2,4,7,8} through the production activation quantizer; the stash rows are bit-exact against prefix re-advance at the kernel level; speculative streams anchor on the loop's own verify.
Companion kernel-package PR: flashrt-project/FlashRT-HF-kernels#11.
Serving-engine adapters (vLLM / SGLang)
Second wave on this branch: the structures layer attaches inside a
serving engine — no fork, no model conversion, one hook installed
before weight load.
adapters/vllm_engine— precision tiers over the host's ownpacks (
autoadopts NVFP4 zero-copy and carries the checkpoint'sFP8 rows to W4;
mirrorstays faithful per position;w8weight-only INT8),
consumereleases every replaced host weight sothe KV pool grows by that amount, a spec-verify attention seat
(opt-in) on the paged BF16-Q/FP8-KV kernel, and a small-M Marlin
relay for the checkpoint's W4A16 head (2.0x the host's tiled head
kernel at M<=16). The head relay steps aside automatically under
speculative decode: the MTP draft shares the target's head, and
relaying one side of that agreement was measured — paired over ten
prompts — as a systematic acceptance loss.
adapters/sglang_engine— the same seat machinery carriedacross the scheduler's spawn boundary via a written
sitecustomize, with filesystem-direct kernel loading forair-gapped serving containers.
docs/serving_engine_attach.md— every configuration in itsmeasured form, plus the judging protocol the numbers require
(paired-by-prompt for speculative arms, multi-boot medians at long
context, real-text prompts with difference-based decode
measurement).
Decode across contexts (single RTX 5090, real code/text prompts, greedy; paired fresh baselines)
Decode-only tok/s; spec = MTP K=6 (vLLM) / DSpark (SGLang); 32K vLLM
cells are 3-boot medians.
vLLM
SGLang
Without speculation the attached arms win every cell: +13–27%
(vLLM) and +9–11% (SGLang). Speculative decode columns ride
acceptance-length content variance (greedy continuations differ per
arm); judged on step rate — the column content variance cannot touch
— the attached arms hold +9–13% uniformly. The vLLM 2K spec row
is a paired-by-prompt mean over five prompts; at 32K, code lands
+52% and text −5–15% against the fresh baselines above.
Time to first token
Long context: the ceiling moves, and decode holds
consumeturns the released weight memory into KV capacity. On the32 GB card that is the difference between cannot serve and serves
at full speed:
Decode along the attached engine's context curve (MTP on, real
prompts, acceptance shown because it rides content):
* the 120K text point is a content effect — that stretch of the
corpus is table/number-dense and the draft stops landing; the step
rate column shows the engine itself does not care.
Step rate is essentially flat from 32K to 200K. The stock engine
cannot reach any of these lengths on this card; the attached engine
serves the model's entire native window at 150+ tok/s decode. This is
enablement, not a percentage: a context band that did not exist on
this hardware now runs at full speed, from the released memory alone.
The same story on SGLang (DSpark serving, identical memory
fraction, real code prompts):
One multiplier, two engines: the released weight memory scales
whatever context budget the engine's own accounting allows — 102K to
native 262144 on vLLM, 34.7K to 87.1K on SGLang, both about 2.5x
under these configurations. The stock SGLang arm was tuned before this table was written — the
draft card's own flags, a memory-fraction sweep (0.85 / 0.92 / 0.95 /
0.97), CUDA-graph trims, and the hybrid-cache knobs; the full sweep
table is in
docs/serving_engine_attach.md. A higher fractiongrows the paper pool to ~56K tokens but 48K requests then fail on
runtime headroom (at both prefill chunk sizes), and neither arm
boots above it — 0.92 is the stable envelope for both arms.
Published higher-context figures for this family come from the model
card's own multi-GPU recipe (tp-size 4); the single-card comparison
above is apples-to-apples, and configurations I missed are welcome.
Speculative memory floors, both engines
The same released memory also carries speculation where the stock
engine is tight: SGLang with the DSpark draft refuses to boot at its
default memory fraction (draft weights leave no room for the KV
pool) and needs 0.90–0.93; the attached engine's freed weights cover
the draft with room to spare.