FLUX.2-klein FP8 weight-only is blocked by the pre-v0.5.11 sglang-diffusion pin — qualify a newer image to unlock ~4GB on GPU7
Finding (dev-CVM tested): the prod FLUX serving image (lmsysorg/sglang@sha256:8ece90ad…, version 0.0.0.dev1+g1e309030e, pre-v0.5.11) does not expose --transformer-path, so the FP8 transformer swap (black-forest-labs/FLUX.2-klein-4b-fp8, float8_e4m3fn, ~4GB vs ~8GB BF16) cannot be deployed on the current engine. ModelOpt diffusion FP8 + --transformer-path only landed in sglang v0.5.11, which the pin deliberately avoids (v0.5.11+ → CUDA 13/torch 2.11; the cu129 torchvision mismatch breaks from torchvision.io import decode_jpeg).
So: prod FLUX (BF16 + --enable-torch-compile + cache-dit, --mem-fraction-static 0.15) is already the best config on the current pin — no config tweak helps.
Opportunity (engine-gated): FLUX.2-klein's FP8 transformer would roughly halve the DiT footprint (~8GB → ~4GB), freeing budget on the heavily-shared gpu11 GPU7 (10 co-tenant models). Published FLUX-FP8 results show quality holds at the 4-step operating point; the speed win is weak (only 4 DiT passes), so this is a VRAM/co-tenant-budget play, not a throughput play.
Ask: qualify a sglang-diffusion v0.5.11+ image that resolves the cu13/torchvision decode_jpeg break (or a torchvision-pinned rebuild). If qualified, add --transformer-path black-forest-labs/FLUX.2-klein-4b-fp8 to the FLUX block, holding all other prod flags, and validate VRAM-freed + SSIM/CLIP quality vs BF16. Until then, FLUX stays BF16. Details: benchmarking/flux/NOTES.md.
FLUX.2-klein FP8 weight-only is blocked by the pre-v0.5.11 sglang-diffusion pin — qualify a newer image to unlock ~4GB on GPU7
Finding (dev-CVM tested): the prod FLUX serving image (
lmsysorg/sglang@sha256:8ece90ad…, version0.0.0.dev1+g1e309030e, pre-v0.5.11) does not expose--transformer-path, so the FP8 transformer swap (black-forest-labs/FLUX.2-klein-4b-fp8, float8_e4m3fn, ~4GB vs ~8GB BF16) cannot be deployed on the current engine. ModelOpt diffusion FP8 +--transformer-pathonly landed in sglang v0.5.11, which the pin deliberately avoids (v0.5.11+ → CUDA 13/torch 2.11; the cu129 torchvision mismatch breaksfrom torchvision.io import decode_jpeg).So: prod FLUX (BF16 +
--enable-torch-compile+ cache-dit,--mem-fraction-static 0.15) is already the best config on the current pin — no config tweak helps.Opportunity (engine-gated): FLUX.2-klein's FP8 transformer would roughly halve the DiT footprint (~8GB → ~4GB), freeing budget on the heavily-shared gpu11 GPU7 (10 co-tenant models). Published FLUX-FP8 results show quality holds at the 4-step operating point; the speed win is weak (only 4 DiT passes), so this is a VRAM/co-tenant-budget play, not a throughput play.
Ask: qualify a sglang-diffusion v0.5.11+ image that resolves the cu13/torchvision
decode_jpegbreak (or a torchvision-pinned rebuild). If qualified, add--transformer-path black-forest-labs/FLUX.2-klein-4b-fp8to the FLUX block, holding all other prod flags, and validate VRAM-freed + SSIM/CLIP quality vs BF16. Until then, FLUX stays BF16. Details:benchmarking/flux/NOTES.md.