diff --git a/docs/source/deployment-guide/deployment-guide-for-glm-5-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-glm-5-on-trtllm.md index 30d9b9eb13ee..25a6b31471a0 100644 --- a/docs/source/deployment-guide/deployment-guide-for-glm-5-on-trtllm.md +++ b/docs/source/deployment-guide/deployment-guide-for-glm-5-on-trtllm.md @@ -4,10 +4,23 @@ This deployment guide provides step-by-step instructions for running the GLM-5 model using TensorRT LLM with FP8 and NVFP4 quantization, optimized for NVIDIA Blackwell GPUs. It covers the complete setup required; from accessing model weights and preparing the software environment to configuring TensorRT LLM parameters, launching the server, and validating inference output. -GLM-5 uses Multi-Latent Attention (MLA) with DeepSeek Sparse Attention (DSA). It shares the same architecture as DeepSeek V3.2 and reuses the `DeepseekV32ForCausalLM` code path in TensorRT LLM. GLM-5 natively supports Multi-Token Prediction (MTP) for speculative decoding. +GLM-5 uses Multi-Latent Attention (MLA) with DeepSeek Sparse Attention (DSA). It shares the same architecture as DeepSeek V3.2 (with minor changes) and is served through the `GlmMoeDsaForCausalLM` model, which reuses the DeepSeek V3.2 code path in TensorRT LLM. GLM-5 natively supports Multi-Token Prediction (MTP) for speculative decoding. + +This guide applies to the GLM-5 family, including GLM-5.2 and GLM-5.3. GLM-5.3 is a weight update over GLM-5.2 with the same architecture and code path, so the server configurations and deployment steps below are identical across versions. Note that GLM-5.3 ships a revised chat template: the `enable_thinking` template kwarg is replaced by `clear_thinking`, and `reasoning_effort` accepts `low`/`high` (default `max`). Clients controlling reasoning behavior via `chat_template_kwargs` should update accordingly. The guide is intended for developers and practitioners seeking high-throughput or low-latency inference using NVIDIA's accelerated stack. +### Validated Features + +The following features have been tested with GLM-5 on TensorRT LLM: + +* CUDA Graph +* Multi-Token Prediction (MTP) +* Disaggregated serving +* Disaggregated serving with MTP + +Because GLM-5 reuses the DeepSeek V3.2 code path, other features supported by `DeepseekV32ForCausalLM` are expected to work but have not been separately validated. See the [Model-Feature Support Matrix](../models/supported-models.md#model-feature-support-matrix-key-models) for the current per-feature status. + ## Prerequisites * GPU: 8x NVIDIA B200 (SM100) @@ -20,13 +33,15 @@ The guide is intended for developers and practitioners seeking high-throughput o The following checkpoints are available: -* FP8 model: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8) — Official FP8 checkpoint -* BF16 model: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5) — Official BF16 checkpoint -* NVFP4 model: [warnold-nv/GLM-5-nvfp4-v1](https://huggingface.co/warnold-nv/GLM-5-nvfp4-v1) — Unofficial NVFP4 checkpoint for experimentation only. *Quantized with ModelOpt by Will Arnold.* +* FP8 model: [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) — Official FP8 checkpoint +* BF16 model: [zai-org/GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16) — Official BF16 checkpoint +* NVFP4 model: [warnold-nv/GLM-5-nvfp4-v1](https://huggingface.co/warnold-nv/GLM-5-nvfp4-v1) — Unofficial NVFP4 checkpoint of GLM-5 for experimentation only. *Quantized with ModelOpt by Will Arnold.* + +> **Note on checkpoint naming:** For GLM-5.3 the base repository (`zai-org/GLM-5.3`) contains the FP8 weights and the BF16 weights live in the `-BF16` repository. This is the reverse of GLM-5 and GLM-5.2, where the base repositories ([zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5), [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)) are BF16 and the FP8 weights live in the `-FP8` repositories ([zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8), [zai-org/GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8)). Check the `quantization_config` in the checkpoint's `config.json` if in doubt. ```bash git lfs install -git clone https://huggingface.co/zai-org/GLM-5-FP8 /models/GLM-5-FP8 +git clone https://huggingface.co/zai-org/GLM-5.3 /models/GLM-5.3 ``` ## MoE Backend Support Matrix @@ -133,7 +148,7 @@ Below is an example command to launch the TensorRT LLM server with GLM-5 from wi ```bash trtllm-serve \ - /models/GLM-5-FP8 \ + /models/GLM-5.3 \ --host 0.0.0.0 \ --port 8000 \ --max_batch_size 128 \ @@ -150,6 +165,40 @@ trtllm-serve \ > - Reduce `--max_num_tokens` to `3072` for max-throughput configs. > - Reduce `--max_batch_size` and `cuda_graph_config.max_batch_size` to `32` or `16` for min-latency configs. +### Disaggregated Serving + +GLM-5 supports disaggregated serving, which separates the prefill (context) and decode (generation) phases onto different workers so each can be scaled and tuned independently. This can be combined with MTP. + +Launch one or more context (prefill) servers and one or more generation (decode) servers with `trtllm-serve`, then start a `trtllm-serve disaggregated` orchestrator that routes client requests between them. Both context and generation workers must carry the same `cache_transceiver_config` to exchange the KV cache. Add the following to the config file used by each worker: + +```yaml +cache_transceiver_config: + backend: NIXL +``` + +Reuse the FP8 config (or the FP8-with-MTP config for Disagg + MTP) from [Recommended Performance Settings](#recommended-performance-settings) for the workers, adding the `cache_transceiver_config` block above. The context workers should also set `disable_overlap_scheduler: true`. + +The orchestrator is launched with a disaggregated config that lists the context and generation worker URLs: + +```bash +trtllm-serve disaggregated -c disagg_config.yaml +``` + +```yaml +hostname: localhost +port: 8000 +context_servers: + num_instances: 1 + urls: + - "localhost:8001" +generation_servers: + num_instances: 1 + urls: + - "localhost:8002" +``` + +Clients then send OpenAI-compatible requests to the orchestrator (`localhost:8000`). For the full walkthrough, per-worker GPU placement, and multi-node/SLURM launch, see the [Disaggregated Serving guide](../features/disagg-serving.md). + ### LLM API Options (YAML Configuration) These options provide control over TensorRT LLM's behavior and are set within the YAML file passed to the `trtllm-serve` command via the `--config` argument. @@ -209,7 +258,7 @@ After the TensorRT LLM server is set up and shows *Application startup complete* curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ - "model": "zai-org/GLM-5-FP8", + "model": "zai-org/GLM-5.3", "prompt": "What is the capital of France?", "max_tokens": 16, "temperature": 0 @@ -222,7 +271,7 @@ Example response: { "id": "cmpl-...", "object": "text_completion", - "model": "zai-org/GLM-5-FP8", + "model": "zai-org/GLM-5.3", "choices": [ { "index": 0, @@ -264,7 +313,7 @@ result_dir=/tmp/glm5_output for concurrency in ${concurrency_list}; do num_prompts=$((concurrency * multi_round)) python -m tensorrt_llm.serve.scripts.benchmark_serving \ - --model zai-org/GLM-5-FP8 \ + --model zai-org/GLM-5.3 \ --backend openai \ --dataset-name "random" \ --random-input-len ${isl} \ diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md index 7e8fb78e6e3a..a1c54a7ae8cb 100644 --- a/docs/source/models/supported-models.md +++ b/docs/source/models/supported-models.md @@ -24,7 +24,7 @@ The following is a table of supported models for the PyTorch backend: | `Gemma4AssistantForCausalLM` | Gemma 4 MTP assistant | `google/gemma-4-E2B-it-assistant`, `google/gemma-4-E4B-it-assistant`, `google/gemma-4-26B-A4B-it-assistant`, `google/gemma-4-31B-it-assistant` | | `Glm4MoeForCausalLM` | GLM-4.5, GLM-4.6, GLM-4.7 | `THUDM/GLM-4-100B-A10B` | | `Glm4MoeLiteForCausalLM` [^5] | GLM-4.7-Flash | `zai-org/GLM-4.7-Flash` | -| `GlmMoeDsaForCausalLM` | GLM-5 | `zai-org/GLM-5` | +| `GlmMoeDsaForCausalLM` | GLM-5, GLM-5.2, GLM-5.3 | `zai-org/GLM-5.3` | | `GraniteForCausalLM` [^5] | Granite 3, Granite Guardian 3 | `ibm-granite/granite-3.1-8b-instruct`, `ibm-granite/granite-3.3-8b-instruct`, `ibm-granite/granite-guardian-3.2-5b` | | `GraniteMoeHybridForCausalLM` [^5] | Granite 4.0 Hybrid MoE | `ibm-granite/granite-4.0-h-small` | | `GptOssForCausalLM` | GPT-OSS | `openai/gpt-oss-20b`, `openai/gpt-oss-120b` | @@ -72,6 +72,7 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl | -------------------------------- | ----------------- | ---------- | -------------------------- | --------------------- | --------------- | --- | ---------------- | ----------------- | ------ | ------------- | ---------------- | -------------- | ------------------------ | --------------------- | --------------- | | `DeepseekV3ForCausalLM` | Yes | Yes | Yes | Yes | Yes [^1] | Yes | No | No | No | Yes | Yes | Yes [^2] | N/A | Yes | Yes | | `DeepseekV32ForCausalLM` | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes | N/A | Yes | Yes | +| `GlmMoeDsaForCausalLM` | Yes | Yes | Untested | Yes | Yes | Yes | No | No | No | Yes | Untested | Untested | N/A | Untested | Untested | | `DeepseekV4ForCausalLM` [^11] | Yes | Yes | Yes | Untested | Yes | Yes | No | No | No | Yes | Yes | Untested | Yes | Untested | Untested | | `Glm4MoeForCausalLM` | Yes | Yes | Yes | Untested | Yes | Yes | No | No | No | Yes | Yes | Untested | N/A | Yes | Yes | | `Qwen3MoeForCausalLM` | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | No | Yes | Yes | Yes | N/A | Yes | Yes |