TensorRT Edge-LLM 0.4.0 Release

[nvluxiaoz]

TensorRT Edge-LLM 0.4.0 Release 2026-01-06

We are very excited to announce the first release of TensorRT Edge-LLM! TensorRT Edge-LLM is NVIDIA's high-performance C++ inference runtime for Large Language Models (LLMs) and Vision-Language Models (VLMs) on embedded platforms. Please follow our Quick Start Guide for the usage.

Key Components

• Flexible Python CLI exporter from HuggingFace checkpoints to ONNX
• C++ TensorRT engine builder
• C++ tokenizers and multi-modal processors
• C++ runtime, including vanilla decoding and EAGLE3 speculative decoding
• Optimized CUDA kernels for Multi-Head Attention, Sampling and EAGLE3 utility
• Examples to run inference and perform accuracy evaluations and perf benchmarks

Model Support

• Llama3.x
• Qwen2/2.5/3 (Dense)
• Qwen2/2.5/3-VL (Dense)
• InternVL3
• Phi4-Multimodal

Please check the model support page for more details.

Key Features

Model Export

• nvfp4/fp8/int4 quantization
• nvfp4/fp8 lm_head quantization
• EAGLE3 draft quantization
• Vocab reduction

Runtime

• Multi-batch EAGLE3 speculative decoding for LLM and VLM
• Decoding CUDA Graph
• System prompt KVCache reuse
• Open-AI style chat template
• Dynamic LoRA switching

---

This discussion was created from the release TensorRT Edge-LLM 0.4.0 Release.