RELEASE pre-fragments 2026-05-24
2026-05-24 llm-d#1134 EPP now strictly parses plugin configurations — unknown fields cause plugin initialization to fail with a clear error rather than being silently ignored. Deprecated fields continue to be accepted with a warning until they are removed.
2026-05-24 llm-d#1079 deprecated UDS-backend in token-producer
2026-05-23 llm-d#1277 New helm charts for deploying the router in standalone and gateway modes, check https://github.com/llm-d/llm-d-router/tree/main/config/charts for the detailed docs.
2026-05-22 llm-d#1071 Introduce standardized EPP metrics under the llm_d_router_epp subsystem and mark legacy metrics as deprecated.
2026-05-21 llm-d#1112 Adds precise-prefix-cache-producer DataProducer plugin. Configure alongside prefix-cache-scorer with prefixMatchInfoProducerName: precise-prefix-cache-producer to opt into precise KV-block scoring; the existing monolithic precise-prefix-cache-scorer is unaffected.
2026-05-20 llm-d#1229 Helm charts now require EPP CPU and memory requests to be set. Default values provided (4 CPU, 8Gi memory).
2026-05-19 llm-d#1208 The Problem: By default, the Helm chart configures the EPP to use the openai-parser. When standard ML workloads send non-LLM payloads (e.g., KServe v2 inputs arrays) to a standard Triton instance, the openai-parser fails to unmarshal the payload, resulting in a 400 Bad Request and routing failure. The Proposed Change: This PR updates the Helm template logic to intelligently default to the passthrough-parser when standard ML servers are used. - Modifies _config.yaml to automatically inject passthrough-parser if modelServerType is set to triton (unless explicitly overridden by the user). - Updates public documentation (README.md and values.yaml) to officially list triton as a valid modelServerType option. Validation: - Added explicit assertions to hack/verify-helm.sh to ensure passthrough-parser is injected into the rendered EndpointPickerConfig when testing the triton model server type. - All Helm validation tests (make verify-helm-charts) pass successfully.
2026-05-19 llm-d#1207 The Problem: Currently, the Endpoint Picker (EPP) auto-populates metric extraction mappings only for LLM-specific engines (vllm, sglang, triton-tensorrt-llm, etc.). Users running standard Machine Learning workloads on the NVIDIA Triton Inference Server must write complex custom EndpointPickerConfig overrides to map the queue metrics, or the queue-scorer breaks. The Proposed Change: This PR introduces native support for standard Triton in the Go codebase. - Adds a triton configuration to defaultEngineConfigs. - Maps QueuedRequestsSpec to nv_inference_pending_request_count. - Maps RunningRequestsSpec to nv_inference_exec_count. - Explicitly leaves LLM-specific features (like KV Cache and LoRA) empty to prevent scraping errors. Validation: - Added TestDefaultEngineConfigsTritonValues to explicitly assert the accuracy of the metric string mappings. - Confirmed TestCoreMetricsExtractorFactoryDefaultEngine/defaultEngine_triton passes successfully.
2026-05-21 llm-d#1155 Add CLI flags --grpc-max-recv-msg-size, --grpc-max-send-msg-size, and --grpc-enable-compression to EPP to support large multi-modal payloads and over-the-wire compression.
2026-05-18 llm-d#1171 EPP scheduler hot path now allocates ~90% less per request on large fleets (100+ endpoints) and runs ~50% faster. Behavior is unchanged; the optimization eliminates per-endpoint allocations that occurred even when DEBUG logging was disabled.
2026-05-14 llm-d#683 Add sidecar configuration specified via either --configuration-file <path> or --configuration <text>. Both YAML and JSON formats are allowed. Configuration field names are the same as the CLI flags. CLI has priority over the configuration file or text.
2026-05-17 llm-d#822 Introduce chunked decode via a sidecar CLI flag, splitting long token generation into multiple bounded iterations. To enable add --decode-chunk-size=<size>. The chunk size should relatively large (512 or more) and be a multiple of the block size.
2026-05-14 llm-d#916 Use prefiller cached token usage for NIXL P/D responses.
2026-05-13 llm-d#959 Add probabilistic-admitter plugin for binary-tier probabilistic load shedding of low-priority inference requests under cluster saturation.
2026-05-13 llm-d#890 Adds new configuration option for the tokens-producer plugin. Existing configurations are unaffected.
2026-05-07 llm-d#931 active-request-scorer now uses inflight-load-producer for in-flight request tracking; requestTimeout is deprecated and ignored, and idleThreshold/maxBusyScore configure idle/busy endpoint scoring.
2026-05-05 llm-d#899 Add datalayer_poll_errors_total and datalayer_extract_errors_total Prometheus counters; remove transition-only error logging.
2026-05-05 llm-d#972 The use of inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated. Please use llm-d.ai/v1alpha1/EndpointPickerConfig instead
2026-05-05 llm-d#863 Deprecating plugin name tokenizer - added token-producer.
2026-05-03 llm-d#941 New configuration parameter in Data section - InjectDefaults controls automatic injection of the default metrics source and extractor. Defaults to true when omitted. Set to false to disable all default source injection.
2026-04-30 llm-d#922 The following Kubernetes label/annotation keys have been migrated from the GAIE prefix to llm-d.ai: - Pod annotation: inference.networking.k8s.io/active-ports is now llm-d.ai/active-ports - Pod label: inference.networking.k8s.io/engine-type is now llm-d.ai/engine-type Both legacy GAIE keys are still checked as fallbacks during the transition period but are deprecated and will be removed in a future release.