Lfm/combined fixes - #7
Open
Lucas-Fernandes-Martins wants to merge 379 commits into
Open
Conversation
…llm-d/llm-d-kv-cache#444) * feat: multimodal-aware prefix-cache routing via extra_keys Incorporate per-block multimodal extra_keys (mm_hashes) into KV block hash computation so different images with identical token IDs produce distinct cache entries. This enables correct prefix-cache-aware routing for multimodal requests. Ingestion path: ParseRawExtraKeys converts vLLM's [][]any extra_keys from BlockStored events into typed BlockExtraFeatures, passed through TokensToKVBlockKeys to taint block hashes. Request path: RenderChatCompletion now returns MultiModalFeatures from the tokenizer. ComputeBlockExtraFeatures maps mm_hashes + placeholder ranges to per-block features matching vLLM's algorithm, ensuring request-side and engine-side block hashes agree. BlockStoredEvent and the vLLM adapter are unchanged — conversion from raw [][]any happens at the pool processing layer. Co-Authored-By: Atharva Pakade <pakade310@gmail.com> * style: fix lint issues in extra_keys and token_processor * bench: add benchmarks for MM extra_keys paths * test: add MM pipeline integration test (adapter → index → lookup) * test: add multimodal E2E tests for UDS tokenizer pipeline Tests against a real tokenizer service with Qwen2-VL-2B-Instruct: - MM features returned with valid placeholder ranges - Block feature assignment matches placeholders exactly - Determinism: same image+prompt → same keys - Different images → different content hashes and block keys - Text blocks before image unaffected by MM tainting - Full index round-trip: ingest with MM keys, lookup matches, different image does not match * test: cross-validate against real vLLM-captured multimodal msgpack The captured file (block_stored_example.msgpack) was recorded from a live Qwen/Qwen2-VL-2B-Instruct two-image inference. The test verifies that ParseRawExtraKeys (ingestion path) and ComputeBlockExtraFeatures (request path) produce identical per-block features and block keys for all 117 blocks. * style: fix lint in benchmarks and integration tests * test: trim redundant tests (benchmarks, synthetic pipeline, round-trip) Removed: - extra_keys_bench_test.go (benchmarks — nice to have, not correctness) - mm_pipeline_test.go (synthetic — vLLM capture test is stronger) - 4 redundant unit tests covered by vLLM capture cross-validation * style: fix gosec G115 int-to-uint32 in test token generation * fix: warm up MM model before first image request in e2e tests The first switchTokenizer to Qwen2-VL downloads tokenizer files, which can exceed the 5s gRPC deadline. A text-only warmup request after switchTokenizer ensures the model is loaded before sending image requests. * fix: eagerly warm up renderer on model load Send a minimal text request during load_renderer to force all lazy HuggingFace downloads (image processor configs, etc.) upfront. This prevents the first real request from hitting download latency and exceeding gRPC deadlines, especially for multimodal models. * fix: warm up renderer eagerly in NewUdsTokenizer Send a minimal RenderChatCompletion request during NewUdsTokenizer setup to force any lazy HuggingFace downloads (image processor configs for multimodal models). This prevents the first real request from exceeding gRPC deadlines. Reverts the Python-side warmup — the Go client now owns readiness. * test: move vLLM capture test to pkg/kvcache/kvblock so CI runs it unit-test-uds runs 'go test ./pkg/...' — tests/integration/ was not covered. Moving the test ensures it runs in the unit-test CI job. * refactor: simplify Tokenizer interface — only RenderChat returns MM features Keep Render() signature unchanged from upstream: ([]uint32, []types.Offset, error). Only RenderChat() changes: returns ([]uint32, *MultiModalFeatures, error) instead of ([]uint32, []types.Offset, error). Removes RenderResult wrapper type. This avoids touching all Render() callers across e2e tests and embedded tokenizer, reducing diff churn by ~195 lines. * style: fix errcheck and gocritic in pool_test mock methods --------- Co-authored-by: Maroon Ayoub <maroon.ayoub@ibm.com>
…-kv-cache#481) Multimodal requests need to download and process images, which can exceed the 5s default timeout in CI. Use 30s for requests with structured content parts.
with vLLM event schema (llm-d/llm-d-kv-cache#484) * fix: single-pass []any decode for forward/backward compat with vLLM event schema vLLM uses msgspec with array_like=True and omit_defaults=True, producing positional msgpack arrays where trailing fields may be absent. The previous typed-struct decode broke when vLLM appended new fields (old consumer fails) or when a newer consumer read from an older vLLM (shorter array than expected). Replace double-decode ([]any for tag + typed struct) with a single unmarshal into []any and positional extraction with length guards. Extra trailing fields from newer vLLM are silently ignored; missing trailing fields from older vLLM get zero values. * test: add decode benchmarks for vLLM event schema compat * fix: address review — reuse shared helpers, fix lint - Reuse convertBlockHashes() from common.go in both vLLM converters - Move convertExtraKeys() back to common.go (shared with SGLang) - Remove unused engineName param from decodeEvent() - Fix bench test lint: error checks, paramTypeCombine, appendCombine * fix: suppress gosec G115 in bench test data construction
* - fix: align MM extra_keys parsing with vLLM v0.18.0 bare-string format - fix blocksize alignment Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com> * fix lint & test Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com> --------- Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com>
…kv-cache#492) binary.BigEndian.Uint64(seqBytes) panics if seqBytes is fewer than 8 bytes. The frame count is validated but individual frame lengths are not. A malformed message from a ZMQ publisher would crash the entire process since the goroutine has no recover. Add a length guard that logs and skips messages with truncated sequence frames. Fixes llm-d#491 Signed-off-by: wenhug <50309350+wenhug@users.noreply.github.com>
* feat: Support heterogeneous block sizes in TokenProcessor Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * feat: Read path support for canonical block size Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * feat: Write path and eviction with canonical block size normalization Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Remove comment from cherry-pick Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Add new tests cases and verify back compat Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * refactor: Extend Index interface with 1:many engine-to-request key mapping Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * refactor: Make Pool stateless, delegate engine-key mapping to Index Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(linter): Fix line length and nesting violations * refactor: Keep original TokenProcessor Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * refactor: Infer engine-request key mapping from array lengths Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Add Index.Add mapping tests Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * docs: Add better comments Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Deduplicate engine-key mappings Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Use Redis sorted sets for ordered engine-key mapping Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Fix linter issues Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Realign extraFeatures to canonical block size Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Remove linter directives Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
…llm-d/llm-d-kv-cache#525) The default podLabelSelector used camelCase (inferenceServing) which does not match the canonical kebab-case label (inference-serving) used across the llm-d org deployment guides and Pod manifests. This aligns the code default and documentation with the org-wide convention. Signed-off-by: Kay Yan <kay.yan@daocloud.io>
* remove preprocess pkg Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove embedded flag Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove embedded pool and tokenizer Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * update makefile Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove embedded from dockerfile Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * update comment Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove cgo lint Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * update arch Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * update git ignore Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * comment Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * Update docs/architecture.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sage <80211083+sagearc@users.noreply.github.com> Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * Update pkg/tokenization/uds_tokenizer.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sage <80211083+sagearc@users.noreply.github.com> Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove e2e tests artifacts Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * unused import Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * fix(build): produce binary in build-uds target Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> --------- Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> Signed-off-by: Sage <80211083+sagearc@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Guangya Liu <gyliu513@gmail.com>
…ache#509) * fix: register MaxPodHitCount metric in Collectors() MaxPodHitCount is defined and used in instrumented_index.go but was not included in the Collectors() slice, so it was never registered with the Prometheus registry. This made the metric invisible to scraping. Also add MaxPodHitCount to the periodic logMetrics() output and add test coverage to prevent future regressions. Fixes llm-d#508 Signed-off-by: Wenhan Guo <whguo@ucsd.edu> Signed-off-by: wenhug <50309350+wenhug@users.noreply.github.com> * fix: address review comments on collector tests - Detect duplicate collectors in TestCollectorsIncludesAllMetrics to guard against MustRegister panics at runtime - Assert exact metric value (max_pod_hit_count=42) instead of only checking field presence Signed-off-by: Wenhan Guo <whguo@ucsd.edu> Signed-off-by: wenhug <50309350+wenhug@users.noreply.github.com> --------- Signed-off-by: Wenhan Guo <whguo@ucsd.edu> Signed-off-by: wenhug <50309350+wenhug@users.noreply.github.com>
…-d-kv-cache#543) * chore: deprecate internal tokenization (phase 1 of llm-d#517) Mark the prompt-string indexer APIs (GetPodScores, ComputeBlockKeys, SetTokenizer) and the internal tokenization.Pool as Deprecated. Make TokenizersPoolConfig optional in the indexer: when nil, no pool is created and the deprecated entry points return an explicit error pointing callers at ScoreTokens. Add Indexer.ComputeBlockKeysFromTokens as the tokens-in counterpart of ComputeBlockKeys. Drop the auto-populated TokenizersPoolConfig from NewDefaultConfig so new integrations opt out by default. Examples and e2e suites that still demonstrate the prompt-string flow opt in explicitly via a new helper.ConfigureInternalTokenizer. Architecture and configuration docs carry deprecation callouts that redirect to the tokens-in path. * address PR llm-d#543 review feedback - Export ErrInternalTokenizationDisabled so callers can detect the missing-pool case via errors.Is. - docs/architecture.md: drop the obsolete CachedLocalTokenizer / CachedHFTokenizer / CompositeTokenizer subsection (those backends were removed in llm-d#473) and distinguish EPP-side tokenization from the indexer-side UDS sidecar. - docs/architecture.md: AllBlocksCleared description and sequence diagram now reflect the current implementation, which logs the event but does not yet clear per-pod entries. - examples/kv_cache_aware_scorer (build-excluded): drop the dead HFTokenizerConfig setup that referenced types removed in llm-d#473.
* Replace BlockSize with BlockSizeTokens Signed-off-by: roytman <roytman@il.ibm.com> * fix lint and e2e test errors Signed-off-by: roytman <roytman@il.ibm.com> * fix comments Signed-off-by: roytman <roytman@il.ibm.com> --------- Signed-off-by: roytman <roytman@il.ibm.com>
…he#561) * fix issues in cpu offloading * fix for a Redis-specific edge case * addressed racing issues and made DeviceTier consistent * update document on CPU Offloading for precise-prefix-cache-aware * update document on CPU Offloading * fix lint issue * add more details on doc * add note on event ordering guarantee * add note for storage to CPU path
…v-cache#586) Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
…m-d/llm-d-kv-cache#577) * fix(tokenization): preserve assistant tool calls in UDS rendering Keep assistant tool calls on the Go-to-Python UDS rendering path so chat templates can render tool-call turns correctly while keeping regression coverage focused on that boundary. Signed-off-by: Kay Yan <kay.yan@daocloud.io> * fix(tokenization): address tool calls review feedback Signed-off-by: Kay Yan <kay.yan@daocloud.io> --------- Signed-off-by: Kay Yan <kay.yan@daocloud.io>
* parse hma kv event metadata Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> Co-authored-by: Kapil Jain <16477749+kapiljain1989@users.noreply.github.com> * reduce noise Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> --------- Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> Co-authored-by: Kapil Jain <16477749+kapiljain1989@users.noreply.github.com>
Signed-off-by: Dong Ma <winterma.dong@gmail.com>
…lm-d/llm-d-kv-cache#619) * feat(evictor): background empty directory cleanup Empty cache directories accumulate as files are evicted. This adds a background folder-cleaner process (P(N+3), gated by ENABLE_DIR_CLEANUP) that removes them. How it works: - The crawler detects empty rank/{hhh}/{hh} directories during its sweep and the deleter offers each freshly-emptied parent directory after a batch delete. Both feed a shared folder_queue. - The folder cleaner pulls paths off the queue and removes them with os.rmdir, which is inherently safe: it is a no-op if a file has landed in the directory in the meantime. Safety: - queue_folder skips directories modified within DIR_CLEANUP_TTL_SECONDS (default 120s) so we don't race a writer that just created a bucket and is about to populate it. This is defense-in-depth on top of rmdir's empty-only semantics. Config / Helm: - New ENABLE_DIR_CLEANUP (default true) and DIR_CLEANUP_TTL_SECONDS (default 120) env vars, wired through config.py, the Helm values and Deployment template, and documented in CONFIGURATION.md. Reporting: - The folder cleaner reports folders_purged via a new folder_cleaner_stats channel surfaced in the aggregated log. The crawler's per-sweep counter is named empty_folders_queued to reflect that it counts directories handed to the cleaner, not directories it deleted itself. The deleter's progress/done result-queue protocol is left unchanged. Signed-off-by: Miro <mironikolov@google.com> * Fix pvc_evictor unit tests due to delete_file_batch signature update Signed-off-by: Miro <mironikolov@google.com> * test: merge empty directory cleanup unit tests from PR llm-d#625 Signed-off-by: Miro <mironikolov@google.com> * fix(test): use moby container package instead of docker docker package to fix testcontainers-go build error Signed-off-by: Miro <mironikolov@google.com> * fix(ci): update golangci-lint configuration format Signed-off-by: Miro <mironikolov@google.com> * fix(kvblock): resolve ZRevRange deprecation and lll warnings Signed-off-by: Miro <mironikolov@google.com> * fix(test): resolve lll and unused gocritic lint warnings in uds_e2e_suite_test.go Signed-off-by: Miro <mironikolov@google.com> * Fix the linting errors Signed-off-by: Miro <mironikolov@google.com> --------- Signed-off-by: Miro <mironikolov@google.com>
…-d-kv-cache#627) * add hma group identity to kvblock entries Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * learn hma groups from kv events Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * expose hma group catalog Expose the learned group catalog so scorer follow-up work can use the event-derived metadata. Co-authored-by: Kapil Jain <kapiljain1989@gmail.com> Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * fix redis pod entry encoding Handle JSON encoding errors and store PodEntry directly for runtime Redis index state. Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * test namings Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * use only redis field Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * encode decode func namings for redis Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove redundant TestPodEntryString after redis keys change Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> * remove noise from git diff Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> --------- Signed-off-by: Sage Ahrac <sagiahrak@gmail.com> Co-authored-by: Kapil Jain <kapiljain1989@gmail.com>
…vability pattern (llm-d/llm-d-kv-cache#653) Signed-off-by: Guangya Liu <gyliu513@gmail.com>
…cal blocks (llm-d/llm-d-kv-cache#645) Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
…te (llm-d/llm-d-kv-cache#626) * feat(kvblock): invalidate KV index on AllBlocksCleared via eager delete Wire the AllBlocksCleared KV-event to a new Index.Clear(podIdentifier) that eagerly removes the pod's entries across all device tiers. vLLM emits AllBlocksCleared only from reset_prefix_cache() — a pod-wide prefix-cache reset (with no tier) that, in RL rollouts, fires once per weight sync inside a drain/barrier (the engine is paused and the cache is cold afterward regardless). That cadence and slack make an O(N) clear off the hot path the right trade: it adds zero steady-state cost to the Lookup/Add paths (the index's hottest path), unlike a per-entry generation filter. - Index.Clear(ctx, podIdentifier): pod-wide, all tiers. A string arg (not a PodEntry) so the signature cannot imply tier scoping the event never carries. - InMemory: iterate keys, reuse evictPodsFromRequestKey for race-safe removal. - CostAware: add a keyIndex set (ristretto has no iteration); prune on clear/evict. - Redis/Valkey: SCAN + HDEL the pod's "<pod>@" fields + prune empty hashes; deletes from the shared store, so it is correct across replicas without cross-process coordination. - pool.go: AllBlocksCleared now calls index.Clear (was a no-op log). - Shared interface tests: basic clear, pod isolation, all-tiers, re-add. Co-Authored-By: Yashwant <35724199+yash9263@users.noreply.github.com> Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com> * fix(kvblock): bound keyIndex and keep Clear off the Lookup hot path CostAware Clear held mu for the whole O(N) scan, blocking every Lookup for the duration; chunk the scan so each mu hold is bounded. keyIndex grew with every key ever added because ristretto evictions had no prune hook, defeating the cost bound; prune it via OnEvict/OnReject under a dedicated keyIndexMu (mu would deadlock against Add's data.Wait()). Log when AllBlocksCleared carries a DeviceTier so a future tier-scoped reset does not over-wipe silently. Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com> --------- Signed-off-by: Maroon Ayoub <maroon.ayoub@ibm.com> Co-authored-by: Yashwant <35724199+yash9263@users.noreply.github.com>
…ibility) (llm-d/llm-d-kv-cache#661) * fix(kvevents): support map-encoded vLLM KV events vLLM dropped msgspec array_like=True from its KV cache event structs (vllm-project/vllm#42892, merged 2026-06-09), so newer vLLM versions publish each event as a field-name map with the tag under the "type" key instead of a positional array. The VLLMAdapter only decoded positional arrays, so every KV event from a new vLLM fails to parse and the whole index goes dark. Normalize map-encoded events to the existing positional layout in decodeVLLMEvent before dispatch, keeping the converters and their forward/backward-compatibility guards encoding-agnostic. Positional arrays from older vLLM versions keep working unchanged; absent map fields become nil exactly like omitted trailing array fields. Verified against a captured event stream from a live vLLM serve run (map encoding, multi-turn traffic with CPU offload store and eviction events): all 313 frames parse, and replaying them through the kvevents.Pool -> kvblock.InMemoryIndex path indexes and evicts every block correctly. Unit tests cover map-encoded BlockStored / BlockRemoved / AllBlocksCleared, a mixed-encoding batch, and malformed map events. Assisted-by: Claude (AI assistance for implementation and tests) Signed-off-by: Change72 <changg@nvidia.com> * fix(kvevents): distinct errors for malformed map-encoded events Address review: report a dedicated error for a missing "type" tag (instead of conflating it with the non-string case), drop the stale "tagged union" wording from the encoding-agnostic unmarshal error, and pin each malformed-map failure mode to its distinct error message in the test. Assisted-by: Claude (AI assistance for implementation and tests) Signed-off-by: Change72 <changg@nvidia.com> * fix(kvevents): tighten unmarshal error, drop unreachable map branch Address review: the event-level unmarshal failure now wraps as "unmarshal event payload" so ParseMessage's "failed to decode vLLM event" wrap no longer doubles the same prefix. Drop the map[any]any normalization branch: msgpack v5 decodes untyped maps via DecodeMap(), which only produces map[string]any and rejects non-string keys inside Unmarshal itself, so the branch was unreachable. Condense the encoding doc comments; the PR description carries the full background. Assisted-by: Claude (AI assistance for implementation and tests) Signed-off-by: Change72 <changg@nvidia.com> --------- Signed-off-by: Change72 <changg@nvidia.com>
…m-d-kv-cache#680) * feat(kvevents): reference-count duplicate KV-event removals vLLM's OffloadingConnector publishes self-describing block-granular KV events. In chunk mode, overlapping offloaded chunks legitimately re-announce the same constituent block hash, so a hash is Stored and Removed more than once on the wire. BlockRemoved was forwarded straight to index.Evict per hash, so the first such duplicate remove evicted a block a sibling chunk still referenced (premature eviction that under-credits the lower tier in routing). Add an eventDedupFilter in pkg/kvevents that mirrors the wire stream: every BlockStored increments a per-(pod, device tier, KV-cache group) reference count, a BlockRemoved is forwarded to the index only once that count returns to zero, unknown removes pass through defensively, and AllBlocksCleared resets a pod. This matches the dimensions of the PodEntry identity an eviction targets, so gpu/cpu copies and distinct groups of a hash are counted independently. The scope's data-parallel rank is a sentinel on current main because the index identity is pod-level and does not distinguish ranks, so counts aggregate across ranks; a future DP-aware index (llm-d/llm-d-kv-cache#370) can wire the real rank with no change to the filter. No changes to EventBatch or the engine adapters. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(kvevents): update pool state comment Signed-off-by: Change72 <changg@nvidia.com> * feat(kvevents): add dedup observability and tighten tests Address review feedback on the KV-event dedup filter: - Add kvcache_kvevents_dedup_removed_hashes_{suppressed,forwarded}_total counters (block-hash granularity, not event count) so suppressed removals are observable, following the pkg/kvcache/metrics global-collector pattern; emit a TRACE log when removals are suppressed. - Document why the filter lives in the Pool rather than as an Index decorator. - Tighten the concurrency doc: the mutex guards the cross-pod map and same-pod events are serialized by Pool.AddTask sharding. - Move the pool-level tests into pool_test.go and add a negative test that a device-tier update resolving no keys is not reference-counted. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(kvevents): restore AllBlocksCleared dedup-reset test The test relocation in the previous commit dropped TestPool_AllBlocksClearedResetsDedup instead of moving it into pool_test.go. Restore it next to TestAllBlocksCleared_Dispatch: it is the only regression guard that the AllBlocksCleared handler resets the dedup refcount (Dispatch only proves Index.Clear ran), so a post-clear store/remove cycle is not wrongly suppressed. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(kvevents): cover dedup counters and harden metrics registration guard Address second-round review feedback: - Add DedupRemovedHashes{Suppressed,Forwarded} to TestCollectorsIncludesAllMetrics so the registration-completeness guard fails if either counter is ever dropped from Collectors() (previously only the 9 pre-existing collectors were guarded). - Add TestPool_DedupMetricsCountBlockHashes verifying the counter arithmetic at block-hash granularity: two duplicate stores then two removes record 4 suppressed (first remove) and 4 forwarded (second). Values are read via the dto.Metric Write pattern already used by metrics.logMetrics, so no new dependency is introduced. - Document that the noGroupIdx (-1) sentinel cannot collide with a real group index, since the engine adapters reject a negative group_idx. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Signed-off-by: Change72 <changg@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lm-d/llm-d-kv-cache#670) * fix(kvcache): unify engine-to-request mapping and fix Redis scoring Extract duplicated proportional distribution logic into a shared engineToRequestMapping helper in index.go to reduce code duplication and improve maintainability. Update CostAwareMemoryIndex, InMemoryIndex, and RedisIndex to use the centralized function. Additionally, correct the Redis ZAdd score calculation to use the relative index within each mapping slice instead of the global loop counter, ensuring accurate positional scoring for request keys. Signed-off-by: Alex <alex.tech.lab@outlook.com> * fix(kvcache): guard engineToRequestMapping against empty slices and add tests Add defensive check for empty engineKeys/requestKeys to prevent index out of range panic. Add comprehensive unit tests covering 1:1, many:1, 1:many, empty slices, and score ordering scenarios. Signed-off-by: Alex <alex.tech.lab@outlook.com> * fix(kvcache): preallocate map capacity and document empty-input behavior Address Copilot review feedback on engineToRequestMapping: - Preallocate map with len(engineKeys) to avoid unnecessary rehashing - Add explicit doc comment stating empty-input returns empty map Signed-off-by: Alex <alex.tech.lab@outlook.com> --------- Signed-off-by: Alex <alex.tech.lab@outlook.com>
…lm-d#1896) * fix(tokenizer): support render tokenizer for Anthropic /v1/messages - Anthropic messages fall through to default in render backend produce() causing precise-prefix-cache not working Signed-off-by: Wen Zhou <wenzhou@redhat.com> * test(tokenizer): add tests for Anthropic /v1/messages render path Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Wen Zhou <wenzhou@redhat.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Wen Zhou <wenzhou@redhat.com> * fix: syntax for copilot Signed-off-by: Wen Zhou <wenzhou@redhat.com> * fix(tokenizer): convert Anthropic Messages payload for /render The HTTP path forwarded the raw request body to vLLM /render, which does not accept the Anthropic Messages schema, so the conversion never ran. Always rebuild the /render chat body from the typed struct. Update TestProduce_MessagesRequest to set a raw payload and assert that RenderChat receives the converted /render body rather than the raw one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Wen Zhou <wenzhou@redhat.com> --------- Signed-off-by: Wen Zhou <wenzhou@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* epp: add --drain-timeout for graceful shutdown
By default the ext_proc and health gRPC servers are manager runnables, so
on SIGTERM they are stopped at the same moment the manager releases the
leader lease. Any request Envoy still sends during that window (in-flight
streams, or a persistent connection not yet re-resolved by DNS) is
rejected.
With --drain-timeout > 0 the ext_proc and health servers are instead run
on a context that outlives the manager. On SIGTERM the EPP:
- flips a `draining` flag so readiness / ext_proc health report
NOT_SERVING (Kubernetes drains the pod from the Service endpoints),
while liveness stays SERVING to avoid a restart mid-drain;
- lets the manager stop, releasing the leader lease immediately so a
standby is promoted;
- keeps the ext_proc server accepting requests for the drain window,
then GracefulStops (draining in-flight streams).
The datastore's pod list survives manager shutdown, so endpoint
resolution keeps working during the drain (with stale metrics).
Default (0) is unchanged upstream behavior.
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
* Make the graceful shutdown the default path.
We activate the graceful shutdown by default (30s timeout) but, even if it’s set to 0, we use that path in the EPP all the time to simplify the code.
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
* fix the drain time in the end2end tests.
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
* Fix e2e test timeout.
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
* More e2e test fixes.
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
---------
Signed-off-by: Mathias Kende <mathias.kende@mistral.ai>
* test(e2e): add cache-affinity test for multimodal routing Adds a Ginkgo case under "Running multimodal cache-affinity configuration" that spins 2 decode pods, wires mm-embeddings-cache-producer + mm-embeddings-cache-scorer, and asserts two identical multimodal requests land on the same decode pod via cache-affinity scoring. Iterates over image, audio, and video in a single It block. Introduces swapToSimRenderSidecar so the EPP's loopback render endpoint runs the inference sim instead of text-only Qwen2.5-1.5B; the sim emits deterministic mm_features for all three modalities while real text-only vLLM does not. Detection is config-content based: any EPP config containing mm-embeddings-cache-producer triggers the sidecar swap. Bumps the pinned sim image to v0.9.2, which includes the audio and video mm_features stub from llm-d-inference-sim PR llm-d#539. Fixes llm-d#1541 Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * test(e2e): extend mm cache-affinity test with discrimination + metrics The original same-content loop passes if the scorer always picks pod 1. Three additional assertion blocks close that gap: - A/B image symmetry: each content key independently routes back to its first-served pod, proving scoring is keyed on content hash rather than "first request wins forever". - Mixed image+audio: per-item match accounting must drive cache affinity on a multi-modality request; probes the empty-hash short-circuit in ExtractMMItems. - Metric assertions on llm_d_router_epp_encoder_cache_{hits,queries}_total with the {modality} label, wrapped in Eventually because PreRequest's LRU write runs in a wg.Go goroutine. Helpers added: - runChatCompletionWithImageAndAudio for the mixed-modality body. - getMetricValue scrapes the EPP /metrics port-forward and sums matching series; sufficient for the small label set this test asserts on. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * test(e2e): address review on multimodal cache-affinity test Rename swapToSimRenderSidecar params (eppManifests/yamlDocs), extend the mixed request to image+audio+video, and note hits<queries in the metrics assertion. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * test(e2e): add estimate token-producer per-modality metric test Runs the mm cache-affinity config with the estimate backend and asserts encoder-cache metrics are labelled per modality (image/audio/video). Fails until the estimate modality fix (llm-d#1618) lands. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> --------- Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Wrap the render HTTP transport with otelhttp.NewTransport so that outbound /render requests carry the W3C traceparent header. This links the tokenizer-to-vLLM hop into the same trace span as the EPP request, matching the existing pattern used by the sidecar proxy transport. Signed-off-by: Mark Schulist <mschulist2@gmail.com>
* feat: preserve picker scores on ProfileRunResult The scheduler computes a weighted score per candidate endpoint, but the picker collapses the result to a plain endpoint list, so the score is lost at pick time. Add an additive, nil-safe ScoredEndpoints slice, parallel to TargetEndpoints, and populate it in the maxscore, random, and weightedrandom pickers. Existing readers of TargetEndpoints are unchanged. Part of llm-d#1843 Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com> * feat: emit per-endpoint scores in EPP dynamic metadata Carry the primary profile's endpoint scores from the scheduling result onto the request context and, when the EPP is started with --emit-endpoint-scores (off by default), emit them under the new x-gateway-destination-endpoint-scores key in the envoy.lb namespace so gateway providers integrating via the metadata path can observe how strongly each returned endpoint was preferred. The x-gateway-destination-endpoint field and header are unchanged, and with the flag off the dynamic metadata is byte-identical to today. Fixes llm-d#1843 Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com> * feat: record scored candidates in the scheduler profile Pickers narrow the candidate set to maxNumOfEndpoints before returning, so scores taken from a picker's selection describe only the endpoints that won, and under the default single-endpoint configuration that is a single score. Consumers cannot see how the alternatives compared, which is the runner-up visibility the scores are meant to provide. Record the scores in the scheduler profile instead, where the weighted score map covers every scored candidate. Pickers reorder and truncate the pointer slice but never the backing array, so the complete set is already in hand and needs no extra allocation. Score availability no longer depends on which picker is configured. Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com> * test: assert scores cover unselected candidates The metadata assertion required one score per destination endpoint, which holds only when scores come from the picker's selection. Scores now cover every scored candidate, so the destination endpoints are a subset and the runner-up's score is present without being a destination. Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com> * docs: drop nil condition from ScoredCandidates comment Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com> --------- Signed-off-by: Samuel Adelman <samuel.adelman@gmail.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
* epp: generalize endpoint identity Rename EndpointMetadata.PodName to Name. The endpoint model is not Kubernetes-only: file-discovered endpoints have no pod, and NamespacedName is the identity. Name is always set, so an endpoint has a name regardless of discovery source. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * epp: rename endpoint identity to ID Rename EndpointMetadata.NamespacedName to ID, the endpoint's unique identity, with GetID as the accessor. ID is a type alias for types.NamespacedName so the identity can later move off the Kubernetes-specific type without churning datastore keys. The discovery docs state that each source must set ID uniquely. Continues the endpoint-identity generalization from the PodName rename. EndpointMetadata is an internal framework type, so this is a plain rename with no deprecation shim. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> --------- Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…#2256) The NamespacedName to ID field rename in llm-d#2147 merged alongside other PRs that added EndpointMetadata{NamespacedName: ...} test usages. They landed without a textual conflict but broke the build (typecheck), turning main red for every open PR. Update the affected test field references to ID. Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
* ci: Add security workflow with Gosec, Gitleaks and CodeQL scanning Nothing in CI currently scans our the source. govulncheck covers known CVEs in dependencies and Trivy covers image layers but both look at code we pull in rather than code we write. Gosec finds insecure Go patterns, gitleaks finds secrets committed to history and CodeQL finds dataflow bugs that pattern matching misses. All three publish SARIF so findings land in the Security tab. Gosec and gitleaks are advisory for now so the baseline can be triaged before anything blocks a merge. CodeQL runs weekly rather than per PR because it is slow. Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * ci: Pin codeql action to SHA and keep scan SARIF as artifact Fork PRs run with a read only token, so the SARIF upload step is skipped and the reports were being thrown away. gosec writes only to the report file and logs nothing useful which left those runs with no way to see findings at all. Keep both reports as artifacts instead. The codeql action refs already had the resolved SHA sitting in a comment but still used a mutable tag. Pin to the SHA so a retargeted tag can't change what runs. Copilot review comments: - llm-d#2203 (review) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * Make checks consistent in security workflow Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * Run gosec through golangci-lint instead of a standalone job The securego/gosec container ships GOTOOLCHAIN=local with a Go older than go.mod requires, so package loading failed and every scan reported nothing. Running it through golangci-lint in the builder container gets the right toolchain. It also picks up the 6 //nolint:gosec directives already in the tree which standalone gosec ignores and it can gate fork PRs since failing a lint step needs no write token. Advisory for now. SECURITY_LINT_EXIT_CODE defaults to 0 so the 50 finding baseline can be triaged before this blocks merges. Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * Remove the security workflow Following discussions in llm-d#2203 (comment) it is decided to srip the security workflow as goign to use native CodeQL and Secret scanning. Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> --------- Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
* Add cross-replica state syncing for datalayer contributors Introduce a CrossReplicaSyncer backend (selected via crossReplicaSyncerPluginRef) and a CrossReplicaContributor opt-in for datalayer extractors. Each replica's local per-endpoint state is published to the syncer by a per-source interval PollingDispatcher, driven by the datalayer's existing per-endpoint Collector; consumers read the aggregate across replicas through a runtime-installed endpoint attribute. Publishing is decoupled from request handling, so state is written once per endpoint per interval rather than per request. A contributor can opt out via SyncDisabled (exposed by InFlightLoadProducer as the syncCrossReplicaState parameter). InFlightLoadProducer is the first consumer. Signed-off-by: Lucas-Fernandes-Martins <lucasfmartins16@gmail.com> * Make cross-replica sync interval configurable Signed-off-by: Lucas-Fernandes-Martins <lucasfmartins16@gmail.com> * Clarify SyncInterval zero-value doc comment Signed-off-by: Lucas-Fernandes-Martins <lucasfmartins16@gmail.com> --------- Signed-off-by: Lucas-Fernandes-Martins <lucasfmartins16@gmail.com>
…-EPP topology (llm-d#2253) * test(coordinator): rewire e2e onto shared inference-gateway component Signed-off-by: Revital Sur <eres@il.ibm.com> * test(coordinator): add 3-EPP topology option to e2e suite Signed-off-by: Revital Sur <eres@il.ibm.com> * Extract common resources. Signed-off-by: Revital Sur <eres@il.ibm.com> * Simplify Signed-off-by: Revital Sur <eres@il.ibm.com> * test(coordinator): substitute VLLM_RENDER_URL for e2e workers Signed-off-by: Revital Sur <eres@il.ibm.com> * Minor change. Signed-off-by: Revital Sur <eres@il.ibm.com> * Address review comments. Signed-off-by: Revital Sur <eres@il.ibm.com> * Fix test. Signed-off-by: Revital Sur <eres@il.ibm.com> --------- Signed-off-by: Revital Sur <eres@il.ibm.com>
…al-plugins flag (llm-d#1912) (llm-d#2073) * feat: implement EPP plugin stability lifecycle and experimentalPlugins feature gate (llm-d#1912) Signed-off-by: Cong Liu <conliu@google.com> * mark session affinity plugins alpha - they have not been used and are under major refactoring Signed-off-by: Cong Liu <conliu@google.com> * fix: restore sessionaffinity filter and scorer to StabilityBeta Signed-off-by: Cong Liu <conliu@google.com> * fix: set sessionaffinity filter and scorer to StabilityAlpha and enable experimentalPlugins in test Signed-off-by: Cong Liu <conliu@google.com> * Merge origin/main into plugin-stability-lifecycle Signed-off-by: Cong Liu <conliu@google.com> * test: enable experimentalPlugins featureGate for test configs using Alpha plugins Signed-off-by: Cong Liu <conliu@google.com> * feat: replace experimentalPlugins feature gate with --allow-experimental-plugins CLI flag Signed-off-by: Cong Liu <conliu@google.com> * feat: enforce experimental plugin check on auto-created system defaults and data producers Signed-off-by: Cong Liu <conliu@google.com> * test: remove experimentalPlugins feature gate reference from coordinator configs_test.go Signed-off-by: Cong Liu <conliu@google.com> * refactor: validate plugin stability once after all plugin initialization Signed-off-by: Cong Liu <conliu@google.com> * fix: remove obsolete activerequest filter package from branch (replaced by utilization filter in upstream llm-d#2218) Signed-off-by: Cong Liu <conliu@google.com> * refactor(runner): move plugin stability validation out of configuration parsing to run after all phases complete Signed-off-by: Cong Liu <conliu@google.com> * refactor(runner): remove unused opts parameter from parseConfigurationPhaseTwo Signed-off-by: Cong Liu <conliu@google.com> * docs & test: address PR llm-d#2073 review feedback Signed-off-by: Cong Liu <conliu@google.com> --------- Signed-off-by: Cong Liu <conliu@google.com>
… with maxPrefixTokensToMatch and blockSizeTokens (llm-d#2072) * docs/configs: replace deprecated maxPrefixBlocksToMatch and blockSize with maxPrefixTokensToMatch and blockSizeTokens Update sample deployment configurations, test fixtures, and documentation to use non-deprecated parameter names. Signed-off-by: Cong Liu <conliu@google.com> * docs/configs: calculate maxPrefixTokensToMatch from maxPrefixBlocksToMatch * blockSizeTokens Scale maxPrefixTokensToMatch accurately based on the block count and token block size. Signed-off-by: Cong Liu <conliu@google.com> * docs/operations: replace deprecated field name directly in sentence with maxPrefixTokensToMatch Signed-off-by: Cong Liu <conliu@google.com> * docs/operations: replace remaining sentence reference of maxPrefixBlocksToMatch with maxPrefixTokensToMatch Signed-off-by: Cong Liu <conliu@google.com> * docs/operations: rephrase CPU overhead sentence to state token caps and blockSizeTokens Signed-off-by: Cong Liu <conliu@google.com> * docs/operations: update performance benchmark tables to use maxPrefixTokensToMatch Signed-off-by: Cong Liu <conliu@google.com> * deploy/config: update maxPrefixTokensToMatch to 16384 to preserve 256-block cap with 64-token min block size Signed-off-by: Cong Liu <conliu@google.com> * docs/configs: fix precise-prefix-cache-producer schema and separate producer/scorer parameters - Fix precise-prefix-cache-producer schema in sim-epp-no-hit-lru.yaml and docs/architecture.md by moving tokenProcessorConfig to top level and removing invalid fields. - Separate approx-prefix-cache-producer parameters from prefix-cache-scorer in docs/disaggregation.md. - Update remaining legacy blockSize: 16 to blockSizeTokens: 16 in test/e2e/configs_test.go. - Rephrase operations doc to reflect 64-token minimum effective block size calculations. Signed-off-by: Cong Liu <conliu@google.com> --------- Signed-off-by: Cong Liu <conliu@google.com>
Bumps the github-actions group with 2 updates: [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) and [actions/stale](https://github.com/actions/stale). Updates `github/codeql-action/upload-sarif` from 4.35.4 to 4.37.4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@68bde55...f205ea1) Updates `actions/stale` from 10 to 11 - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](actions/stale@v10...v11) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: actions/stale dependency-version: '11' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lm-d#2262) Bumps the go-dependencies group with 1 update: [github.com/google/cel-go](https://github.com/google/cel-go). Updates `github.com/google/cel-go` from 0.29.0 to 0.29.2 - [Release notes](https://github.com/google/cel-go/releases) - [Commits](cel-expr/cel-go@v0.29.0...v0.29.2) --- updated-dependencies: - dependency-name: github.com/google/cel-go dependency-version: 0.29.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lm-d#2180) The flowControl feature gate stays disabled by default: saturation detection needs per-deployment tuning, so enabling the layer is an explicit decision by an operator who has read the tuning guidance. The graduation is everything around that decision: - Log lines no longer call the layer experimental. - The loader warns when a flowControl config section is present while the gate is off. Everything in the section except saturationDetector (which the legacy admission path also uses) was silently ignored, and with an opt-in gate that misconfiguration is easy to hit. - A test pins the registered default so changing it is deliberate. - Docs cover enablement, queue memory sizing, per-replica flow control state in Active-Active, and the dependency on the EPP's model-server metrics scrape staying fresh. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
) * perf: Reduce allocations on the response-body streaming path Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Fix linter issues Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Sharpen implementation Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
) pkg/telemetry.Tracer was a verbatim duplicate of pkg/common/observability/tracing.Tracer (the callers all pass an explicit scope, so the differing default name was never used); drop the package and point the kvcache tracing call sites at the observability package. Align their instrumentation scope strings with the router identity (llm-d-router/pkg/...). Move the generic SliceMap helper from pkg/utils to a new pkg/common/collections package. Fast follow to the kvcache/kvevents internalization (llm-d#1886). Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
* epp: add multicluster datalayer plugins Discovery, metrics source, and metrics extractor variants for a cluster-scoped EPP whose endpoints are peer clusters. - multicluster-file-discovery accepts an RFC-1123 hostname or an IP for a cluster gateway, via an injected address validator, so the stock IPv4-only pod discovery is unchanged. - multicluster-metrics-data-source scrapes a peer cluster. Because it crosses a cluster trust boundary it verifies the peer certificate by default and caps the response size. - multicluster-metrics-extractor writes only the pool-aggregate metrics into the llm-d.ai/multicluster-* attributes the scorers read. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * epp: add multicluster scheduling plugin variants Co-located variants beside their stock plugins, for a cluster-scoped EPP. - multicluster-kv-cache-utilization-scorer and multicluster-queue-scorer score on the llm-d.ai/multicluster-* pool-aggregate attributes, since the per-pod metric fields are empty on a cluster-endpoint. - multicluster-session-affinity-filter, multicluster-prefix-cache-scorer, and multicluster-approx-prefix-cache-producer delegate to their stock plugins. Cluster identity comes from discovery and the approx indexer already keys on endpoint identity, so it tracks (cluster, block) unchanged. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> * epp: register and document the multicluster plugin family Register the multicluster datalayer and scheduling variants in the runner, and document the family in the plugins README with a dependency note and a wiring example. The config-load test drives parseConfigurationPhaseTwo so it instantiates each plugin against the registry and asserts it resolves to the expected type; phase one alone would pass even with an unregistered or misspelled factory. Signed-off-by: Sam Batschelet <sbatsche@redhat.com> --------- Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
* docs: Define area/* label taxonomy Maps ten area labels to their directories so /area directives and future path based auto labeling (llm-d#956) agree on the same boundaries. Two labels (epp, dev) already exist. The rest still need to be created by a maintainer with repo write access. Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> * Update after review Review comments: - llm-d#2213 (review) Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> --------- Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…m-d#1834) * feat(epp): add OpenTelemetry spans for the scheduler scoring path Wrap the scorer chain in a parent llm_d.epp.scoring span and each scorer invocation in an llm_d.epp.scorer.<type> child span, so a request trace shows which scorers ran, how long they took, and aggregate score signals. Span attributes are request- and chain-level only (scorer type/name/weight, candidate count, score max/avg); no per-endpoint keys are emitted, keeping span cardinality bounded. Spans are no-ops when tracing is uninitialized. Signed-off-by: Matt Van Horn <mvanhorn@gmail.com> * docs: point multiple-base-models note at the in-page design section The linked upstream guide URL 404s, which fails the Check Markdown Links workflow. The same material is covered by this document's own InferencePool & InferenceModel Design section. Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * fix(epp): reuse TracerScope and unify the request-attribute block Review nits from @elevran: - The scoring path declared its own tracerScope constant that duplicated the exported TracerScope already in constants.go. Dropped the local one and pointed the tracer at the existing constant. - runFilterPlugins, runScorerPlugins and runPickerPlugin each built the same request-identity attributes inline. They now share one requestSpanAttributes helper returning []attribute.KeyValue, so the three call sites cannot drift. Emitted attributes are unchanged: same keys, same conditions, same values. Adds TestRequestSpanAttributes covering nil, empty, and populated requests. Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * test(epp): follow the EndpointMetadata.ID rename Upstream renamed EndpointMetadata.NamespacedName to ID. Update the scheduler profile tests to the current field so the package typechecks again; the scoring spans themselves are unchanged. --------- Signed-off-by: Matt Van Horn <mvanhorn@gmail.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* Add pre-scheduling candidate filter extension point Introduce a global request-control extension point that narrows the located endpoint pool before data producers, admission plugins, and scheduling profiles run. Compose multiple filters by intersecting independent results so configuration order does not affect correctness. Signed-off-by: Mathis Felardos <mathis@mistral.ai> * Fix candidate filter interface assertion Signed-off-by: Mathis Felardos <mathis@mistral.ai> * Address pre-scheduling filter review feedback Signed-off-by: Mathis Felardos <mathis@mistral.ai> * Rename candidate filter extension point to Screener Signed-off-by: Mathis Felardos <mathis@mistral.ai> --------- Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(epp): add session_id_header algorithm to session affinity scorer Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * refactor(epp): extract session affinity algorithms into a strategy interface Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * feat(epp): least-loaded placement for session_id_header strategy Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * fix(epp): close first-bind race and add miss threshold to session_id_header Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * fix(epp): guard eviction delete with CompareAndDelete on stale binding Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * refactor(epp): simplify leastLoadedPod to a plain argmin Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * refactor(epp): drop miss threshold, simplify comments in session affinity Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * fix(epp): close podCount race, rename strategy param, add session_id_header tests Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * refactor(epp): simplify session_id_header podCount updates Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * fix(epp): abstain from scoring when no session id is present Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * feat(epp): share session_id_header between affinity scorer and filter Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * feat(epp): per-strategy session affinity config with ordered sources Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * test(epp): integration coverage for session_id_header and affinity scorer Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * refactor(epp): rename session_id_header strategy to session_id Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> * fix(epp): abstain in session_id Choose when bound pod is unknown Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com> --------- Signed-off-by: Dasari Surya Sai Venkatesh <suryasai.venkatesh@gmail.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com> Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
* Add native DisaggregatedSet revision routing support
Introduces pkg/epp/disaggregation: a single Controller struct that owns
label-aware routing across the roles of a disaggregated inference
deployment (e.g. prefill / decode). Implements
requestcontrol.ResponseHeaderProcessor directly. Filter behaviour is
composed of three wrappers installed by WireInto:
- strict-mode selectors at the head of the profile's filter chain, so
no downstream filter ever sees a wrong-revision candidate;
- prefer-mode selectors at the tail, so their fallback captures the
fully-narrowed pool;
- an optional gating filter also at the tail, dropping candidates whose
revision fails the configured gating check.
No scorer is registered: picker choice is left to the operator's YAML,
and pod-count-proportional distribution among surviving revisions falls
out of any uniform sampling picker.
Config surface — one top-level disaggregation: YAML block:
```yaml
disaggregation:
enabled: true
scope:
labelSelector: "disaggregatedset.x-k8s.io/name=my-set"
selectors:
- name: revision
headerName: x-disagg-revision
labelKey: disaggregatedset.x-k8s.io/revision
mode: strict # strict | prefer
gating: # optional; sub-block chosen by mode
mode: sum # sum | disabled — room for future modes
requireRoles: # required when mode=sum; drops
labelKey: disaggregatedset.x-k8s.io/role
values: [prefill, decode] # candidates whose revision lacks any
# listed role's Ready pods
```
The same file works on every role's EPP; no per-role variation needed
for the common case. Gating.mode gives operators a way to disable the
filter (mode=disabled) while keeping the block for documentation, and
leaves room for additional algorithms without a new top-level key.
Why strict revision routing matters (not just a nice-to-have):
disaggregated inference technically routes fine without any of this.
DisaggregatedSet's placement policy runs at a layer below routing,
and you could argue routing across revisions during a rolling update
is harmless. In production it isn't. When rolling from revision A to
B, you run into two failure classes:
Revision B not backward-compatible with A — either new-revision pods
die outright when they receive a request meant for the old shape, or
they fail gracefully and the retry also fails, dropping requests and
tanking QoS.
Corrupted KVCache on revision A — vLLM produced a bad KV cache on
some requests on a bad version that had a bug, and you do not want
that cache carried into the newer revision's pods.
Both are cases where cross-revision routing during rollout is
actively harmful, which is why the revision selector runs in strict
mode. At the NVL72 level (stride) IB connections still exist between
NVL72 racks, so prefer-mode is the right knob for that selector.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* fix(disaggregation): golangci-lint issues
- gofmt: sort wire_test.go imports
- nilnil: return ErrDisabled sentinel from Register instead of (nil, nil)
- perfsprint: swap fmt.Errorf for errors.New on 8 static messages
- staticcheck: drop unused objs slice in startCache; apply De Morgan
on filter-order assert
- unparam: revLabels always passes "prefill" — drop the role arg
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(framework): expose Manager on plugin Handle; drop disaggregation's own informer
Plugins that need cluster-wide observations (e.g. cross-role pod counts)
had to construct their own kubernetes.Interface and SharedInformerFactory.
The disaggregation controller was the first such case (pkg/epp/disaggregation/
podcache.go). This adds a Manager() accessor to plugin.Handle following
the same pattern as MetricsRecorder (llm-d#1173): an optional WithManager()
option on NewEppHandle, wired by the runner after ctrl.NewManager returns.
Result: the disaggregation controller now attaches its Pod event handler
to the Manager's shared informer via mgr.GetCache().GetInformer(). No
second informer, no second k8s client, and boot-time coverage validation
runs as a Manager Runnable (fails mgr.Start on misconfig).
Framework touch points (small, additive, non-breaking):
- plugin.Handle: +Manager() accessor
- plugin.NewEppHandle: +WithManager() option
- Runner: NewDefaultManager moved before parseConfigurationPhaseTwo so
the Handle can carry the Manager to plugins that opt in
- File-discovery mode: passes nil (no k8s); plugins requiring mgr must
guard against nil accordingly
Disaggregation refactor:
- PodCache drops SharedInformerFactory + Start + WaitForCacheSync;
attaches to the Manager's cache and filters events by label selector
at handler time (the shared cache is unfiltered)
- Register drops the sync/wait boilerplate; the boot-time role coverage
check runs in a mgr.Add(RunnableFunc) after cache sync
- Register returns ErrDisabled for disabled configs (sentinel)
- Tests: seed the cache directly via upsert/remove instead of round-
tripping through a fake clientset + informer sync — smaller and ~4×
faster (0.4s vs 1.7s)
Net LOC: -32 across disaggregation-related files. Two Handle-stub test
helpers in unrelated plugins (multimodal, nohitlru) grow by 3 lines each
to satisfy the new interface method.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* refactor(disaggregation): delete PodCache, list from Manager cache on demand
PodCache maintained a mirrored pods+counts map fed by informer events so
HasRoleForRevision could answer in O(1) on the gating hot path. With the
Manager's shared cache already reachable via handle.Manager(), that
maintained state is redundant: List(scope) is a cache-backed read against
the same informer, cheap enough for a hot filter.
Controller now holds a client.Reader (mgr.GetCache()) + scope selector and
computes coverage on demand via one List per gating Filter call. Gating
memoizes per-revision within a call so cost scales with the number of
distinct revisions in play, not the size of the namespace or the number
of candidates.
Deleted:
- pkg/epp/disaggregation/podcache.go (190 LOC)
- pkg/epp/disaggregation/podcache_test.go (178 LOC)
- cache_ready_pods gauge + recordCacheSet / recordCacheDelete emitters
(operators can use kubectl for the same info; the gauge was always
a maintenance liability, not load-bearing observability)
Net: -264 LOC (+245 / -576) across 10 files. No behavior change for the
gating filter (same revision-drop invariant), no behavior change for
boot validation (same three failure modes surfaced through the same
Manager Runnable). One cache-backed List per gating call replaces the
event-driven count map.
Trade-off: gating cost becomes O(pods in scope) per unique candidate
revision per Filter call, rather than O(1). At target scales (few
hundred pods per namespace) this is a rounding error; if that ever
changes, add a FieldIndexer on (revision, role) to keep it O(k).
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(disaggregation): explicit per-revision weighted pick in gating
The gating filter now shapes traffic explicitly rather than relying on
the picker downstream to sample uniformly among prefill pods. Per Filter
call:
1. Compute per-revision cross-role pod count from the shared cache
List: weight(rev) = Σ Ready pods over gating.requireRoles.values.
2. Drop any revision with weight 0 (any required role missing pods —
rollout drift). Emit disagg_gating_dropped_total, one per revision.
3. Weighted-random-pick ONE surviving revision, chosen with probability
weight(rev) / Σ weight. Keep only that revision's candidates.
Downstream picker (random-picker, weighted-random-picker, whatever the
operator has in YAML) just picks uniformly among the chosen revision's
pods. Traffic converges on the intended cross-role share regardless of
picker choice — no coupling.
Why: under a 2p+18d (v1) / 1p+2d (v2) rollout the prior "gating drops
uncovered, picker chooses uniformly" behaviour gave 67% / 33% traffic
split (proportional to prefill pod count only). The correct target is
20/23 vs 3/23 → 87% / 13%, matching the cross-role capacity ratio. This
change closes that gap by computing the ratio explicitly in gating
rather than treating uniform-picker behaviour as an implicit oracle.
scanCoverage now returns per-role Ready pod COUNTS instead of a
presence boolean; validateRolesObserved uses crossRoleWeight > 0 as its
compound liveness check (same semantics — a role with a positive count
is present — but the counts are what gating.pickWeightedRevision reads
for its distribution).
Tests: existing gating tests updated for the new "one revision per
Filter" contract. New Monte-Carlo test locks the 2p+18d / 1p+2d case
to ~13% v2 over 10k iterations. rand01 is injected on the filter so
deterministic tests pin the pick outcome and a single-revision fast
path is verified not to consume the RNG.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* test(disaggregation): table-driven weighted-pick coverage for 3 shapes
Adds TestGatingFilter_WeightedPickMatchesLiveSlowTransitionShapes, a
Monte-Carlo table covering the middle-rollout step of each shape run by
scripts/verify-slow-transition.sh: 10p10d (balanced control), 2p20d
(decode-heavy) and 20p2d (prefill-heavy), plus the mirror step of the
two unbalanced shapes.
The 2p20d/20p2d step2 rows are the regression case flagged during
review: under the pre-weighted-pick behaviour they landed near 33% v2
share (prefill-count ratio), whereas the correct target is 13% (cross-
role weight ratio, 3/(20+3)). 10k iterations per case at ±3pp locks
the fix.
Consolidates the earlier single-case
TestGatingFilter_WeightedPickAcrossRolesLikeUsersExample into the same
table so we don't carry two overlapping tests.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* chore(disaggregation): un-export test-only helpers; inline gatingForLog
Self-review picked up two symbols that were exported but only ever
called from the package's own _test.go files:
- NewController — sole non-test call site is Register in the same
package. Renamed to newController.
- Controller.Filter — docstring already noted "test-only entry point"
(production wiring uses the modeSelectorsFilter and gatingFilter
wrappers via WireInto). Renamed to filter.
Also inlined gatingForLog: it was a five-line helper called exactly
once, from Register, to format one log field. Now inline at the
ctrllog.Info call.
Net: -2 LOC, but the real win is a smaller public API surface. Nothing
outside the package now reaches for Controller directly beyond the
Register / WireInto / ErrDisabled surface consumed by the runner.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* chore(disaggregation): restore gatingForLog helper
Reverting the gatingForLog inline from the previous chore commit. The
inline made the ctrllog.Info call awkward (a scratch variable declared
just to be a single log field). The five-line helper reads more
cleanly at the call site.
Keeping the other two changes from that commit (NewController and
Controller.Filter un-exports).
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* docs(disaggregation): address PR review comments — filter semantics, outcome labels, ResponseHeader lifecycle
Answers three review comments on PR llm-d#2141:
- filterSelectors switch: per-case comments explaining strict (replace
unconditionally, empty propagates) vs prefer (replace only when
matched is non-empty — the "prefer but tolerate absence" fallback).
- filterOutcomeMatched / NoMatchStrict / NoMatchPreferFallback:
per-constant docs describing what each label value signals to a
dashboard reader (matched = intersection kept, no_match_strict =
503, no_match_prefer_fallback = expected during rollouts).
- ResponseHeader: doc block explaining the lifecycle — the endpoint
metadata is threaded through by requestcontrol.director after the
picker chooses, not carried over from any filter. The stamped
values are the pod's own labels from EndpointSlice discovery.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(disaggregation): gating at head; skip when header pins or ≤1 revision
Moves the gating filter to the head of the scheduler filter chain
(prepended after strict, so it ends up in front of it), making the
revision-axis decision the very first step per request. Chain order
becomes:
gating → strict → operator → prefer
Only one of gating/strict actually makes the revision decision on any
given request; the other detects the situation and short-circuits.
Gating skips (passthrough, no cache List, no RNG) when:
1. The revision-axis strict header is set — strict downstream will
pin the revision. Gating firing here would either waste work or
stochastic-pick a revision that strict then rejects (503).
2. The candidate pool contains ≤1 unique revision — nothing to
shape, either because the fleet has one revision or an upstream
filter already narrowed.
Otherwise gating does its weighted-random-pick as before (weight per
revision = Σ Ready pod count over gating.requireRoles.values), drops
uncovered revisions, and keeps only the chosen revision's pods.
Everything downstream of the head now sees a single-revision candidate
pool. Operator-declared filters and prefer no longer walk multi-
revision pools when the decision has already been made.
Also renames scanCoverage → readyPodsByRevisionRole. The old name
implied "checks coverage" but the function also returns raw per-role
counts used by the gating weights; the new name describes the data
structure directly.
Tests:
- wire_test.go: order assertion flipped to gating < strict < operator
< prefer.
- New: TestGatingFilter_SkipsWhenRevisionHeaderPresent — panicking
rand01 asserts the fast path fires when the header is set.
- New: TestGatingFilter_SkipsWhenSingleRevisionInPool — same, for
the ≤1-revision fast path.
Live-verified on kind against the three shapes covered by
scripts/verify-slow-transition.sh (10p10d, 2p20d, 20p2d) — all steps
land within ±10pp of the expected cross-role weighted share; zero
cross-revision mismatches, zero request failures during transitions.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* refactor(disaggregation): gating skip on ANY strict header, not just [0]
Reviewer flagged the [0] shortcut on filter.go's header-pin check:
Selectors[0] being "the revision axis" is an implicit derivation
(revisionLabelKey = Selectors[0].LabelKey inside newController), not
anything the config surface names. Leaking that convention into the
gating fast path made the code fragile — a config that puts a non-
revision selector first would silently break the check.
Simpler and safer rule: any strict pin from the client means "I've
expressed a constraint, respect it." Skip gating and let the strict
filter do the narrowing. The picker then distributes over what
survives — no stochastic revision guess that could conflict with the
pin and 503.
Bonus: this handles cross-axis edge cases better. If a request pins
`slice=s2` only, and one revision has no s2 pods:
- previous rule: gating fires, weight-picks a revision, strict then
narrows within it — if the picked revision has no s2 pods, 503.
- new rule: gating skips, strict narrows all pods to slice=s2 (the
revision with no s2 pods drops out naturally), picker uniform.
Zero 503.
Rename: hasRevisionHeader → hasStrictHeader (matches the new logic).
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* fix(disaggregation): gating always runs coverage check, even when header pins
d59d301 made gating short-circuit entirely when a strict header was
present. That's correct for the load-shaping half of gating's job
(weighted-random-pick shouldn't override the client's pin), but wrong
for the safety-gate half (coverage check must always fire).
Concrete regression: v1 with 3 prefill and 0 decode pods (drift). A
client that pins x-disagg-revision=v1 on a prefill request under the
short-circuit behaviour got a 200 (prefill has v1 pods) followed by a
503 on the decode leg (decode has no v1 pods). The failure moved from
"loud 503 at prefill" to "silent 200 then confusing 503 at decode".
Split gating into its two responsibilities with different guards:
- Coverage check (drop revisions with crossRoleWeight == 0):
ALWAYS runs when gating is active. Emits disagg_gating_dropped_total
for each revision it drops.
- Weighted-random-pick (collapse the survivor set to one revision):
only runs when no strict header is present. When a header IS set,
return the coverage-filtered pool as-is and let strict downstream
do the pinning.
Net effect: header-pinned requests still fast-path the stochastic step,
but drifted revisions still get dropped upfront, so strict then finds
zero matches and 503s at prefill instead of at decode.
New test: TestGatingFilter_DropsUncoveredRevisionEvenWhenHeaderPinsIt
pins v1 when v1 decode is drained; asserts gating returns only v2 pods
(v1 dropped by coverage) despite the pin — downstream strict then
produces the expected empty candidate set.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(disaggregation): config schema rename — HeaderSelectors, RevisionGating, explicit label keys
Renames the two top-level config keys and adds an explicit revision-axis
declaration so the "Selectors[0] is the revision" convention that was
leaking through the code goes away.
Config shape:
disaggregation:
enabled: true
scope:
labelSelector: "disaggregatedset.x-k8s.io/name=my-set"
headerSelectors: # was: selectors
- name: revision
headerName: x-disagg-revision
labelKey: disaggregatedset.x-k8s.io/revision
mode: strict # strict | prefer
revisionGating: # was: gating
revisionLabelKey: ... # NEW, default: DefaultRevisionLabel
roleLabelKey: ... # NEW, default: DefaultRoleLabel
mode: sum # sum | disabled
requireRoles:
values: [prefill, decode] # (labelKey moved to parent)
Highlights:
* Selectors → HeaderSelectors, Selector → HeaderSelector, config.Selectors
→ config.HeaderSelectors everywhere.
* Gating → RevisionGating (type + json tag).
* RevisionGating.RevisionLabelKey and .RoleLabelKey are now explicit,
with package-level defaults (DefaultRevisionLabel / DefaultRoleLabel)
filled in by Validate() when omitted.
* RequireRoles.LabelKey removed — role label lives on the parent, so
RequireRoles is just { values: [...] }.
* newController reads revisionLabelKey/roleLabelKey from RevisionGating
(with defensive default-fill so tests that skip Validate still work),
not from Selectors[0]. The Selectors[0] convention is gone.
* HeaderSelectors is now optional — gating-only configs (server-side
revision shaping with no client-controlled pin) validate cleanly.
* HasSelectorsInMode → HasHeaderSelectorsInMode.
* Validation errors now say "headerSelectors[i]…" and
"revisionGating.*" instead of the old paths.
* Package doc rewritten to introduce both mechanisms up front.
Tests updated in place; new tests assert that omitted
revisionLabelKey / roleLabelKey get defaulted. Helm values files
(values-prefill.yaml, values-decode.yaml) migrated to the new schema
with a comment noting the defaults.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Revert the entire PR
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Add plugin-based disaggregation revision routing
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Cache disaggregation revision shares
Rebuild namespace-aware revision distributions from the complete Pod map on each notification so request handling can use warm shares without scanning Pods.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Scope disaggregation cache to one namespace
Default the Pod scope to the router namespace while allowing an explicit override, then maintain one warm revision distribution for that namespace.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Warn about disaggregation filter order
Report potentially unsafe router and prefer filter placement while preserving configured execution order.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Reject unsafe disaggregation router order
Fail startup when strict or gating routing is not the first filter, while keeping prefer placement advisory.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Add max-role revision gating mode
Allow revision traffic shares to follow the largest required role while retaining one shared cross-role coverage check.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Test revision gating rollout shapes
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Simplify revision weight calculation
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Run disaggregation routing before scheduling
Apply revision gating and strict selectors through the pre-scheduling candidate filter extension point. Express prefer selectors as soft affinity scores so filter ordering no longer carries disaggregation correctness requirements.
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Mark disaggregation plugins as alpha
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Update PR copyright years to 2026
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Drop unrelated framework file changes
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Move rollout gating into a Screener plugin
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Split disaggregation preference into a scorer plugin
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Use Screener terminology for rollout constraints
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Document rollout Screener behavior
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Document NVL72 slice affinity tuning
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Fix disaggregation README formatting
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Use full DisaggregatedSet plugin names
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Document DisaggregatedSet rollout screening
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Refine DisaggregatedSet rollout observability
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Use one dominant role for rollout shares
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Clarify DisaggregatedSet revision compatibility
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Simplify rollout failure documentation
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Clarify slice scorer weight example
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Require explicit revision gating mode
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Address rollout screener review feedback
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Reuse shared Pod readiness check
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Tidy rollout plugin integration
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Address rollout plugin review nits
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Align rollout metrics with EPP conventions
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Document rollout screener startup behavior
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Replace rollout preference scorer with generic affinity
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* Document DisaggregatedSet slice affinity
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
---------
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
* feat(epp): add port parameter to metrics-data-source The scrape port always follows the InferencePool targetPorts. When a sidecar fronts the model server on the pool target port, the engine's /metrics endpoint lives on a different port and cannot be scraped. The port parameter overrides the scrape port per data source, mirroring the dcgm-data-source port option. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com> * fix(epp): validate metrics scrape port and use strict decoding in test Reject ports outside 0-65535 in the metrics-data-source factory, describe the override as applying to positive values, and exercise the factory tests through the framework's strict decoder including unknown-field and out-of-range cases. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com> * docs(epp): document metrics port override scope for multi-port pools The override applies uniformly to every endpoint of the source, so it is unsuitable for pools with multiple target ports where each data-parallel rank exposes metrics on its own port. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com> * refactor(epp): make metrics port override a pointer A nil port keeps the endpoint's inference port for scraping; a set value is validated to 1-65535 and applied, so an explicit zero is rejected instead of silently meaning unset. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com> --------- Signed-off-by: Kaushik Mitra <kaushikmitra@google.com>
The crossReplicaPublisher was registered as a PollingDispatcher on the shared collector, whose base tick is set by refresh-metrics-interval (typically 60s for vLLM metric scraping). This silently clamped the cross-replica sync interval to 60s regardless of the configured crossReplicaSyncInterval, because periodTicks(1ms, 60s) rounds to 1 tick = 60s. Give the publisher its own per-endpoint goroutine with an independent time.Ticker at the configured sync interval. This decouples lightweight Redis state sync (two int64s per endpoint) from heavy HTTP metric scraping, so the two can run at their natural cadences. Before: crossReplicaSyncInterval: 50ms → silently fires at 60s After: crossReplicaSyncInterval: 50ms → fires at 50ms
Nil reqCtx.Request.RawBody after the body chunks have been sent to Envoy (BodyRequestResponsesComplete). The bytes were already sliced into reqBodyResp (as slice headers into the same backing array) and forwarded; nothing in the request lifecycle reads RawBody past this point. Subsequent phases (response header/body plugins, scheduling result) operate on SchedulingRequest.Body, a separate parsed representation that does not alias RawBody's backing array. For multimodal requests this frees the raw image bytes (often tens of MB per request) for the full inference wait (60-700s) instead of retaining them on reqCtx until the stream completes. At 100 concurrent multimodal requests this is the dominant per-request heap retention. The safety boundary is enforced by the type system: RawBody lives on handlers.Request (internal package), which is unreachable from plugin extension points — they receive *fwksched.InferenceRequest, which has no RawBody field. A plugin cannot read it even if it wanted to.
Lucas-Fernandes-Martins
force-pushed
the
lfm/combined-fixes
branch
from
August 6, 2026 09:57
57fe5b5 to
5279d4b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Release note (write
NONEif no user-facing change):