Repository navigation
SGLang HiCache routing credits Mooncake pages SGLang does not load #18174
Description
With --shared-cache-type hicache, the KV router credits Mooncake (HiCache L3) pages that SGLang never loads, so it underestimates the prefill work some requests cost on some workers:
- It counts every stored page past the worker's GPU prefix, including pages after a missing page. SGLang prefetches only the unbroken run that starts where the worker's local prefix ends.
- It credits pages the worker already holds in host memory twice: once through
host_cache_hit_weightand again throughshared_cache_multiplier. - It credits runs shorter than SGLang's
prefetch_threshold(default 256 tokens), which SGLang skips. - It credits LoRA requests with base-model pages. SGLang keys adapter pages under a separate namespace, so the adapter request cannot load them.
- Under attention DP, workers publish the full
tp_size, but SGLang names storage keys by attention-TP rank, so the router never finds a hit for non-MLA models.
Expected: the shared-cache credit for a worker equals the pages SGLang would prefetch for that worker.
Agent notes
SGLang 0.5.23 references:
managers/scheduler.py: the prefetch starts atmatched_len = req.prefix_len + req.host_hit_length(device plus host match).managers/cache_controller.py_storage_hit_query:batch_existsreturns the number of consecutive existing pages from the start and the query stops at the first miss.mem_cache/unified_radix_cache.py:prefetch_length < self.prefetch_thresholddeclines the prefetch; the threshold comes fromhicache_storage_backend_extra_config["prefetch_threshold"](default 256) and the controller usesmax(threshold, page_size).mem_cache/utils.pyget_storage_hash_strseeds storage chains withstorage_namespace_seed(extra_key, cache_salt);schedule_batch.pyappendslora_idtoextra_key.cache_controller.py_generate_storage_configusesattn_tp_rankandattn_tp_sizewhen DP attention is enabled.
Dynamo: SharedCacheHits::hits_beyond in lib/kv-router/src/protocols.rs, the default scorer in lib/router-plugins/builtin/src/default/scorer.rs, the reference selector, lib/llm/src/kv_router/shared_cache.rs, and components/src/dynamo/sglang/register.py.
Activity
fix(kv-router): align overlap scores and legacy thresholds with HiCac…