Skip to content

Commit 49b9dcf

Browse files
committed
[optimizer] run linear attention on LiteHitMamba and retire the legacy indexer
Linear-attention instances were simulated by the legacy LruCacheIndexer, whose model fuses a Full block and its Mamba checkpoint into one cache entry whose charge doubles as the checkpoint tag. The two could therefore never be evicted independently - exactly the case that matters, since a Full block stays warm through prefix reuse while its checkpoint goes cold. It also built one real LRU per capacity tier, had no facts pipeline, and inferred a checkpoint by comparing a stored charge against a constant. InstanceState now holds exactly one analyzer: LiteHit for full attention, LiteHitMamba for linear attention. The legacy indexer, its factory, its TTL wrapper and its hit-age histogram are deleted (the histogram had no proto field and no client or dashboard consumer). Linear attention with a group TTL is rejected at registration: the Mamba core carries no time axis yet, and the offline runner already refused that combination. Two byte quantities replace the fused wording internally: a Full block always costs full_charge_bytes and a checkpoint additionally stores mamba_charge_bytes, instead of adding them into size_full_linear and subtracting them back out at every use. The register response is unchanged; the fused value is composed once at the proto boundary. The block count in that response is now computed explicitly instead of dividing by an "average bytes per block" that does not exist when two charges differ. It stays exact for full attention, where it doubles as the projection slot, and is labelled an estimate for linear attention, where hits are decided on the byte axis and nothing consumes the estimate.
1 parent 873ccde commit 49b9dcf

26 files changed

Lines changed: 517 additions & 2056 deletions

kv_cache_manager/optimizer/docs/optimizer_architecture.md

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ KVCacheManager Optimizer 是一个独立的缓存优化分析模块,通过回
4141
Optimizer 当前包含离线 trace 回放和在线优化服务两条运行路径:
4242

4343
- 离线回放路径使用 `OptimizerManager``OptIndexerManager``OptEvictionManager``RadixTreeIndex`,依赖 replay 配置、trace loader/converter 和 data storage 类型。
44-
- 在线服务路径使用 `OnlineOptimizerManager`、LiteHit`CacheIndexerFactory``LruCacheIndexer`/`TtlCacheIndexerWrapper` 和 service/protobuf 接口,依赖 online 实例组/实例配置与 registry。full-attention 多容量 LRU 由 `InstanceState` 直接持有 LiteHitlinear attention 仍由通用 `CacheIndexer` 模拟
44+
- 在线服务路径使用 `OnlineOptimizerManager``LiteHit`/`LiteHitMamba` 和 service/protobuf 接口,依赖 online 实例组/实例配置与 registry。`InstanceState` 按实例类型持有唯一一个分析器:full-attention 用 LiteHitlinear attention 用 LiteHitMamba
4545

4646
两条路径共享 optimizer 模块内的命中率建模代码,但运行时、配置 target 和服务边界保持独立,避免在线服务引入离线 replay/data storage 依赖。
4747

@@ -76,9 +76,8 @@ OnlineOptimizerManager (在线运行时协调器)
7676
├── OptimizerRegistryManager (实例组/实例持久化)
7777
└── InstanceState (instance_id 隔离的运行状态)
7878
79-
├── LiteHit (full-attention 多容量 LRU)
80-
└── CacheIndexerFactory (linear attention)
81-
└── LruCacheIndexer / TtlCacheIndexerWrapper
79+
├── LiteHit (full-attention 多容量 LRU,可叠固定 TTL)
80+
└── LiteHitMamba (linear attention:Full + Mamba checkpoint 字节加权 LRU)
8281
```
8382

8483
### 目录结构
@@ -95,10 +94,11 @@ kv_cache_manager/optimizer/
9594
│ └── online_runtime/ # 在线运行时
9695
│ └── online_optimizer_manager.h/cc # 在线实例注册、TraceQuery 和统计
9796
├── index/ # 索引层
98-
│ ├── radix_tree_index.h/cc # 离线 Radix 树索引
99-
│ └── online/ # linear attention 在线容量/TTL 索引器
100-
├── liteHit/ # 轻量多容量 full-attention LRU 命中率核心
101-
│ ├── lite_hit.h/cc # 多容量 LRU 命中率分析器
97+
│ └── radix_tree_index.h/cc # 离线 Radix 树索引
98+
├── liteHit/ # 轻量多容量命中率核心
99+
│ ├── lite_hit.h/cc # full-attention 多容量 LRU 命中率分析器(可叠固定 TTL)
100+
│ ├── lite_hit_mamba.h/cc # linear attention:Full + Mamba checkpoint 命中率分析器
101+
│ ├── weighted_lru_pool.h/cc # 字节加权 LRU 池(typed key:Full / Mamba)
102102
│ └── dynamic_fenwick_tree.h/cc # reuse-distance 用的 order-statistics Fenwick
103103
├── eviction_policy/ # 驱逐策略层
104104
│ ├── base.h # 策略基类
@@ -154,7 +154,9 @@ kv_cache_manager/optimizer/
154154

155155
在线服务协议定义在 `kv_cache_manager/protocol/protobuf/optimizer_service.proto`,由 service 层转换为 optimizer online config/runtime 对象。
156156

157-
full-attention TraceQuery 只把完整 block 放入 `block_keys`,并用 `input_token_len` 传入包含尾部 token 的原始输入长度。Online Manager 用 full location spec group 的固定字节 charge 将 `capacity_gb` 向下取整为 block 容量,再把同一请求交给 LiteHit;请求级与累计命中率均为 `prefix_hit_blocks * block_size_tokens / input_tokens`。旧客户端缺少长度时兼容假设没有尾部 token。full-attention 组配置 `ttl_seconds != 0` 时,固定 TTL 以墙钟时间叠加在 LiteHit 之上(口径与 `TtlCacheIndexerWrapper` 一致);linear attention 的统计口径和 TTL 行为保持 legacy 路径不变。
157+
full-attention TraceQuery 只把完整 block 放入 `block_keys`,并用 `input_token_len` 传入包含尾部 token 的原始输入长度。Online Manager 用 full location spec group 的固定字节 charge 将 `capacity_gb` 向下取整为 block 容量,再把同一请求交给 LiteHit;请求级与累计命中率均为 `prefix_hit_blocks * block_size_tokens / input_tokens`。旧客户端缺少长度时兼容假设没有尾部 token。full-attention 组配置 `ttl_seconds != 0` 时,固定 TTL 以墙钟时间叠加在 LiteHit 之上(严格死线:age 达到 TTL 即失效,每次访问刷新)。
158+
159+
linear attention(`linear_step > 0`)走 LiteHitMamba:Full block 与 Mamba checkpoint 是两个独立对象、各自独立 recency,容量轴是**总字节**而非 block 数,命中语义是"从某个 checkpoint 恢复"。它还没有时间轴,因此 `linear_step > 0``ttl_seconds > 0` 的组合在注册时直接拒绝(离线 runner 同策略)。
158160

159161
---
160162

kv_cache_manager/optimizer/index/BUILD

Lines changed: 0 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -16,48 +16,3 @@ cc_library(
1616
"//kv_cache_manager/optimizer/analysis",
1717
],
1818
)
19-
20-
cc_library(
21-
name = "cache_indexer",
22-
srcs = [
23-
"online/lru_cache_indexer.cc",
24-
],
25-
hdrs = [
26-
"online/cache_indexer.h",
27-
"online/lru_cache_indexer.h",
28-
],
29-
deps = [
30-
"//kv_cache_manager/common:logger",
31-
"//kv_cache_manager/common/cache",
32-
],
33-
)
34-
35-
cc_library(
36-
name = "ttl_cache_indexer_wrapper",
37-
srcs = [
38-
"online/ttl_cache_indexer_wrapper.cc",
39-
],
40-
hdrs = [
41-
"online/ttl_cache_indexer_wrapper.h",
42-
],
43-
deps = [
44-
":cache_indexer",
45-
"//kv_cache_manager/common:timestamp_util",
46-
],
47-
)
48-
49-
cc_library(
50-
name = "cache_indexer_factory",
51-
srcs = [
52-
"online/cache_indexer_factory.cc",
53-
],
54-
hdrs = [
55-
"online/cache_indexer_factory.h",
56-
],
57-
deps = [
58-
":cache_indexer",
59-
":ttl_cache_indexer_wrapper",
60-
"//kv_cache_manager/common:env_util",
61-
"//kv_cache_manager/common:logger",
62-
],
63-
)

kv_cache_manager/optimizer/index/online/cache_indexer.h

Lines changed: 0 additions & 79 deletions
This file was deleted.

kv_cache_manager/optimizer/index/online/cache_indexer_factory.cc

Lines changed: 0 additions & 68 deletions
This file was deleted.

kv_cache_manager/optimizer/index/online/cache_indexer_factory.h

Lines changed: 0 additions & 23 deletions
This file was deleted.

0 commit comments

Comments
 (0)