Skip to content

[LLM.PrefixDiskCache] Fix prefix disk cache not loaded after first response#4316

Merged
wangzhaode merged 13 commits into
alibaba:masterfrom
luckysmg:fix/prefixCache
Mar 26, 2026
Merged

[LLM.PrefixDiskCache] Fix prefix disk cache not loaded after first response#4316
wangzhaode merged 13 commits into
alibaba:masterfrom
luckysmg:fix/prefixCache

Conversation

@luckysmg

@luckysmg luckysmg commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

Description

Fix Prefix Disk Cache should generate sync file after first response.

Module

LLM.PrefixDiskCache

Type

  • Feature
  • Bugfix
  • Perf
  • Refact
  • Style
  • Doc
  • Test
  • Chore

Checklist

  • Commit message follows [Module:Type] Description format
  • Code compiles without errors
  • Tested on relevant platform(s)
  • No unrelated format or style changes included

@luckysmg
luckysmg marked this pull request as draft March 25, 2026 11:45
@luckysmg

luckysmg commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

核心时序:

这里先onClear,再onAlloc

image

然后打了日志(CPUKVCacheManager.mKVCacheInDisk=true的地方):

image image

onClear() this=0x1068cd800 
[KVCache DEBUG] onAlloc() this=0x1068cd800

onClear() this=0x1068cd980
onAlloc() this=0x1068cd980

[KVCache DEBUG] onClear() this=0x1068cdb00
[KVCache DEBUG] onAlloc() this=0x1068cdb00 

[KVCache DEBUG] onClear() this=0x1068cdc80
[KVCache DEBUG] onAlloc() this=0x1068cdc80

每次会是个新的,然后就一直在onClear的时候存不上PrefixCache的sync文件,因为mKVCacheInDisk= false

如果LLM不析构,那么就没机会存

rollback_demo 中,如果在推理的最后,加一个10s的sleep,你会发现prefixCache文件夹中的sync文件一直创建不出来,直到最后LLM析构,触发了CPUAttention析构,接着是CPUKVCacheManager析构--》CPUKVCacheManager.onClear,这里才能走到mKVCacheInDisk=true(上次onAlloc中留下的true),最终才能sync文件创建

@luckysmg

luckysmg commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

修复方案上面,核心是CPUKVCacheManager不太好感知LLM的首次response,所以看看到底是修复放LLM里面,还是放CPUKVCacheManager里面,目前放在onRealloc中,也可能不是很合适

@luckysmg luckysmg changed the title 【TODO】Fix Precache 【TODO】Fix prefix disk cache not loaded after first response Mar 26, 2026
@luckysmg luckysmg changed the title 【TODO】Fix prefix disk cache not loaded after first response 【BugFix】Fix prefix disk cache not loaded after first response Mar 26, 2026
@luckysmg luckysmg changed the title 【BugFix】Fix prefix disk cache not loaded after first response [LLM] Fix prefix disk cache not loaded after first response Mar 26, 2026
@luckysmg luckysmg changed the title [LLM] Fix prefix disk cache not loaded after first response [LLM.PrefixDiskCache] Fix prefix disk cache not loaded after first response Mar 26, 2026
@luckysmg
luckysmg marked this pull request as ready for review March 26, 2026 08:42
Comment thread transformers/llm/engine/demo/rollback_demo.cpp
Comment thread transformers/llm/engine/src/llm.cpp
@wangzhaode
wangzhaode merged commit 622b3fb into alibaba:master Mar 26, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants