Claude Code Cache Performance Issues

[JJOne123]

Describe the bug
I use Claude Code pointing to my local server running oMLX v0.5.3. I dont think it is specific to this version as I remember seeing similar behaviours with previous versions.

When I run Claude Code (CC) I set these variables in my claude settings.json file as is generally recommended when running locally:

...
"env": {
      "CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
      "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "95"
},
...

Recommended here: https://www.reddit.com/r/ClaudeCode/comments/1tfcnac/fix_your_cc_right_now_and_save_tokens/

What I am seeing is that during a session, oMLX will cache CC hefty system etc during the session. But every so often oMLX seems to re-read the entire session again as PP.
I understand that CC has a 5 minute cache timeout at Anthropic servers when using their APIs, but for oMLX I would have thought that it would continue to get the hot-cached data for each call and thus prevent long PP re-processing.

I've attached my server.log below. I noticed a large re-PP event around the 15:40 mark in the log (give or take a few minutes). At this point I had asked the model to write out its plan to a file IIRC.

Using Ornith-1.0-35B-MLX-oQ8 model on oMLX 0.5.3.

To Reproduce
Steps to reproduce the behavior:

1. Use oMLX with Claude Code (currently using CC version 2.1.217)
2. Do your normal work etc
3. On the web UI, keep an eye on your prefill processing total tokens (after the initial code start of course)
4. Periodically you should see a large prefill event take place again even though you have not issued a new large prompt.

Expected behavior
oMLX to use the cache to prevent large prefill events

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• macOS Version: 26.5.2
• oMLX Version 0.5.3

Additional context
If this is a user issue, or if there are any additional Claude Code settings required, it might be useful to put them in the README say.

cc_caching_issue_server.log.gz

[bnightstar]

Yeah this is new behavior it's the same for me as well making it a lot slower.

[jundot]

Thanks for the detailed log, it made this one straightforward to pin down.

First the good part: oMLX did not drop its cache. The SSD prefix cache survived every memory pressure event in your log, there are no cache reconstruction failures, and some of the big re-prefill events (14:43, 15:46) happen with no memory activity around them at all.

The actual trigger is Claude Code itself. Recent CC versions quietly delete old tool results from the conversation when context usage gets high, without running a full /compact. You can see it in your log: the conversation shrank from ~110.8k to ~106.9k tokens between two consecutive requests around 14:58. When CC edits the middle of the conversation like that, the tokens no longer match what the server cached, so oMLX has to re-prefill everything from that point. This is client side behavior, no server cache can survive it.

Two practical notes. There is no switch to turn this off in current CC builds as far as I can tell. And the AUTOCOMPACT_PCT_OVERRIDE=95 trick actually makes it worse on a local server, because it keeps the session parked near the context ceiling where this cleanup fires over and over. I'll add a note about this to the README for Claude Code users.

[bnightstar]

@jundot I'm on older versions on Claude Code 2.1.195 and 2.1.127 and this still happens with the prefix issues and it only started happening when I switched to 0.5.1 was working fine in 0.4.4.

[JJOne123]

Thanks @jundot . Your explanation makes sense though is somewhat alarming that anthropic would do this.
I mean, I use the frontier models also at times via regular Claude Code and my understanding was that cached tokens were a lot cheaper to read / prefill process (once they are written) and that the cache expires every 5 mins server side.
https://platform.claude.com/docs/en/build-with-claude/prompt-caching

With this silent cache updating, anthropic/CC are invalidating the cache thus causing my token cost to skyrocket on the next prompt. This could be interpreted/seen as a token / cost / money grab by some. If so, it would be a bit dishonest assuming this is the actual issue, and I've no reason to doubt you of course.