[BUG] The default cacheLimit causes to create a large amount of swap cache.

[czkoko]

Describe the bug
I'm running a DIT arch image generation model using mlx-swift. Today, I accidentally discovered that the system process kernel_task is writing nearly 1TB of data per day. Initial testing revealed that MLX is the culprit.

Each sampling step creates over 2GB of swap cache. The model uses 10GB RAM, and with 16GB RAM, this shouldn't be triggering such a severe swap caching.

After ruling out various possibilities, I found the key problem was the Memory.cacheLimit value.
Setting it to Memory.cacheLimit = 0, then kernel_task never creating swap caches, and memory consumption dropped to around 8GB when running the model.
Memory.cacheLimit = 0 didn't affect performance, in fact, it made the model run faster.

Desktop (please complete the following information):

• OS Version: MacOS 15.3
• Version MLX 0.36

[leuasseurfarrelds247-arch]

@