🐛 BUG: `.wrangler` KV emulation disk usage grows unbounded, created 100K+ files on my machine

[aroman]

Which Cloudflare product(s) does this pertain to?

KV, Wrangler

What versions are you using?

wrangler 3.107.2

What operating system and version are you using?

macOS Sequoia 15.3.1

Describe the Bug

Over the past several months, I've racked up quite a lot of KV writes in local mode. I discovered that apparently KV really does not scale well - it created hundreds of thousands of files on my machine in .wrangler..

Just to put this in perspective... it took 3.5 minutes to run rm -rf .wrangler. I'm on an M1 Pro machine with a lightning fast SSD. That's an ungodly amount of files, and from my perspective, abuse of the filesystem. Surely this sort of thing would be better off in a single text file, or sharded across several files...

router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a3f99e18fd7f4a3a3024bca7b0ec1a8eb54825e03cdc193e831c8a8d131ca7b0000018d00433e3c
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a3fb21898e5bf15de8389ea975ea7b85daabbe536dc0e39b1d4e3b1f59139730000018eb5e3e6de
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a400bf0b15cd7ff1d051cdff842d87ffbf06cf2dc48e2791daefcc12199d8670000018c029e44ec
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a403925564f3d83e6a60e0c8409fe237cb17043316de54db78f30d2c84358590000018cec1fe08e
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a4076d7ec88cb45516fbb1a1cff91d29242019f5f37ad4b03f89c1ba19245a30000018b24d3a085
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a40ad605dab6ab800b797dc109152b7a2d36ea6f489636acb0bc36b50cc17cd0000018b25571d04
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a40ba6ce5acc98a94b029c6f77ded35088cec268a6e03eba3c4e652969b5ddc0000018c17c296c1
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a40efa059db6df2390df8839c4876259e0146d38175bfb2c0138c03937c48480000018cff65a4fb
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a40f43a9eaee6cf6a365cfdee59b9eaeacff7feddf5fe1a3e969f4c11f9ada00000018b60b47982
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a40f46d926adfb9f19fc6c44c66c6734a5b079ab940d73a43335f18dbc73e840000018b61ea6d29
router/.wrangler/state/v3/kv/_this not needed in Local dev mode_/blobs/2a4106eb6b5b350864db80f5d9a281b0b7f244d952901007ae96ed86cd2fe73c0000018c3ae2fadd

[dario-piotrowicz]

Hi @aroman, thanks for the issue 🙂

I've briefly discussed this with the team and we think that it would be feasible to try improve things in wrangler so that we try to combine data into larger files instead of creating many smaller ones, assuming that that's the issue you're encountering, could you check and let us know generally the size of your files? do you have lots of small files? or is it just that you have lots of data saved in your local KV?

Over the past several months, I've racked up quite a lot of KV writes in local mode

This makes me think that this might just be related to the sheer amount of data you are storing, in that case it could be argued that it'd be up to you to clear the wrangler cache periodically? 🤔

So tlrd, could you clarify the issue you're seeing and what solution you'd like here? is this an issue regarding having lots and lots of small files? of is it just that you have a huge amount of data that you need to deal with?