Future cache system improvements

[fangfufu]

There had been quite a few comments about restricting the use of the cache system:

I guess this should be a part of a wider discussion of cache defaults:

#32

By default cache is disabled (good) but I don't think that allowing httpdirfs to allocate up to 8TB by default of disk space is reasonable. For most users that would mean caching everything httpdirfs comes across...

Something in the area of 100 segments (so 800MB by default) would make more sense, I think.

#93 (comment)

For use cases like media (think performous, though I haven't made sure it's really beneficial there) it would be convenient to have the persistent caching store directory listings (maybe including metadata) but not the large files.

#200 (comment)

A better solution would be to put a limit on cache size, and make it evict cached data based on some policy, like LRU. But > for now, this should suffice for the most part (correctness at least).

What are people's thoughts on this? Please feel free to comment below to provide suggestions.

Here are some ideas of mine that I might go and implement:

• Setting minimum file size threshold for caching. wishlist: Cache for directories / metadata only / small files only #93
• Setting maximum file size threshold for caching. feat: add --cache-min-size and --cache-max-size file size threshold options #284
• Setting the maximum local cache size per HTTP directory.
• LRU replacement for local cache file, when there is a local cache size limit.
• Put metadata for each website into a sqlite database, to avoid creating thousands of small files on the filesystem.