Random questions thread

[solbjorn]

I think this misses a thread for random quick questions.

1. Why is block_cache_allocation_mode set to MALLOC by default, not MMAP? The former uses more memory: I have 10 archives 8 Gb total, peak memory usage bumped by 2.5 Gb after switching from SquashFS to DwarFS. Is the MMAP mode slower (the game runs on Windows)? What would be better then, to reduce the default cache limit from 512 Mb to let's say 128 or to just use MMAP?

[mhx]

1. Why is block_cache_allocation_mode set to MALLOC by default, not MMAP? The former uses more memory: I have 10 archives 8 Gb total, peak memory usage bumped by 2.5 Gb after switching from SquashFS to DwarFS. Is the MMAP mode slower (the game runs on Windows)? What would be better then, to reduce the default cache limit from 512 Mb to let's say 128 or to just use MMAP?

It's the default mainly because the mmap mode was added later (precisely to handle cases where malloc will result in seemingly excessive memory consumption).

The longer story is that the behaviour of malloc mode depends very much on the memory allocator being used and its configuration. I assume you've already read this section. How exactly you want to configure the FUSE driver is very much dependent on your use case.

So malloc shouldn't per-se consume "more" memory, but mmap will make sure to explicitly release memory to the OS once blocks are purged from the cache. If you've configured jemalloc appropriately, you can achieve a similar level of memory consumption as with mmap. Also, if you're using small block sizes, the overhead of mmap can be significant.

Your use case really determines how you want to configure cache size, allocation mode, cache tidying, or sometimes even block size when building the file system.

[solbjorn]

I see. I use the default block size (preset level 3, lz4hc, block size is 1 Mb there IIRC?). I forgot to mention that I use mimalloc3 and the numbers are from its stats. It might be that this memory consumption comes from mimalloc's logics. I've read the section you mentioned, but skipped the part about jemalloc -- I'll take a look whether mimalloc exposes similar options.

Thanks!