Foyer not closing files, causing to exceed ulimit

[Blonteractor]

backend 84602 priyanshu 2503u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2504u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2505u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2506u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2507u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2508u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2509u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2510u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2513u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2514u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2515u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2516u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2517u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2518u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2519u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2520u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2521u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2522u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2523u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2524u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2525u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2526u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2527u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2528u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2529u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2530u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2531u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2532u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2533u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2534u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2535u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2536u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2537u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2538u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2539u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2540u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2541u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2542u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2543u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2544u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2545u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2546u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2547u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2548u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2549u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2551u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2552u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2553u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2554u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2555u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2556u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2557u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2558u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2559u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol

Above is the output for lsof -p $(pgrep backend) | tail -100, it has smol open like 2500 times.

The haskell_cache/smol is the cache unit, created using FileDeviceBuilder. I am running foyer 0.22.2.

[MrCroxx]

Hi, Priyanshu. Thank you for reporting. Sorry for the late response. I missed the email.

Foyer will not proactively close cache files now, because each file may be accessed again. Frequently opening and closing them would cause significant performance overhead.

If the ulimit is too small, you can first consider adjusting the ulimit restriction. If having too many open files causes performance issues in your environment (for example, on some embedded devices—I’m not sure whether Foyer currently needs to serve in such environments), you can consider increasing the cache file size. If neither works, perhaps Foyer should consider supporting an open file timeout to proactively close files that haven’t been accessed recently.

[Blonteractor]

Hey, thanks for replying. I understand foyer doesn't close open files as opening them again would be a performance drain. But it seems like its opening the same file multiple times. Even if I have only 20 blocks in my , those 20 blocks are opened a bunch of times and eventually exhaust the ulimit no matter what it is set to

[MrCroxx]

Hey, thanks for replying. I understand foyer doesn't close open files as opening them again would be a performance drain. But it seems like its opening the same file multiple times. Even if I have only 20 blocks in my , those 20 blocks are opened a bunch of times and eventually exhaust the ulimit no matter what it is set to

Got it! Thank yoi for explaining. Let me check.