[Config Support]: tmpfs /tmp/cache fills up and doesn't release space, causing recording failures #20758
Unanswered
0xjams
asked this question in
Config Support
[Config Support]: tmpfs /tmp/cache fills up and doesn't release space, causing recording failures
#20758
Replies: 7 comments 2 replies
-
|
(1), (2), (3), (4), (5), (6), (7) Analysis of Your tmpfs IssueBased on your symptoms and the available documentation, here's what's happening: The Core Problem: Orphaned File HandlesThe 7.5GB of "ghost" space in Root Cause: go2rtc Codec MismatchYour go2rtc logs show a critical issue: For the Configuration IssuesProblem 1: Audio configuration mismatch output_args:
record: preset-record-generic # This preset doesn't handle audio
inputs:
- path: rtsp://127.0.0.1:8554/patio_lateral_cam
roles:
- record
- audio # But you're trying to use audio hereProblem 2: go2rtc audio transcoding patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0
- ffmpeg:patio_lateral_cam#audio=opusBut your camera likely doesn't provide AAC audio, and you're not handling it correctly(1). SolutionsFix 1: Correct Audio ConfigurationIf you want audio for patio_lateral_cam:
ffmpeg:
output_args:
record: preset-record-generic-audio-copy # Change thisFix 2: Fix go2rtc Audio HandlingIf your NVR doesn't provide AAC audio, update your go2rtc config to properly transcode: patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0
- "ffmpeg:patio_lateral_cam#video=copy#audio=copy#audio=aac"Or if you don't need audio at all, remove the audio transcoding and role: patio_lateral_cam:
- rtsp://REDACTED:[email protected]:554/cam/realmonitor?channel=5&subtype=0 # Remove audio lineAnd remove the inputs:
- path: rtsp://127.0.0.1:8554/patio_lateral_cam
roles:
- record # Remove audio roleFix 3: Increase tmpfs SizeYour calculation for tmpfs may be insufficient. The documentation provides this formula for calculating required tmpfs size(6): # For each camera:
# (width * height * 1.5 * fps + 270480) / 1048576 * <number of cameras> + 40MB
# For your 5 recording cameras, assuming worst case 2560x1920 @ 15fps:
python -c 'print("{:.2f}MB".format(((2560 * 1920 * 1.5 * 15 + 270480) / 1048576) * 5 + 40))'Your 8GB may be too small if you have high-resolution cameras(6). Consider increasing to 16GB: tmpfs:
size: 16000000000 # 16GBAddressing Your Specific QuestionsQ: Why does tmpfs space remain allocated after FFmpeg crashes? When FFmpeg crashes before closing file handles, Linux keeps the space allocated until the process exits(6). The recording maintainer can't clean these up because the files are still "open" from the kernel's perspective(5). Q: Could the go2rtc codec mismatch be causing crashes? Yes, absolutely(1). The codec mismatch between video (H264) and audio (ANY/undefined) is causing FFmpeg to fail when trying to write segments(1). Q: Should the recording maintainer clean up orphaned segments? The maintainer does clean up corrupt segments automatically(5), but it cannot clean up files with open handles from crashed processes(5). You're seeing warnings like "Discarding a corrupt recording segment" in similar issues(7). Q: Manual garbage collection without restart? No documented method exists to force cleanup without restarting. The container restart is currently the only way to release orphaned file handles(6). Recommended Action Plan
The audio codec issue is almost certainly your root cause. Once fixed, the FFmpeg crashes should stop and tmpfs should no longer fill up with orphaned files. ** 📚 Sources:
Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Increasing the size of tmpfs is just delaying the eventual crash. I started at 2GB, and every time I increased the size, the problem would still occur eventually. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
We will need to see a full copy of logs starting at Frigate startup |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Here are the docker logs since the last time the cron job restarted Frigate: Here are the go2rtc logs too: Here are some details: twice during the day, my event-related recordings stopped (regardless of the camera), and I noticed that the problem happened when my Amcrest Doorbell camera started showing connection problems. After I restarted it, Frigate recovered. I was wondering if there's a way to simply alert when it starts failing or maybe just disable the camera temporarily. I think this event is causing a domino effect. I suspect that when that camera starts failing, ffmpeg starts crashing, and the more that happens, the sooner tmpfs gets full, and it all crashes. Any suggestions? I don't know if this could be a viable fix? But I don't want to keep trying things without having an idea of what the problem truly is. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Heads up @NickM-27 I am also seeing this /tmp/cache fill up issue with Amcrest cameras. Looking at other issues on GitHub there seems to be a common thread of the cache not flushing. Happy to contribute logs if I can find them. I've restarted the system several times so there's nothing left at the moment. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi everyone: Update: Config: Logs: Here are the Go2RTC logs in case they are needed: |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
it looks like your cameras are having some timeout issues and then it runs into some kind of crash, I have never seen this crash before. |
Beta Was this translation helpful? Give feedback.
-
|
In any case, this has been refactored in 0.17, so you can try using that unreleased version or for now work on fixing the cameras timeout issues |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much! I'll give the newer version a try, and I'll also look into the timeout issue. I'll report my results |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
Description:
I'm experiencing an issue where my tmpfs partition (/tmp/cache) fills up and space is not released, eventually preventing new recordings from being created. FFmpeg crashes with "No space left on device" errors, but the space remains allocated even after the crashes.
Environment:
Symptoms:
[ERROR:[email protected]] global grfmt_webp.cpp:386 write Only 0 or 9850 bytes are written
[out#0/segment @ 0x5606ab9ba100] Could not write header (incorrect codec parameters ?): No space left on device
0 /tmp/cache/[email protected]
0 /tmp/cache/[email protected]
[... many more 0-byte files ...]
80K /tmp/cache/preview_frames
7.5G /tmp/cache (total reported by du)
error="streams: codecs not matched: video:H264 => audio:ANY" stream=patio_lateral_cam
Troubleshooting completed:
Questions:
Temporary workaround:
Restarting the container releases the space, but the issue recurs. I even created a daily cron for this, but sometimes it fills up quicker than the cron's frequency
Version
0.16.2-4d58206
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
error="streams: codecs not matched: video:H264 => audio:ANY" stream=patio_lateral_camFrigate stats
No response
Operating system
UNRAID
Install method
Docker Compose
docker-compose file or Docker CLI command
Object Detector
Coral
Screenshots of the Frigate UI's System metrics pages
No response
Any other information that may be helpful
I tried using the AI on the website to debug this, but it didn't help much. I've been tweaking my config for days and nothing changes.
Beta Was this translation helpful? Give feedback.
All reactions