We should make sure that our log file size is limited as otherwise this can cause issues like we saw with the attestation bug which happened ~1.8 million times a day and added a error log including stacktrace to the log file which was roughly 50-60 GB in size. This is bad as it can exhaust the disk space of the server and stall the node... and might even require manual clean up from operators.
I suggest we come up with a number eg. 5GB and after that limit is reached we stop writing logs to the file or alternative trim it from the beginning or override it in a way.
Could also consider having a max size on the whole log directory, let's say 25 GB and delete old log files if that limit is reached.
In any case I don't see a good reason why we need to keep writing to log file if it's bloated with the same error million times which is really the only case it can get so huge.
This is from one of the EF devops nodes
48G beacon-2025-03-08.log
50G beacon-2025-03-09.log
60G beacon-2025-03-10.log
4.8G beacon-2025-03-11.log
0 beacon-2025-03-12.log
70G chain-db
1.1G invalidSszObjects
14M peerstore
232G in total... this is bad, Barnabas is not happy about it
We should make sure that our log file size is limited as otherwise this can cause issues like we saw with the attestation bug which happened ~1.8 million times a day and added a error log including stacktrace to the log file which was roughly 50-60 GB in size. This is bad as it can exhaust the disk space of the server and stall the node... and might even require manual clean up from operators.
I suggest we come up with a number eg. 5GB and after that limit is reached we stop writing logs to the file or alternative trim it from the beginning or override it in a way.
Could also consider having a max size on the whole log directory, let's say 25 GB and delete old log files if that limit is reached.
In any case I don't see a good reason why we need to keep writing to log file if it's bloated with the same error million times which is really the only case it can get so huge.
This is from one of the EF devops nodes
232Gin total... this is bad, Barnabas is not happy about it