[Bug]: Cronicle Disk Space Issue

[ctcarino]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Today, we encountered that cronicle just crashed. we tried the following:

1. we tried to restart the server using control.sh restart
2. we tried to do the maintenance using control.sh maint
3. deleting cronicle.pid file

then when i check the Cronicle.log i see logs that says "failed to restore record" and "no space left on device"

see, our "data" folder is a symlink that goes to a much bigger disk(500gb). i checked that disk and saw that its only 33% used. and when i did try to create a folder, it did say no space left. so this baffles me. but now we know what is the issue.

i dont know yet what causes this but i remove the symlink for now and create a new data folder and started cronicle again and now it works.

cronicle is being hosted in AWS EC2 server using Amazon Linux 2023

Operating System

Amazon Linux 2023

Node.js Version

18.18

Cronicle Version

0.9.85

Server Setup

Single Primary with Workers

Storage Setup

Local Filesystem

Relevant log output

[1754484712.874][2025-08-06 12:51:52][ip-10-10-1-160.us-west-1.compute.internal][4382][Storage][error][rollback][Could not rollback transaction: logs/events/emdsu3uvtfu: Failed to restore record: logs/events/emdsu3uvtfu/-139: Failed to write file: logs/events/emdsu3uvtfu/-139: data/_temp/6173945e1e878ff956b624de4fa8a356.json.tmp.1: ENOSPC: no space left on device, open 'data/_temp/6173945e1e878ff956b624de4fa8a356.json.tmp.1'][]

[1754484712.874][2025-08-06 12:51:52][ip-10-10-1-160.us-west-1.compute.internal][4382][Storage][error][fatal][Fatal transaction error: Could not rollback transaction: logs/events/emdsu3uvtfu: Failed to restore record: logs/events/emdsu3uvtfu/-139: Failed to write file: logs/events/emdsu3uvtfu/-139: data/_temp/6173945e1e878ff956b624de4fa8a356.json.tmp.1: ENOSPC: no space left on device, open 'data/_temp/6173945e1e878ff956b624de4fa8a356.json.tmp.1'][]

Code of Conduct

• I agree to follow this project's Code of Conduct

[jhuckaby]

Okay, so running out of disk space is a catastrophically bad error when you are using the local filesystem for storage, because the Cronicle "database" will be corrupted.

Note: You may not have run out of disk "space", but rather INODEs. They are tracked and listed separately. df -i will tell you how many you have left. Cronicle uses a ton of INODEs because it uses the filesystem as a database.

At this point I recommend you do a full export, wipe (delete everything), and import on a fresh install. Instructions here:

https://github.com/jhuckaby/Cronicle/blob/master/docs/CommandLine.md#data-import-and-export

Note that export/import gets everything except individual job history and job logs.

If the export fails due to corruption, which is likely in this case, you can try to repair the storage system:

https://github.com/jhuckaby/Cronicle/wiki/Troubleshooting#storage-repair

If you did run out of INODEs, which I suspect is the case here, and you want to keep using the local filesystem, then it is highly recommended that you increase the INODEs on your big disk.

Good luck!