The daemon died after some time for no reason

[farassadek]

[sebdah]

Do you have any logs from this session?

[downspot]

I have the same thing happening. Nothing useful in my logs. The process just seems to randomly die after a period of time.

[sebdah]

@downspot do you have debug enables logging? Also could you send the log anyway, it might be useful for debugging.

[downspot]

Don't have debugging enabled. I am not sure I see an option for that. Running Automated EBS Snapshots version 0.4.1. Github is not letting me attach the file.

2015-10-11 00:33:42,818 - auto-ebs - INFO - Connecting to AWS EC2 in us-east-1
2015-10-11 00:33:45,274 - auto-ebs - INFO - The newest snapshot for vol-becc92fa is 1402 seconds old
2015-10-11 00:33:45,274 - auto-ebs - INFO - No need for a new snapshot of vol-becc92fa
2015-10-11 00:33:45,396 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:45,547 - auto-ebs - INFO - The newest snapshot for vol-f5e6b8b1 is 1397 seconds old
2015-10-11 00:33:45,548 - auto-ebs - INFO - No need for a new snapshot of vol-f5e6b8b1
2015-10-11 00:33:45,681 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:46,062 - auto-ebs - INFO - The newest snapshot for vol-89109472 is 1394 seconds old
2015-10-11 00:33:46,062 - auto-ebs - INFO - No need for a new snapshot of vol-89109472
2015-10-11 00:33:46,233 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:46,355 - auto-ebs - INFO - The newest snapshot for vol-7c109487 is 1389 seconds old
2015-10-11 00:33:46,355 - auto-ebs - INFO - No need for a new snapshot of vol-7c109487
2015-10-11 00:33:46,506 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:46,639 - auto-ebs - INFO - The newest snapshot for vol-c5f9663e is 1384 seconds old
2015-10-11 00:33:46,639 - auto-ebs - INFO - No need for a new snapshot of vol-c5f9663e
2015-10-11 00:33:46,834 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:46,961 - auto-ebs - INFO - The newest snapshot for vol-a6f9665d is 1380 seconds old
2015-10-11 00:33:46,961 - auto-ebs - INFO - No need for a new snapshot of vol-a6f9665d
2015-10-11 00:33:47,104 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:47,782 - auto-ebs - INFO - The newest snapshot for vol-f3ea7508 is 1376 seconds old
2015-10-11 00:33:47,783 - auto-ebs - INFO - No need for a new snapshot of vol-f3ea7508
2015-10-11 00:33:47,953 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:48,367 - auto-ebs - INFO - The newest snapshot for vol-00ea75fb is 1372 seconds old
2015-10-11 00:33:48,367 - auto-ebs - INFO - No need for a new snapshot of vol-00ea75fb
2015-10-11 00:33:48,528 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:48,870 - auto-ebs - INFO - The newest snapshot for vol-6059499b is 1366 seconds old
2015-10-11 00:33:48,870 - auto-ebs - INFO - No need for a new snapshot of vol-6059499b
2015-10-11 00:33:49,060 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:49,278 - auto-ebs - INFO - The newest snapshot for vol-f9415102 is 1362 seconds old
2015-10-11 00:33:49,278 - auto-ebs - INFO - No need for a new snapshot of vol-f9415102
2015-10-11 00:33:49,402 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:49,537 - auto-ebs - INFO - The newest snapshot for vol-f0f1ec0b is 1356 seconds old
2015-10-11 00:33:49,537 - auto-ebs - INFO - No need for a new snapshot of vol-f0f1ec0b
2015-10-11 00:33:49,658 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:49,804 - auto-ebs - INFO - The newest snapshot for vol-d7ecf12c is 1352 seconds old
2015-10-11 00:33:49,804 - auto-ebs - INFO - No need for a new snapshot of vol-d7ecf12c
2015-10-11 00:33:49,932 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:50,082 - auto-ebs - INFO - The newest snapshot for vol-05812bff is 1348 seconds old
2015-10-11 00:33:50,082 - auto-ebs - INFO - No need for a new snapshot of vol-05812bff
2015-10-11 00:33:50,244 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:50,380 - auto-ebs - INFO - The newest snapshot for vol-62862c98 is 1345 seconds old
2015-10-11 00:33:50,381 - auto-ebs - INFO - No need for a new snapshot of vol-62862c98
2015-10-11 00:33:50,501 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:50,632 - auto-ebs - INFO - The newest snapshot for vol-d4963c2e is 1343 seconds old
2015-10-11 00:33:50,632 - auto-ebs - INFO - No need for a new snapshot of vol-d4963c2e
2015-10-11 00:33:50,747 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:50,900 - auto-ebs - INFO - The newest snapshot for vol-9a973d60 is 1340 seconds old
2015-10-11 00:33:50,900 - auto-ebs - INFO - No need for a new snapshot of vol-9a973d60
2015-10-11 00:33:51,178 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:51,318 - auto-ebs - INFO - The newest snapshot for vol-99e56063 is 1339 seconds old
2015-10-11 00:33:51,318 - auto-ebs - INFO - No need for a new snapshot of vol-99e56063
2015-10-11 00:33:51,449 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:51,602 - auto-ebs - INFO - The newest snapshot for vol-bbe76241 is 1336 seconds old
2015-10-11 00:33:51,603 - auto-ebs - INFO - No need for a new snapshot of vol-bbe76241
2015-10-11 00:33:51,711 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:51,838 - auto-ebs - INFO - The newest snapshot for vol-a2a82f58 is 1333 seconds old
2015-10-11 00:33:51,838 - auto-ebs - INFO - No need for a new snapshot of vol-a2a82f58
2015-10-11 00:33:51,960 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:52,095 - auto-ebs - INFO - The newest snapshot for vol-72b73088 is 1331 seconds old
2015-10-11 00:33:52,096 - auto-ebs - INFO - No need for a new snapshot of vol-72b73088
2015-10-11 00:33:52,225 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:52,480 - auto-ebs - INFO - The newest snapshot for vol-1d9e0b74 is 1329 seconds old
2015-10-11 00:33:52,480 - auto-ebs - INFO - No need for a new snapshot of vol-1d9e0b74
2015-10-11 00:33:52,609 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:52,728 - auto-ebs - INFO - The newest snapshot for vol-12b5dd7b is 1326 seconds old
2015-10-11 00:33:52,728 - auto-ebs - INFO - No need for a new snapshot of vol-12b5dd7b
2015-10-11 00:33:52,860 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:52,990 - auto-ebs - INFO - The newest snapshot for vol-66e2ff0f is 1323 seconds old
2015-10-11 00:33:52,991 - auto-ebs - INFO - No need for a new snapshot of vol-66e2ff0f
2015-10-11 00:33:53,109 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:53,231 - auto-ebs - INFO - The newest snapshot for vol-889c81e1 is 1320 seconds old
2015-10-11 00:33:53,232 - auto-ebs - INFO - No need for a new snapshot of vol-889c81e1
2015-10-11 00:33:53,422 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:53,566 - auto-ebs - INFO - The newest snapshot for vol-c8227681 is 1315 seconds old
2015-10-11 00:33:53,566 - auto-ebs - INFO - No need for a new snapshot of vol-c8227681
2015-10-11 00:33:53,696 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:53,827 - auto-ebs - INFO - The newest snapshot for vol-28346561 is 1310 seconds old
2015-10-11 00:33:53,827 - auto-ebs - INFO - No need for a new snapshot of vol-28346561
2015-10-11 00:33:53,941 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:54,057 - auto-ebs - INFO - The newest snapshot for vol-31b6e478 is 1306 seconds old
2015-10-11 00:33:54,058 - auto-ebs - INFO - No need for a new snapshot of vol-31b6e478
2015-10-11 00:33:54,179 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:54,314 - auto-ebs - INFO - The newest snapshot for vol-c84af683 is 1301 seconds old
2015-10-11 00:33:54,314 - auto-ebs - INFO - No need for a new snapshot of vol-c84af683
2015-10-11 00:33:54,444 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:54,586 - auto-ebs - INFO - The newest snapshot for vol-6a61fd21 is 1295 seconds old
2015-10-11 00:33:54,586 - auto-ebs - INFO - No need for a new snapshot of vol-6a61fd21
2015-10-11 00:33:54,778 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:54,904 - auto-ebs - INFO - The newest snapshot for vol-bb3ba7f0 is 1291 seconds old
2015-10-11 00:33:54,904 - auto-ebs - INFO - No need for a new snapshot of vol-bb3ba7f0
2015-10-11 00:33:55,131 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:55,263 - auto-ebs - INFO - The newest snapshot for vol-31342cdf is 1286 seconds old
2015-10-11 00:33:55,263 - auto-ebs - INFO - No need for a new snapshot of vol-31342cdf
2015-10-11 00:33:55,428 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:55,593 - auto-ebs - INFO - The newest snapshot for vol-db31f33a is 1281 seconds old
2015-10-11 00:33:55,593 - auto-ebs - INFO - No need for a new snapshot of vol-db31f33a
2015-10-11 00:33:55,802 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:55,922 - auto-ebs - INFO - The newest snapshot for vol-7c36f49d is 1277 seconds old
2015-10-11 00:33:55,922 - auto-ebs - INFO - No need for a new snapshot of vol-7c36f49d
2015-10-11 00:33:56,076 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:56,229 - auto-ebs - INFO - The newest snapshot for vol-ec42ef0d is 1272 seconds old
2015-10-11 00:33:56,230 - auto-ebs - INFO - No need for a new snapshot of vol-ec42ef0d
2015-10-11 00:33:56,351 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:56,501 - auto-ebs - INFO - The newest snapshot for vol-8142ef60 is 1268 seconds old
2015-10-11 00:33:56,502 - auto-ebs - INFO - No need for a new snapshot of vol-8142ef60
2015-10-11 00:33:56,608 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:56,751 - auto-ebs - INFO - The newest snapshot for vol-049307e5 is 1262 seconds old
2015-10-11 00:33:56,751 - auto-ebs - INFO - No need for a new snapshot of vol-049307e5
2015-10-11 00:33:56,886 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:57,006 - auto-ebs - INFO - The newest snapshot for vol-149c08f5 is 1258 seconds old
2015-10-11 00:33:57,006 - auto-ebs - INFO - No need for a new snapshot of vol-149c08f5
2015-10-11 00:33:57,113 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:57,247 - auto-ebs - INFO - The newest snapshot for vol-f482c215 is 1253 seconds old
2015-10-11 00:33:57,247 - auto-ebs - INFO - No need for a new snapshot of vol-f482c215
2015-10-11 00:33:57,365 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:57,500 - auto-ebs - INFO - The newest snapshot for vol-7e82c29f is 1249 seconds old
2015-10-11 00:33:57,501 - auto-ebs - INFO - No need for a new snapshot of vol-7e82c29f
2015-10-11 00:33:57,763 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:57,910 - auto-ebs - INFO - The newest snapshot for vol-2aace7cb is 1243 seconds old
2015-10-11 00:33:57,910 - auto-ebs - INFO - No need for a new snapshot of vol-2aace7cb
2015-10-11 00:33:58,055 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:58,190 - auto-ebs - INFO - The newest snapshot for vol-28a4f4c9 is 1239 seconds old
2015-10-11 00:33:58,190 - auto-ebs - INFO - No need for a new snapshot of vol-28a4f4c9
2015-10-11 00:33:58,326 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:58,462 - auto-ebs - INFO - The newest snapshot for vol-a6a5f547 is 1234 seconds old
2015-10-11 00:33:58,462 - auto-ebs - INFO - No need for a new snapshot of vol-a6a5f547
2015-10-11 00:33:58,591 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:58,719 - auto-ebs - INFO - The newest snapshot for vol-46a5f5a7 is 1229 seconds old
2015-10-11 00:33:58,719 - auto-ebs - INFO - No need for a new snapshot of vol-46a5f5a7
2015-10-11 00:33:58,827 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:58,959 - auto-ebs - INFO - The newest snapshot for vol-56f1a3b7 is 1224 seconds old
2015-10-11 00:33:58,959 - auto-ebs - INFO - No need for a new snapshot of vol-56f1a3b7
2015-10-11 00:33:59,094 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:59,222 - auto-ebs - INFO - The newest snapshot for vol-6e6c318f is 1221 seconds old
2015-10-11 00:33:59,222 - auto-ebs - INFO - No need for a new snapshot of vol-6e6c318f
2015-10-11 00:33:59,348 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:59,485 - auto-ebs - INFO - The newest snapshot for vol-2a81dfcb is 1216 seconds old
2015-10-11 00:33:59,485 - auto-ebs - INFO - No need for a new snapshot of vol-2a81dfcb
2015-10-11 00:33:59,600 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:33:59,766 - auto-ebs - INFO - The newest snapshot for vol-312e14d0 is 1211 seconds old
2015-10-11 00:33:59,766 - auto-ebs - INFO - No need for a new snapshot of vol-312e14d0
2015-10-11 00:33:59,910 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:00,118 - auto-ebs - INFO - The newest snapshot for vol-d0494c31 is 1208 seconds old
2015-10-11 00:34:00,118 - auto-ebs - INFO - No need for a new snapshot of vol-d0494c31
2015-10-11 00:34:00,268 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:00,418 - auto-ebs - INFO - The newest snapshot for vol-de27263f is 1203 seconds old
2015-10-11 00:34:00,419 - auto-ebs - INFO - No need for a new snapshot of vol-de27263f
2015-10-11 00:34:00,558 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:00,728 - auto-ebs - INFO - The newest snapshot for vol-402a2ba1 is 1197 seconds old
2015-10-11 00:34:00,728 - auto-ebs - INFO - No need for a new snapshot of vol-402a2ba1
2015-10-11 00:34:00,872 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:00,996 - auto-ebs - INFO - The newest snapshot for vol-c5bc0025 is 1192 seconds old
2015-10-11 00:34:00,996 - auto-ebs - INFO - No need for a new snapshot of vol-c5bc0025
2015-10-11 00:34:01,123 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:01,278 - auto-ebs - INFO - The newest snapshot for vol-bda5195d is 1190 seconds old
2015-10-11 00:34:01,278 - auto-ebs - INFO - No need for a new snapshot of vol-bda5195d
2015-10-11 00:34:01,413 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:01,561 - auto-ebs - INFO - The newest snapshot for vol-f8a11d18 is 1188 seconds old
2015-10-11 00:34:01,561 - auto-ebs - INFO - No need for a new snapshot of vol-f8a11d18
2015-10-11 00:34:01,692 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:01,907 - auto-ebs - INFO - The newest snapshot for vol-bee0655e is 1185 seconds old
2015-10-11 00:34:01,907 - auto-ebs - INFO - No need for a new snapshot of vol-bee0655e
2015-10-11 00:34:02,051 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:02,210 - auto-ebs - INFO - The newest snapshot for vol-43e267a3 is 1184 seconds old
2015-10-11 00:34:02,210 - auto-ebs - INFO - No need for a new snapshot of vol-43e267a3
2015-10-11 00:34:02,360 - auto-ebs - INFO - No old snapshots to remove
2015-10-11 00:34:02,522 - auto-ebs - INFO - The newest snapshot for vol-4f645d86 is 1181 seconds old
2015-10-11 00:34:02,522 - auto-ebs - INFO - No need for a new snapshot of vol-4f645d86
2015-10-11 00:34:02,635 - auto-ebs - INFO - Deleting snapshot snap-3a6d2e76
2015-10-11 00:34:02,719 - auto-ebs - ERROR - 400 Bad Request
2015-10-11 00:34:02,719 - auto-ebs - ERROR -
InvalidSnapshot.InUseThe snapshot snap-3a6d2e76 is currently in use by ami-xxxxxxxx003bde58-e7cb-4fda-bd62-c066643c89c7
2015-10-11 00:34:02,719 - auto-ebs - WARNING - Could not remove snapshot: The snapshot snap-3a6d2e76 is currently in use by ami-833771e6
2015-10-11 00:34:02,719 - auto-ebs - INFO - Done deleting snapshots
2015-10-11 00:34:02,719 - auto-ebs - INFO - Sleeping 300 seconds until next check

[farassadek]

I do not see the debug enabled option, how can enable it?

[farassadek]

any update about this.

[sebdah]

Sorry, have been too busy with other stuff. I intent to take a look sometime during this week or weekend. But a quick look in the code says at least that the debug option is always enabled per default. But there is not so much debug logging :/

[farassadek]

good, keep me posted please.
If you need any help with the code, I would be happy to help.

[sebdah]

I have not been able to reproduce this, my daemon runs continuously without any crashes. @farassadek do you have an error like the one @downspot reported too in your logs? I can't really see why that error would pose an issue for the daemon though, but it's the only oddity I see in the log..

[downspot]

Yeah that error seems like a moot point it's only referring to a snapshot that can't be removed as it's in use for an AMI. On a side note I should point out that that I am also running a cron job in addition to the daemon that is automatically generating the config files for watched and unwatched volumes. I shouldn't be doing something like this however as the daemon could be in the middle of reading those files while running and possibly crash the process? Basically I am doing this on a job, every 45 mins:

/opt/rh/python27/root/usr/bin/automated-ebs-snapshots --config /etc/aws/auto-ebs-snapshots.conf --watch-file /etc/aws/auto-ebs-snapshots-volumes-active.conf

/opt/rh/python27/root/usr/bin/automated-ebs-snapshots --config /etc/aws/auto-ebs-snapshots.conf --unwatch-file /etc/aws/auto-ebs-snapshots-volumes-inactive.conf

Which is adding and removing volumes based on whether the instance is running or not. The daemon could be reading these files while running and possibly be causing my crashes. Also since reporting this in the ticket I haven't had a crash, so there's that.

[downspot]

Just looked in the logs again and there is nothing that's error based other than the one pasted above. It's been running clean. My thinking is there is some other underlying problem unrelated to automated-ebs-snapshots. Will continue to monitor on my side.

[farassadek]

YES,
I got this ERROR too. But I don't think this is the cause of the issue.

On Sat, Oct 17, 2015 at 2:12 PM, Sebastian Dahlgren <
notifications@github.com> wrote:

I have not been able to reproduce this, my daemon runs continuously
without any crashes. @farassadek https://github.com/FarasSadek do you
have an error like the one @downspot https://github.com/downspot
reported too in your logs? I can't really see why that error would pose an
issue for the daemon though, but it's the only oddity I see in the log..

—
Reply to this email directly or view it on GitHub
#21 (comment)
.

Faras Sadek
Senior System Engineer
Computing office
Harvard John A. Paulson School of Engineering and Applied Sciences
33 Oxford Street , MD G109
Cambridge MA 02138
fadel@seas.harvard.edu
http://www.seas.harvard.edu/directory/fadel

[downspot]

Possibly found what is killing the damon on my end. This is the last log entry I see before the damon died.

2015-12-06 01:57:25,042 - auto-ebs - ERROR - 400 Bad Request
2015-12-06 01:57:25,042 - auto-ebs - ERROR -
ConcurrentSnapshotLimitExceededToo many concurrent snapshot requests for a volume7a132795-ef08-4d45-929b-635552f24bc6