This is a problem that causes pod to restart e.g. if the Internet connectivity is lost and messages can not be received/checked.
Most likely we can just do nothing on errors occured during receiving.
Similar logic should be implemented for removing messages from queue and removing files from S3, but in this case a proper retry mechanism (with exponential backoff preferably) should be implemented.
Whatever logic would be implemented for these two operations has to be atomic. One problematic flow that comes to mind:
- message is received
- file is removed from S3
- program fails to remove the message from queue and gets restarted
This would currently cause a crash loop on the "remove file from S3" operation, so especially the "you're trying to remove a file that doesn't exist" scenario should be implemented with care.
Preferably I'd like to read the error message and decide based on that if possible, instead of adding an unnecessary HEAD call or similar that would first check if the file exists.
This is a problem that causes pod to restart e.g. if the Internet connectivity is lost and messages can not be received/checked.
Most likely we can just do nothing on errors occured during receiving.
Similar logic should be implemented for removing messages from queue and removing files from S3, but in this case a proper retry mechanism (with exponential backoff preferably) should be implemented.
Whatever logic would be implemented for these two operations has to be atomic. One problematic flow that comes to mind:
This would currently cause a crash loop on the "remove file from S3" operation, so especially the "you're trying to remove a file that doesn't exist" scenario should be implemented with care.
Preferably I'd like to read the error message and decide based on that if possible, instead of adding an unnecessary HEAD call or similar that would first check if the file exists.