Universal re-try limits on celery tasks

If a task crashes a pod, It will keep doing so for quite some until we stop generating pods.
We should impose some reasonable retry limit with a descriptive log message for celery tasks.

https://lcoglobal.redmineup.com/issues/2707