[ENHANCEMENT] easier identification of failure nodes and failure reason

[ischlag]

runs fail often but understanding the reason of the failure is currently manual labor which makes it hard to add faulty nodes to the node exclude list and communicate those failures to CSCS. We need a solution that can automatically identify known issues from the slurm err log.

I'm not sure what the best is to solve this issue.

[dhia680]

I suggest we first manually collect some error logs and cluster them in groups with descriptions (NCCL, OOM, Node failure, intervention in the cluster...).
Then write a simple bash script that checks for these in the error log file and eventually sends us a slack message (using the slackbot implemented here) that localizes the error and probably explains it.