Singularity containers do not stop gracefully

[brianaydemir]

Unlike Docker containers, Singularity containers appear to timeout when being asked to stop:

$ singularity instance stop xfer-ep
INFO:    Stopping xfer-ep instance of /home/baydemir/.singularity/cache/oras/sha256:f0932903acf8151974e7f4cbff169f83854bba1ba1e00aaa48a9daecdc6fb978 (PID=12719)
INFO:    Killing xfer-ep instance of /home/baydemir/.singularity/cache/oras/sha256:f0932903acf8151974e7f4cbff169f83854bba1ba1e00aaa48a9daecdc6fb978 (PID=12719) (Timeout)

The result is that collector retains an entry for the EP until it decides that it has been too long without an update.