Loki Backend hangs indefinitely during shutdown [revisit]

[ellisonch]

Describe the bug
I'm having the problem described in #9523, in loki 3.5.7. Loki backend cannot shutdown cleanly without killing loki-read. Issue #9523 claims the bug has been fixed, but I'm seeing this bug in 3.5.7.

I am unable to re-open that issue, so I'm creating a new issue. Feel free to close and reopen the original.

To Reproduce
Steps to reproduce the behavior:

1. delete the loki-backend-0 pod while loki-read-* is running

Expected behavior
Loki-backend should restart within a reasonable amount of time. Instead, it hangs for 5 minutes and then gets forcibly terminated.

If I kubectl delete pod on loki-read while backend is shutting down, backend immediately shuts down cleanly.

Environment:

• Infrastructure: kubernetes
• Deployment tool: terraform and helm chart

repository       = "https://grafana.github.io/helm-charts"
chart            = "loki"
version          = "6.46.0"
...
deploymentMode   = "SimpleScalable"