CILR: sequencer not completing many supervised futures, waiting for a long time (over 5 hours) #3473
Description
Some sequencers on CILR are not functioning at the moment (many in flight submissions in the participant).
See cilr logs for sv-7 which is still in a broken state.
(We restarted sv-12, which got unstuck and started to catch up.)
The symptom we see in the logs:
Supervised futures for the following trace IDs have not completed with alert log level WARN:
b10f5505bfe86df148c93766a49b43fc(19700 seconds)
2546c788078f86edcd309c033e1b56f5(19700 seconds)
e0fab624f66659bd589d78c17d7a8b89 x2(19700 seconds..19700 seconds)
8895e695cbcf535ebeb264c87bdfb4c1 x2(19700 seconds..19700 seconds)
3c22b6b98cf1716fd6466877c3e5abda x3(19700 seconds..19700 seconds)
63c481c45199809b18423ab66a7235c4 x2(19700 seconds..19700 seconds)
388c465fc12ff0acd31db6d0ba5d543f(19700 seconds)
0cd9364f872497536a153222042a594f(19700 seconds)
5888f3c9e7660f579213d6de8e8c64ad x3(19700 seconds..19700 seconds)
Looks like these tasks are stuck.
Grafana sequencer sv-7
Grafana participant sv-7
