Infrequent hang on shutdown

[nmichlo]

It seems as though very infrequently the Cilicon runner hangs on shutdown. Not sure how to reproduce this or debug this, but it sometimes just says terminating with nothing happening after that?

This does cause a problem as manual intervention is needed for automated runners to get them working again.

[Marcocanc]

Hi @nmichlo, we are aware of this issue. Unfortunately I couldn't find a solution and I believe it might be an issue internal to Virtualization.framework. Basically, when calling try await vm.stop(), it neither returns nor throws an error, leading to the stuck state. I've been trying out different strategies for a potential workaround, but haven't found a good one yet.

[nmichlo]

Thank you for the explanation @Marcocanc

It might be unsafe (so possibly an experimental config option to enable), but would it be possible to be able to set a timeout on this buggy operation, and if the timeout is reached we just force kill everything?