RSDK-10618: Fix data race when modules crashes during reconfiguration

[jmatth]

This has a matching change in viamrobotics/goutils#432 that actually fixes the race condition. Unfortunately the way modmanager is written meant that fixing the race caused a deadlock, which this PR addresses.

[dgottlieb]

If I'm reading this right, if a module crashes at the same time the mod manager has the lock for an unrelated reason (perhaps adding a resource to a different module), we'll give up on restarting the module? Is there some other means that would restart it?

And I suppose the problem (based on the comment) from just using the mod.inRecoveryLock here is that module reconfiguration does not take the in recovery lock, but would need to (for the purpose of serializing with restarts -- which it definitely does from a logic perspective)?

[jmatth]

we'll give up on restarting the module

Correct

Is there some other means that would restart it?

Not that I'm aware of

And I suppose the problem (based on the comment) from just using the mod.inRecoveryLock here is that module reconfiguration does not take the in recovery lock...

Correct, that lock was only ever taken by the restart callback. I actually removed it (and that weird atomic bool) since taking the modmananger lock made it unnecessary.