Question: supervisor tree fault tolerance

[photoszzt]

Hi Monarch developer,

I don't find this anywhere in the document. What happen when the process of the root of the supervisor tree fails due to either OOM or machine failure of the running node? Would everything below the root just die? Is there a leader election process to establish the new leader to be the new supervisor? I try to search the codebase and I don't see any raft or paxos references.

[dulinriley]

Is your question answered by this existing issue?
#2198

Right now if any actor (including the client) hits a fault such as an OOM and cannot run cleanup, any actors it spawned will
be orphaned and unable to deliver any further supervision messages up the tree.

This is not what we want, we want the behavior you mention where everything below the dead actor would die. I'm working on this right now, will have a PR up in hopefully a few days

[photoszzt]

Yes, partially. I'm not talking about the GC question.

My question is would there be fault tolerance on the owner of the mesh where a new owner will be elected then the child reconnect with the new leader. Ray does this via GCS fault tolerance with reliance on a external HA redis server. Even head is dead, the actor won't be influenced and can reconnect back when the GCS server is back. If head doesn't go back, then I think Ray worker node would gc the actor and the whole cluster would die.

I'm wondering whether Monarch would choose a different solution, e.g. without relying on external service for fault tolerance. For long running training, fault tolerance is essential.

[dulinriley]

My question is would there be fault tolerance on the owner of the mesh where a new owner will be elected then the child reconnect with the new leader

If the owner of a resource (host, proc, actor, meshes of these) is dead, then the resource itself will die. We do not currently support moving the resource to a new owner either before or after the fault which destroys the owner. What we do have is that the owner can receive errors from the children it owns via __supervise__, and then restart the mesh those children live on (fine-grained recovery of a single resource in a mesh is also TODO). If you can find a way to invert the relationship such that you only care if resources you own die instead of the owner, then you could accomplish this with existing APIs.

We could consider a proposal for this, for example another dunder method on an actor for choosing a new owner if its current one is dead. It would probably mean you'll need a Mesh of possible owners and then you choose one of those as an ActorRef and send it a message requesting for it to become the owner. If you'd like to you can write this issue as an RFC (can be this issue or a new issue) where you propose a design for solving this problem.