Environment
- exo official app 1.0.71, 5 nodes: 4× Mac mini M4 (macOS 26.5.1) cabled Thunderbolt (full RDMA ring visible in topology) + 1 Ethernet-only node
Symptom
When a node leaves the cluster (app quit or its exo process killed), its identity stays in GET /state → nodeIdentities for a long time (well over 10 minutes in our tests — it survived repeated checks and individual node restarts). Because MlxJaccl placement appears to require all current cluster members to be part of the RDMA ring, the lingering identity blocks RDMA placement even though every live node is RDMA-connected:
GET /instance/previews?model_id=...
→ "Requested RDMA (MlxJaccl) but no RDMA-connected cycles available"
GET /state
→ nodeIdentities: 5 entries # one is the departed node
→ topology RDMA links: 4/4 live nodes each with degree 2 (complete ring)
Restarting individual nodes makes it worse: each restart mints a new node ID and the old one lingers too, so the ghost count grows. The only reliable way we found to clear ghosts was a coordinated stop of every node followed by a fresh start (then nodeIdentities returns to exactly the live set).
Repro
- Form a cluster of N nodes (N−1 of them RDMA/Thunderbolt-connected, 1 Ethernet-only).
- Quit/kill exo on the Ethernet-only node.
- Wait —
nodeIdentities keeps reporting N entries long after the node is gone.
GET /instance/previews for any model keeps reporting MlxJaccl unavailable ("no RDMA-connected cycles") despite the live nodes forming a complete RDMA ring in topology.
Expected
Departed node identities should expire from cluster state within a bounded, short interval (or on liveliness loss), and placement should consider only live members — otherwise a single flaky/departed device disables RDMA placement for the whole cluster until a full-cluster restart.
(Separate but related observation while testing: on a clean 4-node cluster where all members are in the RDMA ring, MlxJaccl still reported no cycles with links negotiated at 20 Gb/s over plain USB-C cabling — if certified 40 Gb/s Thunderbolt cables are a hard requirement for JACCL, it would be great to surface that in the error message instead of the generic "no RDMA-connected cycles available".)
Environment
Symptom
When a node leaves the cluster (app quit or its
exoprocess killed), its identity stays inGET /state → nodeIdentitiesfor a long time (well over 10 minutes in our tests — it survived repeated checks and individual node restarts). Because MlxJaccl placement appears to require all current cluster members to be part of the RDMA ring, the lingering identity blocks RDMA placement even though every live node is RDMA-connected:Restarting individual nodes makes it worse: each restart mints a new node ID and the old one lingers too, so the ghost count grows. The only reliable way we found to clear ghosts was a coordinated stop of every node followed by a fresh start (then
nodeIdentitiesreturns to exactly the live set).Repro
nodeIdentitieskeeps reporting N entries long after the node is gone.GET /instance/previewsfor any model keeps reporting MlxJaccl unavailable ("no RDMA-connected cycles") despite the live nodes forming a complete RDMA ring intopology.Expected
Departed node identities should expire from cluster state within a bounded, short interval (or on liveliness loss), and placement should consider only live members — otherwise a single flaky/departed device disables RDMA placement for the whole cluster until a full-cluster restart.
(Separate but related observation while testing: on a clean 4-node cluster where all members are in the RDMA ring, MlxJaccl still reported no cycles with links negotiated at 20 Gb/s over plain USB-C cabling — if certified 40 Gb/s Thunderbolt cables are a hard requirement for JACCL, it would be great to surface that in the error message instead of the generic "no RDMA-connected cycles available".)