Skip to content

fix: prune stale node identity state on timeout - #2250

Open
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/2194-prune-stale-node-identities
Open

fix: prune stale node identity state on timeout#2250
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/2194-prune-stale-node-identities

Conversation

@abendrothj

Copy link
Copy Markdown

Summary

  • remove timed-out nodes from node_identities so ghost peers cannot block RDMA placement
  • remove the same node from the newer per-node node_backends map
  • preserve identity and backend state for live nodes
  • add regression coverage for the timeout transition

Testing

  • uv run pytest src/exo/shared/tests/test_apply/test_apply_node_timed_out.py
  • uv run basedpyright src/exo/shared/apply.py src/exo/shared/tests/test_apply/test_apply_node_timed_out.py
  • uv run ruff check src/exo/shared/apply.py src/exo/shared/tests/test_apply/test_apply_node_timed_out.py

Fixes #2194


Refiled from #2211, which was closed unintentionally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale node identities linger in cluster state after a node exits, blocking MlxJaccl (RDMA) placement

1 participant