Harden event/connection handling after zenoh merge - #2232
Open
caesarniseko-coder wants to merge 1 commit into
Open
Harden event/connection handling after zenoh merge#2232caesarniseko-coder wants to merge 1 commit into
caesarniseko-coder wants to merge 1 commit into
Conversation
Reconcile local reliability fixes on top of the libp2p→zenoh update: - election: add configurable connection-election cooldown (default 25s) that drains topology-formation bursts so each distinct topology change triggers exactly one re-election; injectable via constructor for fast tests. - apply: skip out-of-order indexed events instead of asserting (defensive against event-idx gaps), and recompute thunderbolt_bridge_cycles when network interfaces or out-RDMA connections change. - download/coordinator, worker: guard event sends against Broken/ClosedResourceError on stream teardown. - test_election: run the connection-round test with a 0s cooldown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reliability hardening reconciled on top of the libp2p→zenoh migration. Two changes are new; two overlap fixes already merged and are included defensively (happy to drop them if redundant).
New:
25s). After a connection-triggered campaign,_connection_receiverwaits then drains connection events that arrived during topology formation, so each distinct topology change triggers exactly one re-election instead of perpetual re-elections from zenoh gossip/reconnect bursts. Injectable via the constructor (connection_election_cooldown) so tests can set it to0.thunderbolt_bridge_cycleswhen network interfaces or out-RDMA connections change (previously only updated onThunderboltBridgeInfo).Defensive / possibly-redundant with existing fixes:
assert-ing, guarding against event-idx gaps. Overlaps Fix out of order event idx causing fatal crashes #1894 / Fix event mutation causing indexed vs event mismatch #1964.BrokenResourceError/ClosedResourceErroron stream teardown. Overlaps fix: handle BrokenResourceError in download progress callback #1846 (kept only in the secondary send sites the broad guard doesn't cover).Testing
uv run pytest src -m "not slow" --import-mode=importlib→ 469 passed, 3 skippedruff check,ruff format --check,basedpyright→ clean on changed filesuv run exo(Apple Silicon, MLX + zenoh): server boots, node elects master + starts Worker/DownloadCoordinator/API, dashboard renders,GET /v1/modelsandGET /statereturn 200, hardware telemetry (macmon) live.🤖 Generated with Claude Code