feat(networking): allow dialing peers over unicast via EXO_ZENOH_CONNECT - #2243
Open
Bawan-Wang wants to merge 3 commits into
Open
feat(networking): allow dialing peers over unicast via EXO_ZENOH_CONNECT#2243Bawan-Wang wants to merge 3 commits into
Bawan-Wang wants to merge 3 commits into
Conversation
Peer discovery relies solely on IPv6 link-local multicast to the group ff12::e0a1:de89. Many Wi-Fi access points isolate wireless clients and do not forward link-local multicast between them, so two nodes on the same subnet never discover each other even though unicast between them works fine. There is currently no fallback: --bootstrap-peers raises "Bootstrap peers has been temporarily removed". Multicast is only used to learn a peer's address; the actual link is a unicast TCP connection established by connect_peer(). This adds an optional EXO_ZENOH_CONNECT environment variable holding a comma-separated list of zenoh endpoints (e.g. "tcp/192.168.1.2:52414") that is injected into the session's connect/endpoints, letting an operator dial known peers directly and bypass discovery entirely. The variable is opt-in: when unset, configuration is unchanged. Only one side needs to set it, since the TCP link is bidirectional and scouting/gossip/multihop is already enabled. Verified on two WSL2 nodes over Wi-Fi where discovery had never worked: tcpdump on udp/52413 showed only locally originated multicast and no inbound packets from the peer. With EXO_ZENOH_CONNECT set on one node only, both nodes reported nodes: 2 with a populated connections map, and the link remained stable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…EP_SIZE
The prefill step size was hardcoded to 4096. On a small card the shard's
weights leave little room: with a 14B 4-bit model split across two 8GB GPUs
each runner holds ~4.15GB of weights and has only 2.5-3.4GB left, and a
4096-token prefill chunk exceeds that. The request then dies with
auto_parallel.py:167 mx.eval(output)
RuntimeError: cudaMallocAsync(&data, size, stream) failed: out of memory
which kills the runner and tears the instance down, so a client that sends
long prompts (an agent system prompt plus a repo map) never reaches decode.
Reading the value from the environment lets an operator trade prefill
throughput for headroom without patching the source on every node. Unset
keeps upstream behaviour.
Measured on that 2x8GB setup: lowering it is not by itself a fix — 512 still
OOMs on a ~1000-token prompt, so the dominant term is elsewhere. The knob is
still worth having on machines where the shard leaves more room.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uster Both nodes need a handful of environment variables to start exo at all, and the two sets genuinely differ, so keeping them in one file in git is what stops the machines drifting apart — a split we have now paid for twice, once with mlx-cpu vs mlx-cuda and once with mlx-cuda12 vs mlx-cuda13. Shared: LD_PRELOAD (anaconda ships libstdc++ 3.4.29, transformers loads it first and libmlx then fails on GLIBCXX_3.4.30) and OVERRIDE_MEMORY_MB (exo reports system RAM but CUDA runs out of VRAM, so placement over-assigns layers without it). MSI only: EXO_ZENOH_CONNECT, since the Wi-Fi AP drops IPv6 link-local multicast and one side dialling the other over unicast is enough. APU-TPNB04 only: CUDA_HOME and LD_LIBRARY_PATH, because its /usr/local/cuda is 12.6 — that cuda_fp8.h has no __nv_fp8_e8m0 so nvrtc cannot JIT mlx kernels, and its lib64 only carries libcublasLt.so.12. The script refuses to run on an unknown host rather than guessing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Peer discovery relies solely on IPv6 link-local multicast to the group ff12::e0a1:de89. Many Wi-Fi access points isolate wireless clients and do not forward link-local multicast between them, so two nodes on the same subnet never discover each other even though unicast between them works fine. There is currently no fallback: --bootstrap-peers raises "Bootstrap peers has been temporarily removed".
Multicast is only used to learn a peer's address; the actual link is a unicast TCP connection established by connect_peer(). This adds an optional EXO_ZENOH_CONNECT environment variable holding a comma-separated list of zenoh endpoints (e.g. "tcp/192.168.1.2:52414") that is injected into the session's connect/endpoints, letting an operator dial known peers directly and bypass discovery entirely.
The variable is opt-in: when unset, configuration is unchanged. Only one side needs to set it, since the TCP link is bidirectional and scouting/gossip/multihop is already enabled.
Verified on two WSL2 nodes over Wi-Fi where discovery had never worked: tcpdump on udp/52413 showed only locally originated multicast and no inbound packets from the peer. With EXO_ZENOH_CONNECT set on one node only, both nodes reported nodes: 2 with a populated connections map, and the link remained stable.
Motivation
Changes
Why It Works
Test Plan
Manual Testing
Automated Testing