bugfix: mlx5 ci hung - #622
Open
QizhouZhang97 wants to merge 9 commits into
Open
Conversation
Remove leftover mori* containers and GPU/RDMA processes on both nodes before starting internode CI to avoid EP stress hangs on dirty runners. Co-authored-by: Cursor <cursoragent@cursor.com>
Run remote host cleanup via bash -s heredoc so pkill patterns are not embedded in the ssh command line, which was killing the session. Co-authored-by: Cursor <cursoragent@cursor.com>
Bench can leave torchrun/mlx5 state that makes the first stress case hit timeout 600 (exit 124). Reset both nodes before stress and retry the sweep once after killing lingering GPU workers. Co-authored-by: Cursor <cursoragent@cursor.com>
Test 2 launched its kernel straight after the local memset, so PE 0's put-imm could land before PE 1 had memset its own buffer. PE 1 then overwrote the delivered 42s with 1 and its kernel spun on the clobbered values forever, hanging hipDeviceSynchronize until CI's timeout killed the job (exit 124). Test 1 and every other cross-PE kernel launch in the shmem examples already barrier at this point; only this one was missing it. Reproduced deterministically by delaying PE 1's memset: 6/6 hangs without the barrier, 6/6 passes with it. Co-authored-by: Cursor <cursoragent@cursor.com>
The self-hosted runner moved off banff-ccs-aus-p19-29, so node1 (the runner itself) and node2 (its ssh peer) now resolve to p19-05 and p19-14. Both boxes have the same NIC layout as the old pair -- mlx5_0..7 on InfiniBand plus mlx5_8 behind ens14np0 -- so rdma_devices and the ifnames are unchanged. The runner label still reads 300_mlx_p19_29 because the agent registration was moved rather than recreated; renaming it needs a fresh registration token. Co-authored-by: Cursor <cursoragent@cursor.com>
The CI container mounts the workspace and runs as root, so every internode job leaves root-owned build output behind on node2. node1 recovers from this via "Fix workspace permissions" before checkout, but node2 has no checkout and no equivalent step, so the next job's rsync hit EPERM on those files and failed "Start container on node2" within seconds, before anything was built. Co-authored-by: Cursor <cursoragent@cursor.com>
The collapsed-CQ drain bounded its reconstruction by postIdx when called from the live quiet. postIdx is the reservation counter, bumped before a warp passes the recycle gate, so it runs tens of thousands of entries ahead of what is actually in the send queue. That slack let a wqe_counter from outside the live window pass the sanity check -- the 16-bit value wraps into a plausible delta -- and pushed doneIdx past WQEs the NIC had not fetched yet. The recycle gate then handed those still-live SQ slots out for reuse and the overwritten writes were lost, which hangs any peer spinning on the data. Reproduced with concurrent_put_thread under MORI_DISABLE_P2P=ON: test 3 pushes 262144 single-dword puts through one QP against a 4096-entry SQ, so the gate recycles constantly. It hung in roughly 21% of runs (15/70), always in test 3, with the receiver missing scattered wave-sized clusters of writes -- up to 131072 of them. Bounding by dbTouchIdx instead drops that to 2/165. Co-authored-by: Cursor <cursoragent@cursor.com>
jhchouuu
approved these changes
Sep 2, 2026
Comment on lines
+900
to
+941
| - name: Reset GPU processes before EP internode stress | ||
| run: | | ||
| # Bench runs 20 torchrun sessions; lingering workers or mlx5 UAR mappings can | ||
| # leave the first stress case hung until timeout 600 (exit 124). | ||
| $CT exec $CONTAINER bash -c 'pkill -9 -f "[t]orchrun" 2>/dev/null || true; pkill -9 -f "[t]est_dispatch_combine_internode" 2>/dev/null || true; pkill -9 -f "[t]ests\.python\.io\.benchmark" 2>/dev/null || true; sleep 3' | ||
| ssh $SSH_OPTS -p $NODE2_PORT $(whoami)@$NODE2_HOST \ | ||
| "$CT exec $CONTAINER bash -c 'pkill -9 -f \"[t]orchrun\" 2>/dev/null || true; pkill -9 -f \"[t]est_dispatch_combine_internode\" 2>/dev/null || true; pkill -9 -f \"[t]ests\\.python\\.io\\.benchmark\" 2>/dev/null || true; sleep 3'" | ||
|
|
||
| - name: MORI-EP internode normal kernel stress | ||
| run: | | ||
| SCRIPT="$GITHUB_WORKSPACE/tools/run_internode_test.sh" | ||
| PORT=29160 | ||
| for KERNEL in v1 v1_ll; do | ||
| for TOKENS in 64 128 1024 2048 4096; do | ||
| echo "=== stress kernel=$KERNEL max-tokens=$TOKENS port=$PORT ===" | ||
| NODE2_CMD=( | ||
| $CT exec -e MORI_INTERNODE_TIMEOUT=600 $CONTAINER bash $SCRIPT | ||
| --rank 1 --master-addr $NODE1_HOST --master-port $PORT | ||
| --ifname $NODE2_IFNAME | ||
| --cmd stress --kernel-type $KERNEL --max-tokens $TOKENS | ||
| ) | ||
| ssh $SSH_OPTS -p $NODE2_PORT $(whoami)@$NODE2_HOST "${NODE2_CMD[*]}" & | ||
| $CT exec -e MORI_INTERNODE_TIMEOUT=600 $CONTAINER bash $SCRIPT \ | ||
| --rank 0 --master-addr $NODE1_HOST --master-port $PORT \ | ||
| --ifname $NODE1_IFNAME \ | ||
| --cmd stress --kernel-type $KERNEL --max-tokens $TOKENS | ||
| wait | ||
| sleep 1 | ||
| PORT=$((PORT + 1)) | ||
| reset_ep_gpu() { | ||
| $CT exec $CONTAINER bash -c 'pkill -9 -f "[t]orchrun" 2>/dev/null || true; pkill -9 -f "[t]est_dispatch_combine_internode" 2>/dev/null || true; pkill -9 -f "[t]ests\.python\.io\.benchmark" 2>/dev/null || true; sleep 3' | ||
| ssh $SSH_OPTS -p $NODE2_PORT $(whoami)@$NODE2_HOST \ | ||
| "$CT exec $CONTAINER bash -c 'pkill -9 -f \"[t]orchrun\" 2>/dev/null || true; pkill -9 -f \"[t]est_dispatch_combine_internode\" 2>/dev/null || true; pkill -9 -f \"[t]ests\\.python\\.io\\.benchmark\" 2>/dev/null || true; sleep 3'" | ||
| } | ||
| run_stress_sweep() { | ||
| local rc=0 | ||
| SCRIPT="$GITHUB_WORKSPACE/tools/run_internode_test.sh" | ||
| PORT=29160 | ||
| for KERNEL in v1 v1_ll; do | ||
| for TOKENS in 64 128 1024 2048 4096; do | ||
| echo "=== stress kernel=$KERNEL max-tokens=$TOKENS port=$PORT ===" | ||
| NODE2_CMD=( | ||
| $CT exec -e MORI_INTERNODE_TIMEOUT=600 $CONTAINER bash $SCRIPT | ||
| --rank 1 --master-addr $NODE1_HOST --master-port $PORT | ||
| --ifname $NODE2_IFNAME | ||
| --cmd stress --kernel-type $KERNEL --max-tokens $TOKENS | ||
| ) | ||
| ssh $SSH_OPTS -p $NODE2_PORT $(whoami)@$NODE2_HOST "${NODE2_CMD[*]}" & | ||
| $CT exec -e MORI_INTERNODE_TIMEOUT=600 $CONTAINER bash $SCRIPT \ | ||
| --rank 0 --master-addr $NODE1_HOST --master-port $PORT \ | ||
| --ifname $NODE1_IFNAME \ | ||
| --cmd stress --kernel-type $KERNEL --max-tokens $TOKENS | ||
| wait | ||
| case_rc=$? | ||
| if [ $case_rc -ne 0 ]; then | ||
| rc=$case_rc | ||
| break 2 | ||
| fi | ||
| sleep 1 | ||
| PORT=$((PORT + 1)) | ||
| done |
Collaborator
There was a problem hiding this comment.
remove reset gpu if not necessary
The container/process sweep and the stress retry were workarounds for zombie IBGDA repros left on p19-29 / p20-29. The pair now runs on p19-05 / p19-14, so let a hang fail the job instead of hiding it. Co-authored-by: Cursor <cursoragent@cursor.com>
QizhouZhang97
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
migrate mlx ci to 19-05 & 19-14
fix mlx poll cq bug: clamp doneIdx against dbTouchIdx, not postIdx — the old
window exceeded 65536, so a stale wqe_counter wrapped into a bogus advance and
the recycle gate overwrote live SQ slots.