libteam: Fix teamsyncd fd corruption during teamd -r#28618
Open
markx-arista wants to merge 1 commit into
Open
Conversation
…t_fd team_handle is allocated with myzalloc which zeroes all fields, setting event_fd to 0. If team_init fails before reaching team_init_event_fd (e.g. port_list_init returns EADDRNOTAVAIL during teamd -r restart), event_fd remains 0. team_free then calls close(0), closing stdin. nl_cli.sock is the only socket connected during team_alloc. After stdin is closed, each subsequent team_alloc gets nl_cli.sock on fd 0 (lowest available). Each subsequent failed team_free repeats close(event_fd=0), which closes whatever currently holds fd 0 — including a previously successful TeamPortSync's nl_cli.sock. This cascading corruption causes NLE_BAD_SOCK (-3) and NLE_SEQ_MISMATCH (-16) on affected handles, preventing team_ifindex2ifname from resolving member ports. Fix by initializing event_fd to -1 in team_alloc and guarding the close in team_free. Signed-off-by: markxiao <markxiao@arista.com>
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
markx-arista
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
process_monitoring/test_critical_process_monitoring.py has been flaky on t1 topos with portchannels and Arista-7060X6-64PE-B-P32O64 had 100% failure rate on t1-f2-d10u8 since sonic-net/sonic-swss#3984 merged.
After SIGKILL + config reload, some PortChannels have members "not synced" (S* in show int portchannel). APP_LAG_MEMBER_TABLE is empty for affected LAGs, so orchagent never programs members in the ASIC. BGP sessions on these LAGs stay permanently down.
sonic-net/sonic-swss#3984 is not the root cause, but it changed timing and exposed the issue.
Work item tracking
How I did it
team_handle is allocated with myzalloc which zeroes all fields, setting event_fd to 0. If team_init fails before reaching team_init_event_fd (e.g. port_list_init returns EADDRNOTAVAIL during teamd -r restart), event_fd remains 0. team_free then calls close(0), closing stdin.
nl_cli.sock is the only socket connected during team_alloc. After stdin is closed, each subsequent team_alloc gets nl_cli.sock on fd 0 (lowest available). Each subsequent failed team_free repeats close(event_fd=0), which closes whatever currently holds fd 0 — including a previously successful TeamPortSync's nl_cli.sock. This cascading corruption causes NLE_BAD_SOCK (-3) and NLE_SEQ_MISMATCH (-16) on affected handles, preventing team_ifindex2ifname from resolving member ports.
Fix by initializing event_fd to -1 in team_alloc and guarding the close in team_free.
How to verify it
Run process_monitoring/test_critical_process_monitoring.py on t1 topos with portchannels
Which release branch to backport (provide reason below if selected)
Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
Failure type:
Tested branch
Test result
On 202511, process_monitoring/test_critical_process_monitoring.py passed with Arista-7060X6-64PE-B-P32O64 passed on t1-f2-d10u
Description for the changelog
Initialize event_fd to -1 in team_alloc and guarding the close in team_free
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)