Azure: check SRIOV on all nics and retry switch_sriov - #4659
Azure: check SRIOV on all nics and retry switch_sriov#4659mcgov (mcgov) wants to merge 3 commits into
Conversation
784f8c6 to
1418d99
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates Azure SR-IOV handling in LISA’s Azure feature layer to make SR-IOV enablement checks accurate for multi-NIC VMs and to improve resiliency of SR-IOV toggling against transient Azure Network RP/ARM errors.
Changes:
- Add retry handling to
switch_sriovto tolerate transientHttpResponseErrorresponses. - Update
is_enabled_sriovto validate accelerated networking state across all VM NICs (not just the primary).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
1418d99 to
57b3d7c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:1048
nic_info.enable_accelerated_networkingcan beNone(it’s treated as optional elsewhere in this file), so using&=can raiseTypeError(bool&NoneType) and also reads like a bitwise operation. Use a booleanandcheck (oris True) so the method always returns a realbooland treatsNoneas disabled.
sriov_enabled &= nic_info.enable_accelerated_networking
lisa/sut_orchestrator/azure/features.py:991
- The retry decorator uses a fixed 30s delay with no backoff/jitter. This can significantly extend test runtime (up to ~60s extra plus Azure calls) on transient failures, and it’s inconsistent with other
HttpResponseErrorretries in this module that use short delays with backoff. Consider adding a brief rationale for the 30s delay and/or using a backoff strategy.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
57b3d7c to
c49b9aa
Compare
c49b9aa to
6ba6238
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1048
nic_info.enable_accelerated_networkingfrom the Azure SDK can beNone(Optional[bool]). Usingsriov_enabled &= nic_info.enable_accelerated_networkingcan raise aTypeErrorwhen the value isNone, and&=is a bitwise op that’s less clear than logical AND for booleans. Coerce the SDK value to a boolean (and treatNoneas disabled) while keeping the result strictlybool.
sriov_enabled &= nic_info.enable_accelerated_networking
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:1048
- Major:
nic_info.enable_accelerated_networkingcan beNone(elsewhere this field is compared withis False), andsriov_enabled &= Nonewill raiseTypeError. Coerce the value to a strict bool (treatNoneas disabled) before combining results.
self._resource_group_name, nic_name
)
sriov_enabled &= nic_info.enable_accelerated_networking
lisa/sut_orchestrator/azure/features.py:991
- Minor:
@retry(HttpResponseError, tries=3, delay=30)retries all HttpResponseError cases with a fixed 30s delay, which can make permanent failures slower and is inconsistent with the shorter/backoff-based HttpResponseError retries elsewhere in this file. Consider either (a) documenting why 30s is needed here, or (b) aligning with the backoff pattern used in other Azure RP retries / narrowing the retried conditions to transient cases.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
6ba6238 to
786f228
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1049
sriov_enabled &= nic_info.enable_accelerated_networkingrelies onenable_accelerated_networkingbeing a strictbool. Elsewhere in this file the code checksx.enable_accelerated_networking is False(around line ~809), which suggests this field may also beNone. In that case,&=will raiseTypeError: unsupported operand type(s) for &= 'bool' and 'NoneType'. Consider using logicalandand coercingNonetoFalse.
self._resource_group_name, nic_name
)
sriov_enabled &= nic_info.enable_accelerated_networking
return sriov_enabled
786f228 to
802beb1
Compare
0e45b16 to
c0e79ea
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1051
- The boolean accumulation logic is hard to read and non-idiomatic in Python (
&=on a bool plus a conditional expression), and it also prevents early-exit once a NIC is found with accelerated networking disabled. Consider returning early when any NIC is not enabled (this also naturally treatsNoneas disabled).
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
return sriov_enabled
c0e79ea to
7ae2d00
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:991
- Minor: The new retry uses hard-coded values (tries=3, delay=30) without any rationale. Since this affects how long SR-IOV enable/disable can stall on transient Azure NRP errors, please add a brief inline explanation so future changes don’t accidentally shorten/extend this wait.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
lisa/sut_orchestrator/azure/features.py:1050
- Minor:
is_enabled_sriovuses a&=reduction with an inline conditional, which is hard to read and doesn’t short-circuit once a NIC is found without accelerated networking. This can be simplified to an early return and will avoid extra ARM GET calls after the first failure.
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
7ae2d00 to
d83b8e2
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:991
- The new retry on
switch_sriovintroduces magic-number retry parameters (tries=3,delay=30) without documenting why those values are appropriate. Please add a short comment describing the transient failure being handled and the intended total retry window, so future readers can tune it safely.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
lisa/sut_orchestrator/azure/features.py:1051
sriov_enabled &= ...is hard to read and doesn’t short-circuit; it also makes it less obvious that onlyTrueshould be treated as enabled (everything else should be treated as disabled). Consider returning early when any NIC is not explicitlyTruefor accelerated networking.
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
return sriov_enabled
d83b8e2 to
f89b1d4
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:994
- Wrapping the entire
switch_sriov()in@retry(HttpResponseError, ...)can skip the_check_sriov_enabled()wait when anHttpResponseErrorhappens after at least one NIC was updated. On the retry attempt,status_changedcan remainFalse(because the NICs now report the desired setting), so the method returns without waiting for SRIOV to become effective in-guest, which can reintroduce flakiness.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
def switch_sriov(
self, enable: bool, wait: bool = True, reset_connections: bool = True
) -> None:
lisa/sut_orchestrator/azure/features.py:1051
sriov_enabled &= ...uses bitwise-and assignment with a conditional expression, which is hard to read and doesn’t short-circuit. This can be simplified to an explicit boolean check (is True) and an early return for the first NIC that is not fully enabled (includingNone).
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
return sriov_enabled
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:991
- Minor: This introduces retry timing constants (tries=3, delay=30) that directly affect test runtime/flakiness, but the rationale for the 30s delay isn’t documented. Adding a brief inline note will make it easier to tune later and understand why retries are so slow.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
lisa/sut_orchestrator/azure/features.py:1052
- Minor: Using
&=to accumulate a boolean result is harder to read than an early-return check, and it also prevents short-circuiting (it will still GET every NIC even after one is already known to be disabled). A simple early-return makes the intent clearer and avoids unnecessary Azure API calls.
# check if all nics have accelnet enabled
sriov_enabled &= (
accelnet_setting if isinstance(accelnet_setting, bool) else False
)
is_enabled_sriov only inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. Check every nic on the VM instead. switch_sriov also failed outright on transient HttpResponseError replies from the network resource provider, so retry it a few times before giving up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5d5f58ad-b9df-4420-ad37-22caee78e925
f414931 to
1527bc9
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1052
is_enabled_sriov()currently accumulates NIC state withsriov_enabled &= (...), which is harder to read than a simple early-return and also forces an Azure NIC GET for every NIC even after one NIC is already known to be disabled. Consider short-circuiting on the first NIC that isn't explicitlyTrueforenable_accelerated_networking.
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= (
accelnet_setting if isinstance(accelnet_setting, bool) else False
)
Part 6 of 9 of a stacked series that reworks the DPDK SRIOV hot plug tests. Stacked on #4658, review only the last commit.
is_enabled_sriovonly inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. It now checks every nic on the VM.switch_sriovfailed outright on transientHttpResponseErrorreplies from the network resource provider, so it is retried a few times before giving up.Key Test Cases:
verify_dpdk_sriov_rescind_failover_send_only|verify_sriov_disable_enable|verify_sriov_add_max_nics
Impacted LISA Features:
Sriov, NetworkInterface
Tested Azure Marketplace Images:
canonical 0001-com-ubuntu-server-jammy 22_04-lts latestredhat rhel 9_5 latest