Skip to content

Azure: check SRIOV on all nics and retry switch_sriov - #4659

Open
mcgov (mcgov) wants to merge 3 commits into
mcgov/stack-5-wget-skip-existsfrom
mcgov/stack-6-azure-sriov
Open

Azure: check SRIOV on all nics and retry switch_sriov#4659
mcgov (mcgov) wants to merge 3 commits into
mcgov/stack-5-wget-skip-existsfrom
mcgov/stack-6-azure-sriov

Conversation

@mcgov

Copy link
Copy Markdown
Collaborator

Part 6 of 9 of a stacked series that reworks the DPDK SRIOV hot plug tests. Stacked on #4658, review only the last commit.

  • is_enabled_sriov only inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. It now checks every nic on the VM.
  • switch_sriov failed outright on transient HttpResponseError replies from the network resource provider, so it is retried a few times before giving up.

Key Test Cases:
verify_dpdk_sriov_rescind_failover_send_only|verify_sriov_disable_enable|verify_sriov_add_max_nics

Impacted LISA Features:
Sriov, NetworkInterface

Tested Azure Marketplace Images:

  • canonical 0001-com-ubuntu-server-jammy 22_04-lts latest
  • redhat rhel 9_5 latest

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Azure SR-IOV handling in LISA’s Azure feature layer to make SR-IOV enablement checks accurate for multi-NIC VMs and to improve resiliency of SR-IOV toggling against transient Azure Network RP/ARM errors.

Changes:

  • Add retry handling to switch_sriov to tolerate transient HttpResponseError responses.
  • Update is_enabled_sriov to validate accelerated networking state across all VM NICs (not just the primary).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lisa/sut_orchestrator/azure/features.py
Comment thread lisa/sut_orchestrator/azure/features.py Outdated
Copilot AI review requested due to automatic review settings August 17, 2026 23:09
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 1418d99 to 57b3d7c Compare August 17, 2026 23:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:1048

  • nic_info.enable_accelerated_networking can be None (it’s treated as optional elsewhere in this file), so using &= can raise TypeError (bool & NoneType) and also reads like a bitwise operation. Use a boolean and check (or is True) so the method always returns a real bool and treats None as disabled.
            sriov_enabled &= nic_info.enable_accelerated_networking

lisa/sut_orchestrator/azure/features.py:991

  • The retry decorator uses a fixed 30s delay with no backoff/jitter. This can significantly extend test runtime (up to ~60s extra plus Azure calls) on transient failures, and it’s inconsistent with other HttpResponseError retries in this module that use short delays with backoff. Consider adding a brief rationale for the 30s delay and/or using a backoff strategy.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

Copilot AI review requested due to automatic review settings August 17, 2026 23:34
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 57b3d7c to c49b9aa Compare August 17, 2026 23:34
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from c49b9aa to 6ba6238 Compare August 17, 2026 23:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1048

  • nic_info.enable_accelerated_networking from the Azure SDK can be None (Optional[bool]). Using sriov_enabled &= nic_info.enable_accelerated_networking can raise a TypeError when the value is None, and &= is a bitwise op that’s less clear than logical AND for booleans. Coerce the SDK value to a boolean (and treat None as disabled) while keeping the result strictly bool.
            sriov_enabled &= nic_info.enable_accelerated_networking

Copilot AI review requested due to automatic review settings August 17, 2026 23:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:1048

  • Major: nic_info.enable_accelerated_networking can be None (elsewhere this field is compared with is False), and sriov_enabled &= None will raise TypeError. Coerce the value to a strict bool (treat None as disabled) before combining results.
                self._resource_group_name, nic_name
            )
            sriov_enabled &= nic_info.enable_accelerated_networking

lisa/sut_orchestrator/azure/features.py:991

  • Minor: @retry(HttpResponseError, tries=3, delay=30) retries all HttpResponseError cases with a fixed 30s delay, which can make permanent failures slower and is inconsistent with the shorter/backoff-based HttpResponseError retries elsewhere in this file. Consider either (a) documenting why 30s is needed here, or (b) aligning with the backoff pattern used in other Azure RP retries / narrowing the retried conditions to transient cases.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

Copilot AI review requested due to automatic review settings August 18, 2026 05:57
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 6ba6238 to 786f228 Compare August 18, 2026 05:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1049

  • sriov_enabled &= nic_info.enable_accelerated_networking relies on enable_accelerated_networking being a strict bool. Elsewhere in this file the code checks x.enable_accelerated_networking is False (around line ~809), which suggests this field may also be None. In that case, &= will raise TypeError: unsupported operand type(s) for &= 'bool' and 'NoneType'. Consider using logical and and coercing None to False.
                self._resource_group_name, nic_name
            )
            sriov_enabled &= nic_info.enable_accelerated_networking
        return sriov_enabled

Copilot AI review requested due to automatic review settings August 18, 2026 06:09
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 786f228 to 802beb1 Compare August 18, 2026 06:09
Copilot AI review requested due to automatic review settings August 18, 2026 07:37
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 0e45b16 to c0e79ea Compare August 18, 2026 07:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1051

  • The boolean accumulation logic is hard to read and non-idiomatic in Python (&= on a bool plus a conditional expression), and it also prevents early-exit once a NIC is found with accelerated networking disabled. Consider returning early when any NIC is not enabled (this also naturally treats None as disabled).
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
        return sriov_enabled

Copilot AI review requested due to automatic review settings August 18, 2026 10:01
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from c0e79ea to 7ae2d00 Compare August 18, 2026 10:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:991

  • Minor: The new retry uses hard-coded values (tries=3, delay=30) without any rationale. Since this affects how long SR-IOV enable/disable can stall on transient Azure NRP errors, please add a brief inline explanation so future changes don’t accidentally shorten/extend this wait.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

lisa/sut_orchestrator/azure/features.py:1050

  • Minor: is_enabled_sriov uses a &= reduction with an inline conditional, which is hard to read and doesn’t short-circuit once a NIC is found without accelerated networking. This can be simplified to an early return and will avoid extra ARM GET calls after the first failure.
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False

Copilot AI review requested due to automatic review settings August 18, 2026 10:28
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 7ae2d00 to d83b8e2 Compare August 18, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:991

  • The new retry on switch_sriov introduces magic-number retry parameters (tries=3, delay=30) without documenting why those values are appropriate. Please add a short comment describing the transient failure being handled and the intended total retry window, so future readers can tune it safely.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

lisa/sut_orchestrator/azure/features.py:1051

  • sriov_enabled &= ... is hard to read and doesn’t short-circuit; it also makes it less obvious that only True should be treated as enabled (everything else should be treated as disabled). Consider returning early when any NIC is not explicitly True for accelerated networking.
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
        return sriov_enabled

Copilot AI review requested due to automatic review settings August 18, 2026 10:38
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from d83b8e2 to f89b1d4 Compare August 18, 2026 10:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:994

  • Wrapping the entire switch_sriov() in @retry(HttpResponseError, ...) can skip the _check_sriov_enabled() wait when an HttpResponseError happens after at least one NIC was updated. On the retry attempt, status_changed can remain False (because the NICs now report the desired setting), so the method returns without waiting for SRIOV to become effective in-guest, which can reintroduce flakiness.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore
    def switch_sriov(
        self, enable: bool, wait: bool = True, reset_connections: bool = True
    ) -> None:

lisa/sut_orchestrator/azure/features.py:1051

  • sriov_enabled &= ... uses bitwise-and assignment with a conditional expression, which is hard to read and doesn’t short-circuit. This can be simplified to an explicit boolean check (is True) and an early return for the first NIC that is not fully enabled (including None).
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= accelnet_setting if isinstance(accelnet_setting, bool) else False
        return sriov_enabled

Copilot AI review requested due to automatic review settings August 18, 2026 10:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:991

  • Minor: This introduces retry timing constants (tries=3, delay=30) that directly affect test runtime/flakiness, but the rationale for the 30s delay isn’t documented. Adding a brief inline note will make it easier to tune later and understand why retries are so slow.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

lisa/sut_orchestrator/azure/features.py:1052

  • Minor: Using &= to accumulate a boolean result is harder to read than an early-return check, and it also prevents short-circuiting (it will still GET every NIC even after one is already known to be disabled). A simple early-return makes the intent clearer and avoids unnecessary Azure API calls.
            # check if all nics have accelnet enabled
            sriov_enabled &= (
                accelnet_setting if isinstance(accelnet_setting, bool) else False
            )

mcgov (mcgov) and others added 3 commits August 19, 2026 11:47
is_enabled_sriov only inspected the primary nic, so a multi nic VM
reported accelerated networking as enabled while a secondary nic was
still being updated. Check every nic on the VM instead.

switch_sriov also failed outright on transient HttpResponseError
replies from the network resource provider, so retry it a few times
before giving up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d5f58ad-b9df-4420-ad37-22caee78e925
Copilot AI review requested due to automatic review settings August 19, 2026 18:47
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from f414931 to 1527bc9 Compare August 19, 2026 18:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1052

  • is_enabled_sriov() currently accumulates NIC state with sriov_enabled &= (...), which is harder to read than a simple early-return and also forces an Azure NIC GET for every NIC even after one NIC is already known to be disabled. Consider short-circuiting on the first NIC that isn't explicitly True for enable_accelerated_networking.
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= (
                accelnet_setting if isinstance(accelnet_setting, bool) else False
            )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants