Dpdk: roll back broken installs and reuse downloaded assets

[mcgov (mcgov)]

Part 7 of 9 of a stacked series that reworks the DPDK SRIOV hot plug tests. Stacked on #4659, review only the last commit.

• A failure part way through a source installation left the node with a half installed dpdk or rdma-core, and every later test on that node failed for an unrelated reason. The install steps are now wrapped so a failure uninstalls what was applied, removes the extracted source, marks the node dirty if even the cleanup fails, and re-raises the original error.
• The asset removal guards that were specific to the rdma-core installer now live on the base Installer as _delete_assets, so every installer is protected against deleting / or the working path.
• Downloads and extraction are skipped when the asset is already on the node, and dpdk-stable is fetched from the github mirror, which is far more reliable than dpdk.org from Azure.

Key Test Cases:
verify_dpdk_build_netvsc|verify_dpdk_build_failsafe|verify_dpdk_build_gb_hugepages_netvsc

Impacted LISA Features:
Sriov, NetworkInterface, Infiniband

Tested Azure Marketplace Images:

• canonical 0001-com-ubuntu-server-jammy 22_04-lts latest
• microsoftcblmariner azure-linux-3 azure-linux-3 latest
• redhat rhel 9_5 latest

[Copilot]

Pull request overview

This PR improves resiliency and reusability of the DPDK/rdma-core source installation flow used by the DPDK SRIOV hot-plug tests, aiming to prevent nodes from being left in a broken “half-installed” state and to reduce redundant downloads/extractions.

Changes:

• Bump rdma-core default source tarball to v59.0 and centralize asset-delete safety checks into the base Installer.
• Add rollback logic in the base Installer to attempt cleanup after installation failures and mark nodes dirty when cleanup fails.
• Reuse already-present downloaded/extracted assets and switch dpdk-stable fetch to the GitHub mirror for reliability.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
lisa/microsoft/testsuites/dpdk/rdmacore.py	Updates rdma-core source URL and removes per-installer asset cleanup in favor of base installer handling.
lisa/microsoft/testsuites/dpdk/common.py	Adds shared asset deletion guards + rollback flow, skips redundant download/extract work, and updates the dpdk-stable repo URL.

Suppressed comments (1)

lisa/microsoft/testsuites/dpdk/common.py:283

• Major: do_installation() uses raise e, which resets the original traceback to this handler. Use a bare raise after rollback so failures point to the actual install step that failed.

            except Exception as e:
                self._rollback_installation()
                raise e

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

lisa/microsoft/testsuites/dpdk/common.py:248

• _rollback_installation() re-raises the cleanup exception, which can mask the original installation failure. This contradicts the PR description (mark node dirty if cleanup fails, but re-raise the original error). Rollback should best-effort cleanup, mark the node dirty on rollback failures, and not raise.

    def _rollback_installation(self) -> None:
        try:
            if self._check_if_installed():
                self._uninstall()
            self._delete_assets()
        except Exception as err:
            self._node.log.debug(
                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()
            raise err

lisa/microsoft/testsuites/dpdk/common.py:192

• _setup_node() only checks hasattr(self, "asset_path"), which can be true even when the directory was deleted on the node (or never created successfully). That can cause later install/uninstall steps to run with a missing asset_path. Prefer checking both attribute presence and remote path existence via _asset_path_exists().

    def _setup_node(self) -> None:
        if not hasattr(self, "asset_path"):
            self._download_assets()

lisa/microsoft/testsuites/dpdk/common.py:283

• In do_installation(), raise e resets the traceback, which makes the original failure harder to debug. Use a bare raise to preserve the original traceback (especially important here since you’re deliberately catching only to rollback).

            try:
                self._download_assets()
                self._uninstall()
                self._install_dependencies()
                self._install()
            except Exception as e:
                self._rollback_installation()
                raise e

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (4)

lisa/microsoft/testsuites/dpdk/common.py:247

• _rollback_installation currently raise err on cleanup failures, which can override the original install exception from do_installation. Since this method is used as a best-effort rollback, it should mark the node dirty and return without raising (or at least preserve traceback with bare raise).

            self._node.log.debug(
                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()
            raise err

lisa/microsoft/testsuites/dpdk/common.py:236

• _delete_assets executes rm -rf {asset_path} via the shell without quoting. Even with the guard assertions, using node.shell.remove() avoids shell-escaping issues and reduces the chance of path-injection bugs.

            f"Test bug: Installer source path {asset_path} was set to working path "
            f"'{working_path}' during attempted cleanup!"
        ).is_not_equal_to(working_path)
        self._node.execute(f"rm -rf {str(asset_path)}", shell=True)

lisa/microsoft/testsuites/dpdk/common.py:283

• do_installation re-raises with raise e, which drops the original traceback, and a failure inside _rollback_installation() can mask the original install failure (contradicting the PR description that the original error is re-raised). Wrapping _setup_node()/install steps in a single try/except and using bare raise preserves the original exception while still doing best-effort rollback.

            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:177

• Skipping extraction based only on asset_path existence can leave a partially-extracted source tree (e.g., if a prior run failed mid-extract), and subsequent runs will silently reuse the incomplete directory. Consider always running Tar.extract(..., skip_existing_files=True) to ensure missing files are populated.

        if not node.shell.exists(self.asset_path):
            node.tools[Tar].extract(
                file=str(remote_path),
                dest_dir=str(work_path),
                gzip=True,

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lisa/microsoft/testsuites/dpdk/common.py:283

• If an install step fails, _rollback_installation() can raise and mask the original failure, and raise e drops the original traceback. This contradicts the PR description (re-raise original error) and makes debugging harder. Wrap rollback in its own try/except and use bare raise to preserve the original exception/traceback.

            try:
                self._download_assets()
                self._uninstall()
                self._install_dependencies()
                self._install()
            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:247

• _rollback_installation() currently raises its own cleanup exception (raise err), which can override the original installation failure (and also resets the cleanup traceback). Since callers already handle the original error, rollback should mark the node dirty + log and then return without raising.

    def _rollback_installation(self) -> None:
        try:
            if self._check_if_installed():
                self._uninstall()
            self._delete_assets()
        except Exception as err:
            self._node.log.debug(
                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()
            raise err

lisa/microsoft/testsuites/dpdk/common.py:235

• rm -rf is executed via shell=True with an unquoted path. Even with the / and working-path guards, this is still brittle (spaces/shell metacharacters) and is a command-injection footgun if asset_path ever comes from variables. Prefer the built-in shell file API to remove the directory safely.

            f"Test bug: Installer source path {asset_path} was set to working path "
            f"'{working_path}' during attempted cleanup!"
        ).is_not_equal_to(working_path)
        self._node.execute(f"rm -rf {str(asset_path)}", shell=True)

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/dpdk/common.py:242

• _rollback_installation() currently (1) skips _uninstall() if _check_if_installed() is false (which can miss partially-applied installs), and (2) uses raise err, which loses traceback context. Rollback should be best-effort (attempt uninstall even if detection fails) and should re-raise with bare raise so debugging context is preserved. (The caller can decide whether rollback failures should mask the original install error.)

        try:
            if self._check_if_installed():
                self._uninstall()
            self._delete_assets()
        except Exception as err:

lisa/microsoft/testsuites/dpdk/common.py:283

• do_installation re-raises with raise e, which drops the original traceback, and it also lets _rollback_installation() failures replace the original install error. This contradicts the PR description (“re-raises the original error”) and makes failures harder to diagnose. Preserve the original exception with bare raise, and ensure rollback errors don’t mask it (rollback can still mark the node dirty).

            except Exception as e:
                self._rollback_installation()
                raise e

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/dpdk/common.py:284

• raise e resets the traceback to this except block, which hides the original failure location. Use a bare raise to re-raise the original exception after rollback while preserving its traceback.

            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:248

• The rollback path currently emits an empty debug log line and raises AssertionError(err, ...) without exception chaining, which makes failures harder to diagnose (traceback/messaging becomes a tuple of args). Prefer removing the blank log and raising a single-message AssertionError chained from the original exception.

This issue also appears on line 282 of the same file.

            self._node.log.debug("")
            raise AssertionError(err, "Test bug: rollback of installation failed")

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lisa/microsoft/testsuites/dpdk/common.py:284

• Minor: raise e resets the traceback, which makes the original installation failure harder to diagnose. Use a bare raise to preserve the original traceback after rollback.

            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:270

• Major: do_installation() accepts required_version but currently ignores it by calling _should_install() without passing the parameter, so callers cannot enforce a minimum version.

        if self._should_install():

lisa/microsoft/testsuites/dpdk/common.py:249

• Major: _rollback_installation() currently raises a new AssertionError if cleanup fails, which can mask the original install/build exception. This contradicts the PR description that says cleanup failures should mark the node dirty and then re-raise the original error.

This issue also appears on line 282 of the same file.

                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()
            self._node.log.debug("")
            raise AssertionError(err, "Test bug: rollback of installation failed")

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/microsoft/testsuites/dpdk/common.py:272

• do_installation() runs _setup_node() (which can download/extract assets) outside the try/except, so failures during download/extraction won’t be rolled back. Also, re-raising with raise e drops the original traceback. Wrapping the whole flow in the try/except and using bare raise preserves context and ensures rollback runs for setup/download failures too.

    def do_installation(self, required_version: Optional[VersionInfo] = None) -> None:
        self._setup_node()
        if self._should_install():
            # any issues here could result in a broken installation.
            # If the node is still usable, we don't want to discard it.

lisa/microsoft/testsuites/dpdk/common.py:248

• AssertionError(err, "...") sets the assertion message to a tuple of args, which makes failures harder to read and loses standard exception chaining. Prefer a single message and use from err so the cleanup failure retains its traceback.

This issue also appears on line 268 of the same file.

            raise AssertionError(err, "Test bug: rollback of installation failed")

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lisa/microsoft/testsuites/dpdk/common.py:248

• Major: _rollback_installation() raises AssertionError(err, ...), which produces an exception with tuple args and loses proper exception chaining. Also the blank debug("") line adds noise. Prefer raising a clear AssertionError from the original exception so logs/tracebacks are actionable.

        except Exception as err:
            self._node.log.debug(
                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()
            self._node.log.debug("")
            raise AssertionError(err, "Test bug: rollback of installation failed")

lisa/microsoft/testsuites/dpdk/common.py:284

• Critical: do_installation() will call _rollback_installation() even if the failure happens during _download_assets() (before any uninstall/install occurs). In that case rollback may uninstall a previously working installation, leaving the node worse off. Also raise e resets the traceback. Download assets outside the try, and re-raise with bare raise after rollback.

            try:
                self._download_assets()
                self._uninstall()
                self._install_dependencies()
                self._install()
            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:236

• Major: _delete_assets() deletes the asset_path attribute before validating/removing the directory. If an assertion fails or shell.remove() raises, the attribute is already gone, which can prevent later cleanup attempts and makes debugging harder. Delete the attribute only after a successful removal.

This issue also appears in the following locations of the same file:

• line 242
• line 277

        if not self._asset_path_exists():
            return
        asset_path = self.asset_path
        delattr(self, "asset_path")
        working_path = str(self._node.get_working_path())

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lisa/microsoft/testsuites/dpdk/common.py:177

• Skipping tar extraction solely based on the root folder already existing can leave the node stuck with a partially extracted tree if a prior tar run failed part-way (the directory exists, but files are missing). Previously the extract step was idempotent via skip_existing_files=True, which could recover by extracting only the missing files.

        if not node.shell.exists(self.asset_path):
            node.tools[Tar].extract(
                file=str(remote_path),
                dest_dir=str(work_path),
                gzip=True,

lisa/microsoft/testsuites/dpdk/common.py:284

• This except block uses raise e, which discards the original traceback, and _rollback_installation() raising would also mask the original install failure (contrary to the PR goal of re-raising the original error). Use a bare raise to preserve the traceback and ensure rollback failures don’t override the initial exception.

            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:248

• _rollback_installation() currently raises AssertionError(err, ...), which produces a tuple-like message and can override the original installation exception. Since this is best-effort cleanup, mark the node dirty and log the failure, but don’t raise here (let the original install error be the one that propagates).

        except Exception as err:
            self._node.log.debug(
                f"Installer cleanup failed; marking node dirty. {str(err)}"
            )
            self._node.mark_dirty()

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (4)

lisa/microsoft/testsuites/dpdk/common.py:284

• Major: raise e will reset the traceback to this line, making failures harder to debug. Re-raise the original exception with a bare raise after rollback so the original stack trace is preserved.

            except Exception as e:
                self._rollback_installation()
                raise e

lisa/microsoft/testsuites/dpdk/common.py:280

• Major: Assets are already downloaded in _setup_node() when asset_path is missing, but do_installation() downloads them again, which can trigger redundant downloads/clone operations and complicate rollback tracking. Consider relying on _setup_node() and removing this extra _download_assets() call.

            try:
                self._download_assets()
                self._uninstall()
                self._install_dependencies()

lisa/microsoft/testsuites/dpdk/common.py:248

• Major: raise AssertionError(err, ...) produces an AssertionError with confusing args and loses exception chaining. Prefer a single clear message and chain the original exception with from err (and drop the blank debug line).

            self._node.log.debug("")
            raise AssertionError(err, "Test bug: rollback of installation failed")

lisa/microsoft/testsuites/dpdk/common.py:241

• Major: _rollback_installation() only uninstalls when _check_if_installed() returns true. In a partial/failed install, _check_if_installed() may still be false, so rollback can skip uninstalling and leave the node in a broken state (contrary to the PR goal of rolling back broken installs). Calling _uninstall() unconditionally (and letting each installer decide if it can safely uninstall) is more robust.

This issue also appears in the following locations of the same file:

• line 247
• line 277
• line 282

            if self._check_if_installed():
                self._uninstall()
            self._delete_assets()