Skip to content

Perftune ci cd#3323

Draft
vladzcloudius wants to merge 6 commits into
scylladb:masterfrom
vladzcloudius:perftune-ci-cd
Draft

Perftune ci cd#3323
vladzcloudius wants to merge 6 commits into
scylladb:masterfrom
vladzcloudius:perftune-ci-cd

Conversation

@vladzcloudius

Copy link
Copy Markdown
Contributor

This pull request introduces a comprehensive unit testing framework for perftune.py, improves its testability, and adds a new GitHub Actions workflow for automated testing. The main changes include adding dataclass-based parsing for ethtool output, improving the handling of network queue counts (especially for Mellanox devices), and restructuring the script to support direct import and testing. Additionally, the PR sets up the necessary dependencies and configuration for robust test coverage.

Testing infrastructure and configuration:

  • Added a GitHub Actions workflow (.github/workflows/perftune-tests.yaml) to automatically run unit tests on changes to perftune.py and its tests, ensuring continuous integration for performance tuning scripts.
  • Introduced a pyproject.toml configuration for pytest and coverage, and added a requirements.txt file specifying all required dependencies for running tests. [1] [2]
  • Added a detailed README.md in scripts/tests/perftune/ with setup, usage instructions, and test layout documentation.

Test suite and support:

  • Created a conftest.py to safely import perftune.py as a module for testing, avoiding accidental script execution.
  • Added a comprehensive unit test file for the ClocksourceManager class (test_clocksource.py), covering various scenarios and platform behaviors.

Codebase improvements for testability and maintainability:

  • Refactored perftune.py to use a main() function and guarded script entry with if __name__ == '__main__':, enabling import without side effects. [1] [2]
  • Introduced dataclass-based parsing for ethtool -l output, including new LabeledDataclass, EthtoolChannelPropertiesValues, and EthtoolLChannelInfo classes, to make parsing more robust and testable. [1] [2]
  • Improved handling of network queue counts and RPS (Receive Packet Steering) setup, especially for Mellanox mlx5 devices, to ensure correct queue detection and assignment. [1] [2] [3]

These changes collectively provide a solid foundation for reliable, automated testing of perftune.py and facilitate future maintenance and enhancements.

…thtool -l' command for a given interface

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
This method returns rps_cpus file names - not prints them.
And also added a missing coma.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
…g for mlx5 devices

Mellanox mlx5e devices had a rather counterintuitive feature in the kernel/driver versions
5.3-6.0 where they were allocating extra netdev Rx queues for XSK purposes effectively
doubling the number of Rx queues. In particular, if one was setting the number of combined
queues using `ethtool -L <iface> N` command, the driver would allocate Rx queues as follows:
   * RSS queues: 0..N-1
   * XSK queues: N..2*N-1

This was breaking the perftune's assumption that the number of Rx queues under
/sys/class/net/<iface>/queues/ matches the number of corresponding RSS IRQ lines.

Luckily, this was fixed my mlx5 developers in kernels 6.1 and up however, we still
need to support the affected kernels: kernel 5.15 is an LTS kernel of Ubuntu 22.04
for example.

This patch adds a custom handling of the case in question by reverse-engineering the
driver's logic.

Fixes scylladb#3313

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
@vladzcloudius vladzcloudius force-pushed the perftune-ci-cd branch 4 times, most recently from 6553018 to a12ba92 Compare March 27, 2026 04:13
99% code coverage, fully mocked.
See tests/perftune/README.md for more details on how to execute the tests.

Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Signed-off-by: Vlad Zolotarov <vladz@scylladb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant