Skip to content

[release-23.0] ci: extract os tuning (#18824) #18827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: release-23.0
Choose a base branch
from
+337 −1,211
Open

[release-23.0] ci: extract os tuning (#18824) #18827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/actions/tune-os/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: "Tune OS"
description: "Tune the OS for better performance during CI runs"
runs:
using: "composite"
steps:
- name: Tune the OS
shell: bash
run: |
# Install eatmydata and ensure it's used for every operation
sudo apt-get update && sudo apt-get install -y eatmydata
echo "/usr/\${LIB}/libeatmydata.so" | sudo tee -a /etc/ld.so.preload
# Increase the range of ephemeral ports.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
13 changes: 4 additions & 9 deletions .github/workflows/check_make_vtadmin_authz_testgen.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,21 +43,16 @@ jobs:
- 'go/vt/vtadmin/**'
- '.github/workflows/check_make_vtadmin_authz_testgen.yml'

- name: Tune the OS
if: steps.changes.outputs.vtadmin_changes == 'true'
uses: ./.github/actions/tune-os

- name: Set up Go
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5.5.0
if: steps.changes.outputs.vtadmin_changes == 'true'
with:
go-version-file: go.mod

- name: Tune the OS
if: steps.changes.outputs.vtadmin_changes == 'true'
run: |
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db

- name: Get dependencies
if: steps.changes.outputs.vtadmin_changes == 'true'
run: |
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/check_make_vtadmin_web_proto.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ jobs:
with:
persist-credentials: 'false'

- name: Tune the OS
uses: ./.github/actions/tune-os

- name: Check for changes in relevant files
uses: dorny/paths-filter@ebc4d7e9ebcb0b1eb21480bb8f43113e996ac77a # v3.0.1
id: changes
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_12.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -145,7 +134,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 12 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard 12 | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_13.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -145,7 +134,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 13 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard 13 | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_15.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -145,7 +134,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 15 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard 15 | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_18.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -150,7 +139,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 18 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard 18 | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_21.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -151,7 +140,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 21 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard 21 | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_backup_pitr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -145,7 +134,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard backup_pitr | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard backup_pitr | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_backup_pitr_mysqlshell.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -108,7 +97,7 @@ jobs:
sudo dpkg -i libtinfo5_6.3-2ubuntu0.1_amd64.deb

# Install everything else we need, and configure
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y mysql-server mysql-shell mysql-client make unzip g++ etcd-client etcd-server curl git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -145,7 +134,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard backup_pitr_mysqlshell | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard backup_pitr_mysqlshell | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/cluster_endtoend_backup_pitr_xtrabackup.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,7 @@ jobs:

- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"

# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf

# Don't waste a bunch of time processing man-db triggers
echo "set man-db/auto-update false" | sudo debconf-communicate
sudo dpkg-reconfigure man-db
uses: ./.github/actions/tune-os

- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
Expand All @@ -100,7 +89,7 @@ jobs:
sudo apt-get -qq update

# Install everything else we need, and configure
sudo apt-get -qq install -y percona-server-server percona-server-client make unzip g++ etcd-client etcd-server git wget eatmydata xz-utils libncurses6
sudo apt-get -qq install -y percona-server-server percona-server-client make unzip g++ etcd-client etcd-server git wget xz-utils libncurses6

sudo service mysql stop
sudo service etcd stop
Expand Down Expand Up @@ -139,7 +128,7 @@ jobs:
# Some of these tests require specific locales to be installed.
# See https://github.com/cncf/automation/commit/49f2ad7a791a62ff7d038002bbb2b1f074eed5d5
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard backup_pitr_xtrabackup | tee -a output.txt | go-junit-report -set-exit-code > report.xml
go run test.go -docker=false -follow -shard backup_pitr_xtrabackup | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Record test results in launchable if PR is not a draft
if: github.event_name == 'pull_request' && github.event.pull_request.draft == 'false' && steps.changes.outputs.end_to_end == 'true' && github.base_ref == 'main' && !cancelled()
Expand Down
Loading
Loading