Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mariadb upgrade scenario testing #594

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions tests/test_mariadb.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
"""Tests for the MariaDB related application container images."""

from itertools import product
from pathlib import Path
from typing import List
from typing import Optional

import pymysql
import pytest
from _pytest.mark import ParameterSet
from pymysql.err import OperationalError
from pytest_container.container import BindMount
from pytest_container.container import ContainerData
from pytest_container.container import ContainerLauncher
from pytest_container.container import DerivedContainer
from pytest_container.container import container_and_marks_from_pytest_param
from pytest_container.pod import Pod
Expand All @@ -21,6 +24,7 @@
from bci_tester.data import MARIADB_CLIENT_CONTAINERS
from bci_tester.data import MARIADB_CONTAINERS
from bci_tester.data import MARIADB_ROOT_PASSWORD
from bci_tester.data import OS_VERSION

CONTAINER_IMAGES = MARIADB_CONTAINERS

Expand Down Expand Up @@ -276,3 +280,76 @@ def test_mariadb_healthcheck_galera_cluster_disabled(auto_container_per_test):
_wait_for_server(conn)

conn.run_expect([1], "healthcheck.sh --su-mysql --galera_online")


_DB_ENV = {
"MARIADB_USER": _OTHER_DB_USER,
"MARIADB_PASSWORD": _OTHER_DB_PW,
"MARIADB_DATABASE": _TEST_DB,
"MARIADB_ROOT_PASSWORD": MARIADB_ROOT_PASSWORD,
"MARIADB_AUTO_UPGRADE": "1",
}


@pytest.mark.parametrize("ctr_image", MARIADB_CONTAINERS)
@pytest.mark.skipif(
OS_VERSION in ("15.5",), reason="MariaDB upgrade scenario not supported"
)
def test_mariadb_upgrade(
container_runtime: OciRuntimeBase,
pytestconfig: pytest.Config,
ctr_image: DerivedContainer,
tmp_path: Path,
) -> None:
mounts = [BindMount(host_path=tmp_path, container_path="/var/lib/mysql")]
mariadb_old = DerivedContainer(
base="registry.suse.com/suse/mariadb:10.6",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no guarantee that you will be able to migrate from 10.6 to newer MariaDB containers.

I believe the test should include a mapping from which version to which newer version, otherwise this test could fail on a new container image, not because it is broken, but because MariaDB does not support migrating 10.6 to version X.Y.

So instead of MARIADB_CONTAINERS it should be passed something like MARIADB_CONTAINERS_UPGRADE_VERSIONS, which would be a tuple, or another map style type.

containerfile='RUN set -euo pipefail; head -n -1 /usr/local/bin/gosu > /tmp/gosu; echo \'exec setpriv --pdeathsig=keep --reuid="$u" --regid="$u" --clear-groups -- "$@"\' >> /tmp/gosu; mv /tmp/gosu /usr/local/bin/gosu; chmod +x /usr/local/bin/gosu',
volume_mounts=mounts,
extra_environment_variables=_DB_ENV,
)
mariadb_new = DerivedContainer(
base=ctr_image,
volume_mounts=mounts,
extra_environment_variables=_DB_ENV,
)
with ContainerLauncher.from_pytestconfig(
mariadb_old, container_runtime, pytestconfig
) as launcher:
launcher.launch_container()
con = launcher.container_data.connection
_wait_for_server(con)
mariadb_cmd = f"mariadb --user={_OTHER_DB_USER} --password={_OTHER_DB_PW} --host=0.0.0.0 {_TEST_DB}"

con.check_output(
f'echo "CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar(32));" | {mariadb_cmd}'
)
con.check_output(
f"echo 'INSERT INTO test (num, data) VALUES (100, \"abcdef\")' | {mariadb_cmd}"
Copy link
Member

@alexandrevicenzi alexandrevicenzi Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs the upgrade touches these:

  • Updating the system tables in the mysql database to the newest version. This is very quick.
  • mariadb-upgrade also runs mariadb-check --check-upgrade to check if there have been any collation changes between the major versions. This recreates indexes in old tables that are using any of the changed collations. This can take a bit of time if there are a lot of tables or there are many tables which used the changed collation. The last time a collation changed was in MariaDB/MySQL 5.1.23.

It would be nice to touch system tables, such as changing the database charset to a non-standard one.

It would be nice to have a data file that has multiple commands and tables. Ideally creating indexes, so we know that MariaDB can migrate indexes, plus perhaps including more data types than just integer and varchar.

Regular tables are not very likely to break or change unless there's a change in the data type of a column I believe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any existing SQL file that has a good load of data(we might have used in earlier experiments) or would you like to recommend any specific dataset for this or we can create our own random data(with indexes for sure)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have any, maybe the maintainer has one. Otherwise, we could grab some dataset from Kaggle if needed.

There's no need to be a big dump, just having a few tables, indexes, and changing things that touch system tables would be sufficient I believe.

)
rows = (
con.check_output(f'echo "SELECT * FROM test;" | {mariadb_cmd}')
.strip()
.splitlines()
)

assert rows and len(rows) == 2
_, num, data = rows[-1].split()
assert num == "100" and data == "abcdef"

with ContainerLauncher.from_pytestconfig(
mariadb_new, container_runtime, pytestconfig
) as launcher:
launcher.launch_container()
con = launcher.container_data.connection
_wait_for_server(con)
mariadb_cmd = f"mariadb --user={_OTHER_DB_USER} --password={_OTHER_DB_PW} --host=0.0.0.0 {_TEST_DB}"
rows = (
con.check_output(f'echo "SELECT * FROM test;" | {mariadb_cmd}')
.strip()
.splitlines()
)

assert rows and len(rows) == 2
_, num, data = rows[-1].split()
assert num == "100" and data == "abcdef"
Loading