Skip to content

Commit 7f4d3f5

Browse files
armruNiccoloFeimnencia
committed
test(e2e): replace MinIO with RustFS in the object storage test harness (cloudnative-pg#10865)
The MinIO and mc images used by the e2e suite are pinned to the last Apache-2.0 releases and no longer receive updates. This PR replaces the whole stack: the in-cluster object storage server becomes RustFS, and the client pod used to inspect the store becomes the AWS CLI. The server keeps the existing TLS setup, with the certificates generated by the test suite projected under the names RustFS expects. Since the RustFS image runs as a non-root user, the pod sets an fsGroup so the data volume and the certificates stay accessible. The query helpers keep their signatures: listing and counting enumerate every bucket through the S3 API and filter the results with a regular expression derived from the previous glob semantics, tags are read with get-object-tagging, and the forged-WAL helper uses a server-side copy. A failure enumerating the buckets aborts the listing, so an unreachable endpoint surfaces as an error instead of an empty result, and the per-bucket listings run in parallel so the cost of a poll does not grow linearly with the number of buckets accumulated during a run. Checksum calculation is pinned to when_required because the CRC-based defaults introduced in AWS CLI 2.23 are not supported by every S3-compatible object store. With MinIO gone, the harness is also renamed to provider-neutral terms: the minio Go packages become objectstore, the in-cluster resources move to the object-store namespace and service with matching secret names, the fixtures and their file names are renamed accordingly, the timeout keys become walsInObjectStore and objectStoreInstallation, and the spec texts stop mentioning MinIO. Note that the spec renames are a one-time discontinuity for anything tracking test names across runs, and any out-of-tree TEST_TIMEOUTS override needs the new keys. The user-facing documentation, where MinIO is presented as one of the supported object stores, is intentionally unchanged. Verified on kind by running the backup and restore, backup tags, and parallel WAL restore specs against the new harness: barman-cloud bucket auto-creation, TLS, object tagging, and server-side copy all work against RustFS with both the old and the new client. Closes cloudnative-pg#6572 Closes cloudnative-pg#8967 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> (cherry picked from commit e7419df)
1 parent b90bd79 commit 7f4d3f5

70 files changed

Lines changed: 1445 additions & 1596 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/e2e_test_timeout.json‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
"newTargetOnFailover": 120,
1111
"operatorIsReady": 120,
1212
"largeObject": 300,
13-
"walsInMinio": 60,
14-
"minioInstallation": 300,
13+
"walsInObjectStore": 60,
14+
"objectStoreInstallation": 300,
1515
"backupIsReady": 180,
1616
"drainNode": 900,
1717
"short": 5
@@ -27,8 +27,8 @@
2727
"newTargetOnFailover": 120,
2828
"operatorIsReady": 120,
2929
"largeObject": 300,
30-
"walsInMinio": 60,
31-
"minioInstallation": 300,
30+
"walsInObjectStore": 60,
31+
"objectStoreInstallation": 300,
3232
"backupIsReady": 180,
3333
"drainNode": 900,
3434
"short": 5
@@ -44,8 +44,8 @@
4444
"newTargetOnFailover": 120,
4545
"operatorIsReady": 120,
4646
"largeObject": 300,
47-
"walsInMinio": 60,
48-
"minioInstallation": 300,
47+
"walsInObjectStore": 60,
48+
"objectStoreInstallation": 300,
4949
"backupIsReady": 180,
5050
"drainNode": 900,
5151
"short": 10
@@ -61,8 +61,8 @@
6161
"newTargetOnFailover": 120,
6262
"operatorIsReady": 120,
6363
"largeObject": 300,
64-
"walsInMinio": 60,
65-
"minioInstallation": 300,
64+
"walsInObjectStore": 60,
65+
"objectStoreInstallation": 300,
6666
"backupIsReady": 180,
6767
"drainNode": 900,
6868
"short": 10
@@ -78,8 +78,8 @@
7878
"newTargetOnFailover": 120,
7979
"operatorIsReady": 120,
8080
"largeObject": 300,
81-
"walsInMinio": 60,
82-
"minioInstallation": 300,
81+
"walsInObjectStore": 60,
82+
"objectStoreInstallation": 300,
8383
"backupIsReady": 180,
8484
"drainNode": 900,
8585
"short": 10
@@ -95,8 +95,8 @@
9595
"newTargetOnFailover": 120,
9696
"operatorIsReady": 120,
9797
"largeObject": 300,
98-
"walsInMinio": 60,
99-
"minioInstallation": 300,
98+
"walsInObjectStore": 60,
99+
"objectStoreInstallation": 300,
100100
"backupIsReady": 180,
101101
"drainNode": 900,
102102
"short": 10

‎.github/renovate.json5‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -98,14 +98,14 @@
9898
{
9999
customType: 'regex',
100100
managerFilePatterns: [
101-
'/^tests\\/utils\\/minio\\/minio\\.go$/',
101+
'/^tests\\/utils\\/objectstore\\/objectstore\\.go$/',
102102
],
103103
matchStrings: [
104-
'minioImage = "(?<depName>.+?):(?<currentValue>.*?)"',
105-
'minioClientImage = "(?<depName>.+?):(?<currentValue>.*?)"',
104+
'rustfsImage = "(?<depName>.+?):(?<currentValue>.*?)"',
105+
'awsCliImage = "(?<depName>.+?):(?<currentValue>.*?)"',
106106
],
107107
datasourceTemplate: 'docker',
108-
versioningTemplate: 'regex:^RELEASE\\.(?<major>\\d{4})-(?<minor>\\d{2})-(?<patch>\\d{2})T\\d{2}-\\d{2}-\\d{2}Z$',
108+
versioningTemplate: 'semver',
109109
},
110110
{
111111
customType: 'regex',
@@ -227,7 +227,8 @@
227227
pinDigests: false,
228228
matchPackageNames: [
229229
'vmware-tanzu{/,}**',
230-
'minio{/,}**',
230+
'rustfs{/,}**',
231+
'amazon{/,}**',
231232
],
232233
},
233234
{

‎hack/testing-tools/common/10-config.sh‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,11 @@ export E2E_PRE_ROLLING_UPDATE_IMG=${E2E_PRE_ROLLING_UPDATE_IMG:-${POSTGRES_IMG%.
3939
# PGBouncer Image
4040
export PGBOUNCER_IMG=${PGBOUNCER_IMG:-$(grep 'DefaultPgbouncerImage.*=' "${ROOT_DIR}/pkg/versions/versions.go" | awk -F '"' '{print $2}' | tr -d '[:space:]')}
4141

42-
# MINIO Image
43-
export MINIO_IMG=${MINIO_IMG:-$(grep 'minioImage.*=' "${ROOT_DIR}/tests/utils/minio/minio.go" | awk -F '"' '{print $2}' | tr -d '[:space:]')}
42+
# RustFS Image
43+
export RUSTFS_IMG=${RUSTFS_IMG:-$(grep 'rustfsImage.*=' "${ROOT_DIR}/tests/utils/objectstore/objectstore.go" | awk -F '"' '{print $2}' | tr -d '[:space:]')}
44+
45+
# AWS CLI Image (S3 client used by the e2e object storage tests)
46+
export AWSCLI_IMG=${AWSCLI_IMG:-$(grep 'awsCliImage.*=' "${ROOT_DIR}/tests/utils/objectstore/objectstore.go" | awk -F '"' '{print $2}' | tr -d '[:space:]')}
4447

4548
# Apache Image (Hardcoded stable default)
4649
export APACHE_IMG=${APACHE_IMG:-"httpd"}
@@ -61,13 +64,17 @@ if [ -z "${PGBOUNCER_IMG}" ]; then
6164
echo "ERROR: Failed to extract PGBOUNCER_IMG from ${ROOT_DIR}/pkg/versions/versions.go" >&2
6265
exit 1
6366
fi
64-
if [ -z "${MINIO_IMG}" ]; then
65-
echo "ERROR: Failed to extract MINIO_IMG from ${ROOT_DIR}/tests/utils/minio/minio.go" >&2
67+
if [ -z "${RUSTFS_IMG}" ]; then
68+
echo "ERROR: Failed to extract RUSTFS_IMG from ${ROOT_DIR}/tests/utils/objectstore/objectstore.go" >&2
69+
exit 1
70+
fi
71+
if [ -z "${AWSCLI_IMG}" ]; then
72+
echo "ERROR: Failed to extract AWSCLI_IMG from ${ROOT_DIR}/tests/utils/objectstore/objectstore.go" >&2
6673
exit 1
6774
fi
6875

6976
# Define the full array of helper images used by load-helper-images
70-
HELPER_IMGS=("$POSTGRES_IMG" "$E2E_PRE_ROLLING_UPDATE_IMG" "$PGBOUNCER_IMG" "$MINIO_IMG" "$APACHE_IMG")
77+
HELPER_IMGS=("$POSTGRES_IMG" "$E2E_PRE_ROLLING_UPDATE_IMG" "$PGBOUNCER_IMG" "$RUSTFS_IMG" "$AWSCLI_IMG" "$APACHE_IMG")
7178
export HELPER_IMGS
7279

7380
# Testing the upgrade will require generating a second operator image, `-prime`

‎hack/testing-tools/k8s-engines/manage.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -222,7 +222,8 @@ case "$ACTION" in
222222
echo "CONTROLLER_IMG (default): $(print_image)"
223223
echo "POSTGRES_IMG: ${POSTGRES_IMG}"
224224
echo "E2E_PRE_ROLLING_UPDATE_IMG: ${E2E_PRE_ROLLING_UPDATE_IMG}"
225-
echo "MINIO_IMG: ${MINIO_IMG}"
225+
echo "RUSTFS_IMG: ${RUSTFS_IMG}"
226+
echo "AWSCLI_IMG: ${AWSCLI_IMG}"
226227
echo "HELPER_IMGS count: ${#HELPER_IMGS[@]}"
227228
echo "TEST_UPGRADE_TO_V1: ${TEST_UPGRADE_TO_V1}"
228229

‎tests/e2e/backup_restore_azure_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import (
2626
"github.com/cloudnative-pg/cloudnative-pg/tests"
2727
backupasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/backup"
2828
clusterasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/cluster"
29-
minioasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/minio"
29+
objectstoreasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/objectstore"
3030
pgasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/postgres"
3131
"github.com/cloudnative-pg/cloudnative-pg/tests/internal/resources"
3232
"github.com/cloudnative-pg/cloudnative-pg/tests/utils/backups"
@@ -468,7 +468,7 @@ func assertArchiveWalOnAzureBlob(namespace, clusterName string, configuration ba
468468
By("archiving WALs and verifying they exist", func() {
469469
primary, err := clusterutils.GetPrimary(env.Ctx, env.Client, namespace, clusterName)
470470
Expect(err).ToNot(HaveOccurred())
471-
latestWAL := minioasserts.SwitchWalAndGetLatestArchive(env, primary.Namespace, primary.Name)
471+
latestWAL := objectstoreasserts.SwitchWalAndGetLatestArchive(env, primary.Namespace, primary.Name)
472472
// Define what file we are looking for in Azure.
473473
// Escapes are required since az expects forward slashes to be escaped
474474
path := fmt.Sprintf("wals\\/0000000100000000\\/%v.gz", latestWAL)

‎tests/e2e/backup_restore_azurite_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import (
2626
"github.com/cloudnative-pg/cloudnative-pg/tests"
2727
backupasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/backup"
2828
clusterasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/cluster"
29-
minioasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/minio"
29+
objectstoreasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/objectstore"
3030
pgasserts "github.com/cloudnative-pg/cloudnative-pg/tests/internal/asserts/postgres"
3131
"github.com/cloudnative-pg/cloudnative-pg/tests/utils/backups"
3232
"github.com/cloudnative-pg/cloudnative-pg/tests/utils/clusterutils"
@@ -397,7 +397,7 @@ func assertArchiveWalOnAzurite(namespace, clusterName string) {
397397
// Create a WAL on the primary and check if it arrives at the Azure Blob Storage within a short time
398398
By("archiving WALs and verifying they exist", func() {
399399
primary := clusterName + "-1"
400-
latestWAL := minioasserts.SwitchWalAndGetLatestArchive(env, namespace, primary)
400+
latestWAL := objectstoreasserts.SwitchWalAndGetLatestArchive(env, namespace, primary)
401401
// verifying on blob storage using az
402402
// Define what file we are looking for in Azurite.
403403
// Escapes are required since az expects forward slashes to be escaped

0 commit comments

Comments
 (0)