Sync master with trigger 13-04-26 (#1196)

- Add PostgreSQL 18 and remove PostgreSQL 13: c432557
- Patroni 4.1.0
- WAL-G 3.0.8
- PostGIS 3.6: #1153
- Remove WAL-E and make WAL-G the default backup tool: #1143
- Enable pg_basebackup server compression: #1155
- Update bg_mon, pg_mon, pg_profile, pg_permissions refs
- Use pgdg for set_user and pg_permissions


Bugfixes
- Allow admin selecting from hypopg views: #1162
- Fix logic for keeping timescaledb minor versions: #1173
- Properly setup clean env for ext build across major versions: #1174
- Properly format ipv6 address for rsync: #1187
- Ensure CLONE_HOST is IPv6 compatible: #1190

Author: hughcapet

ENVIRONMENT.rst

@@ -11,11 +11,11 @@ Environment Configuration Settings
 - **ETCD_KEY**: Etcd client certificate key. Can be empty if the key is part of certificate.
 - **PGHOME**: filesystem path where to put PostgreSQL home directory (/home/postgres by default)
 - **APIPORT**: TCP port to Patroni API connections (8008 by default)
-- **BACKUP_SCHEDULE**: cron schedule for doing backups via WAL-E (if WAL-E is enabled, '00 01 * * *' by default)
+- **BACKUP_SCHEDULE**: cron schedule for doing backups via WAL-G ('00 01 * * *' by default)
 - **CLONE_TARGET_TIMELINE**: timeline id of the backup for restore, 'latest' by default.
 - **CRONTAB**: anything that you want to run periodically as a cron job (empty by default)
 - **PGROOT**: a directory where we put the pgdata (by default /home/postgres/pgroot). One may adjust it to point to the mount point of the persistent volume, such as EBS.
-- **WALE_TMPDIR**: directory to store WAL-E temporary files. PGROOT/../tmp by default, make sure it has a few GBs of free space.
+- **WALE_TMPDIR** or **WALG_TMPDIR**: directory to store WAL-G temporary files. PGROOT/../tmp by default, make sure it has a few GBs of free space.
 - **PGDATA**: location of PostgreSQL data directory, by default PGROOT/pgdata.
 - **PGUSER_STANDBY**: username for the replication user, 'standby' by default.
 - **PGPASSWORD_STANDBY**: a password for the replication user, 'standby' by default.
@@ -47,22 +47,22 @@ Environment Configuration Settings
 - **SSL_RESTAPI_PRIVATE_KEY**: content of the REST Api SSL private key in the SSL_PRIVATE_KEY_FILE file (by default /run/certs/server.key).
 - **SSL_TEST_RELOAD**: whenever to test for certificate rotation and reloading (by default True if SSL_PRIVATE_KEY_FILE has been set).
 - **RESTAPI_CONNECT_ADDRESS**: when you configure Patroni RESTAPI in SSL mode some safe API (i.e. switchover) perform hostname validation. In this case could be convenient configure ````restapi.connect_address````as a hostname instead of IP. For example, you can configure it as "$(POD_NAME).<service name>".
-- **WALE_BACKUP_THRESHOLD_MEGABYTES**: maximum size of the WAL segments accumulated after the base backup to consider WAL-E restore instead of pg_basebackup.
-- **WALE_BACKUP_THRESHOLD_PERCENTAGE**: maximum ratio (in percents) of the accumulated WAL files to the base backup to consider WAL-E restore instead of pg_basebackup.
-- **WALE_ENV_DIR**: directory where to store WAL-E environment variables
+- **WALG_BACKUP_THRESHOLD_MEGABYTES** or **WALE_BACKUP_THRESHOLD_MEGABYTES**: maximum size of the WAL segments accumulated after the base backup to consider WAL-G restore instead of pg_basebackup.
+- **WALG_BACKUP_THRESHOLD_PERCENTAGE** or **WALE_BACKUP_THRESHOLD_PERCENTAGE**: maximum ratio (in percents) of the accumulated WAL files to the base backup to consider WAL-G restore instead of pg_basebackup.
+- **WALG_ENV_DIR** or **WALE_ENV_DIR**: directory where to store WAL-G environment variables
 - **WAL_RESTORE_TIMEOUT**: timeout (in seconds) for restoring a single WAL file (at most 16 MB) from the backup location, 0 by default. A duration of 0 disables the timeout.
-- **WAL_S3_BUCKET**: (optional) name of the S3 bucket used for WAL-E base backups.
+- **WAL_S3_BUCKET**: (optional) name of the S3 bucket used for WAL-G base backups.
 - **AWS_ACCESS_KEY_ID**: (optional) aws access key
 - **AWS_SECRET_ACCESS_KEY**: (optional) aws secret key
 - **AWS_REGION**: (optional) region of S3 bucket
 - **AWS_ENDPOINT**: (optional) in format 'https://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_REGION
-- **WALE_S3_ENDPOINT**: (optional) in format 'https+path://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_ENDPOINT or AWS_REGION
-- **WALE_S3_PREFIX**: (optional) the full path to the backup location on S3 in the format s3://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_S3_BUCKET.
-- **WAL_GS_BUCKET**: ditto for the Google Cloud Storage (WAL-E supports both S3 and GCS).
-- **WALE_GS_PREFIX**: (optional) the full path to the backup location on the Google Cloud Storage in the format gs://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_GS_BUCKET.
-- **GOOGLE_APPLICATION_CREDENTIALS**: credentials for WAL-E when running in Google Cloud.
+- **WALG_S3_ENDPOINT** or **WALE_S3_ENDPOINT**: (optional) in format 'https+path://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_ENDPOINT or AWS_REGION
+- **WALG_S3_PREFIX** or **WALE_S3_PREFIX**: (optional) the full path to the backup location on S3 in the format s3://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_S3_BUCKET.
+- **WAL_GS_BUCKET**: ditto for the Google Cloud Storage (WAL-G supports both S3 and GCS).
+- **WALG_GS_PREFIX** or **WALE_GS_PREFIX**: (optional) the full path to the backup location on the Google Cloud Storage in the format gs://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_GS_BUCKET.
+- **GOOGLE_APPLICATION_CREDENTIALS**: credentials for WAL-G when running in Google Cloud.
 - **WAL_SWIFT_BUCKET**: ditto for the OpenStack Object Storage (Swift)
-- **SWIFT_AUTHURL**: see wal-e documentation https://github.com/wal-e/wal-e#swift
+- **SWIFT_AUTHURL**: see wal-g documentation https://wal-g.readthedocs.io/STORAGES/#swift
 - **SWIFT_TENANT**:
 - **SWIFT_TENANT_ID**:
 - **SWIFT_USER**:
@@ -79,7 +79,7 @@ Environment Configuration Settings
 - **SWIFT_PROJECT_ID**:
 - **SWIFT_PROJECT_DOMAIN_NAME**:
 - **SWIFT_PROJECT_DOMAIN_ID**:
-- **WALE_SWIFT_PREFIX**: (optional) the full path to the backup location on the Swift Storage in the format swift://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_SWIFT_BUCKET.
+- **WALG_SWIFT_PREFIX** or **WALE_SWIFT_PREFIX**: (optional) the full path to the backup location on the Swift Storage in the format swift://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_SWIFT_BUCKET.
 - **SSH_USERNAME**: (optional) the username for WAL backups.
 - **SSH_PORT**: (optional) the ssh port for WAL backups.
 - **SSH_PRIVATE_KEY_PATH**: (optional) the path to the private key used for WAL backups.
@@ -109,18 +109,17 @@ Environment Configuration Settings
 - **KUBERNETES_BOOTSTRAP_LABELS**: a JSON describing names and values of labels used by Patroni as ``kubernetes.bootstrap_labels``. Default is empty.
 - **INITDB_LOCALE**: database cluster's default UTF-8 locale (en_US by default)
 - **ENABLE_WAL_PATH_COMPAT**: old Spilo images were generating wal path in the backup store using the following template ``/spilo/{WAL_BUCKET_SCOPE_PREFIX}{SCOPE}{WAL_BUCKET_SCOPE_SUFFIX}/wal/``, while new images adding one additional directory (``{PGVERSION}``) to the end. In order to avoid (unlikely) issues with restoring WALs (from S3/GC/and so on) when switching to ``spilo-13`` please set the ``ENABLE_WAL_PATH_COMPAT=true`` when deploying old cluster with ``spilo-13`` for the first time. After that the environment variable could be removed. Change of the WAL path also mean that backups stored in the old location will not be cleaned up automatically.
-- **WALE_DISABLE_S3_SSE**, **WALG_DISABLE_S3_SSE**: by default wal-e/wal-g are configured to encrypt files uploaded to S3. In order to disable it you can set this environment variable to ``true``.
+- **WALG_DISABLE_S3_SSE** or **WALE_DISABLE_S3_SSE**: by default wal-g is configured to encrypt files uploaded to S3. In order to disable it you can set this environment variable to ``true``.
 - **USE_OLD_LOCALES**: whether to use old locales from Ubuntu 18.04 in the Ubuntu 22.04-based image. Default is false.
 
 wal-g
 -----
 
-`wal-g` is used by default for Azure and SSH backups and restore.
-In case of S3, `wal-e` is used for backups and `wal-g` for restore.
-
-- **USE_WALG_BACKUP**: (optional) Enforce using `wal-g` instead of `wal-e` for backups (Boolean)
-- **USE_WALG_RESTORE**: (optional) Enforce using `wal-g` instead of `wal-e` for restores (Boolean)
-
+wal-g is used everywhere in Spilo to perform backups and restore from them. **Support for wal-e has been removed**.
+Backward compatibility is ensured for environment variables containing **WALE**, the env-dir layout, and bootstrap method names.
+This allows existing configurations and clusters to continue working without requiring immediate changes.
+Regardless of which variable is set, all backups and restores will be performed using wal-g.
+However, if both **WALE** and **WALG** variables are present, the latter will take precedence.
 - **WALG_DELTA_MAX_STEPS**, **WALG_DELTA_ORIGIN**, **WALG_DOWNLOAD_CONCURRENCY**, **WALG_UPLOAD_CONCURRENCY**, **WALG_UPLOAD_DISK_CONCURRENCY**, **WALG_DISK_RATE_LIMIT**, **WALG_NETWORK_RATE_LIMIT**, **WALG_COMPRESSION_METHOD**, **WALG_BACKUP_COMPRESSION_METHOD**, **WALG_BACKUP_FROM_REPLICA**, **WALG_SENTINEL_USER_DATA**, **WALG_PREVENT_WAL_OVERWRITE**: (optional) configuration options for wal-g.
 - **WALG_S3_CA_CERT_FILE**: (optional) TLS CA certificate for wal-g (see [wal-g configuration](https://github.com/wal-g/wal-g#configuration))
 - **WALG_SSH_PREFIX**: (optional) the ssh prefix to store WAL backups at in the format ssh://host.example.com/path/to/backups/ See `Wal-g <https://github.com/wal-g/wal-g#configuration>`__ documentation for details.

delivery.yaml

@@ -3,7 +3,8 @@ allow_concurrent_steps: true
 
 build_env: &BUILD_ENV
   BASE_IMAGE: container-registry.zalando.net/library/ubuntu-22.04
-  PGVERSION: 17
+  PGVERSION: 18
+  PGOLDVERSIONS: "16 17"
   MULTI_ARCH_REGISTRY: container-registry-test.zalando.net/acid
 
 pipeline:
@@ -32,7 +33,7 @@ pipeline:
       docker buildx build --platform "linux/amd64,linux/arm64" \
                           --build-arg PGVERSION="$PGVERSION" \
                           --build-arg BASE_IMAGE="$BASE_IMAGE" \
-                          --build-arg PGOLDVERSIONS="14 15 16" \
+                          --build-arg PGOLDVERSIONS="$PGOLDVERSIONS" \
                           -t "$ECR_TEST_IMAGE" \
                           --push .
 
@@ -61,7 +62,7 @@ pipeline:
       docker buildx build --platform "linux/amd64,linux/arm64" \
                           --build-arg PGVERSION="$PGVERSION" \
                           --build-arg BASE_IMAGE="$BASE_IMAGE" \
-                          --build-arg PGOLDVERSIONS="14 15 16" \
+                          --build-arg PGOLDVERSIONS="$PGOLDVERSIONS" \
                           -t "$ECR_TEST_IMAGE" \
                           --push .
       cdp-promote-image "$ECR_TEST_IMAGE"
@@ -92,7 +93,7 @@ pipeline:
       docker buildx build --platform "linux/amd64,linux/arm64" \
                           --build-arg PGVERSION="$PGVERSION" \
                           --build-arg BASE_IMAGE="$BASE_IMAGE" \
-                          --build-arg PGOLDVERSIONS="14 15 16" \
+                          --build-arg PGOLDVERSIONS="$PGOLDVERSIONS" \
                           -t "$ECR_TEST_IMAGE" \
                           --push .
       cdp-promote-image "$ECR_TEST_IMAGE"

postgres-appliance/Dockerfile

@@ -1,5 +1,5 @@
 ARG BASE_IMAGE=ubuntu:22.04
-ARG PGVERSION=17
+ARG PGVERSION=18
 ARG DEMO=false
 ARG COMPRESS=false
 ARG ADDITIONAL_LOCALES=
@@ -18,7 +18,7 @@ FROM $BASE_IMAGE as dependencies-builder
 
 ARG DEMO
 
-ENV WALG_VERSION=v3.0.5
+ENV WALG_VERSION=v3.0.8
 
 COPY build_scripts/dependencies.sh /builddeps/
 
@@ -46,37 +46,34 @@ ARG PGVERSION
 ARG TIMESCALEDB_APACHE_ONLY=true
 ARG TIMESCALEDB_TOOLKIT=true
 ARG COMPRESS
-ARG PGOLDVERSIONS="13 14 15 16"
+ARG PGOLDVERSIONS="14 15 16 17"
 ARG WITH_PERL=false
 
 ARG DEB_PG_SUPPORTED_VERSIONS="$PGOLDVERSIONS $PGVERSION"
 
 # Install PostgreSQL, extensions and contribs
-ENV POSTGIS_VERSION=3.5 \
-    BG_MON_COMMIT=7f5887218790b263fe3f42f85f4ddc9c8400b154 \
+ENV POSTGIS_VERSION=3.6 \
+    BG_MON_COMMIT=a73c6bcd10dfdf9feaf5eabab7eb6b12d167680d \
     PG_AUTH_MON_COMMIT=fe099eef7662cbc85b0b79191f47f52f1e96b779 \
-    PG_MON_COMMIT=ead1de70794ed62ca1e34d4022f6165ff36e9a91 \
-    SET_USER=REL4_1_0 \
+    PG_MON_COMMIT=88ac7b58348aa061c814982defc170644f368f39 \
     PLPROFILER=REL4_2_5 \
-    PG_PROFILE=4.7 \
-    PAM_OAUTH2=v1.0.1 \
-    PG_PERMISSIONS_COMMIT=f4b7c18676fa64236a1c8e28d34a35764e4a70e2
+    PG_PROFILE=4.11 \
+    PAM_OAUTH2=v1.0.1
 
 WORKDIR /builddeps
 RUN bash base.sh
 
 # Install wal-g
 COPY --from=dependencies-builder /builddeps/wal-g /usr/local/bin/
 
-COPY build_scripts/patroni_wale.sh build_scripts/compress_build.sh /builddeps/
+COPY build_scripts/patroni.sh build_scripts/compress_build.sh /builddeps/
 
-# Install patroni and wal-e
-ENV PATRONIVERSION=4.0.4
-ENV WALE_VERSION=1.1.1
+# Install patroni
+ENV PATRONIVERSION=4.1.0
 
 WORKDIR /
 
-RUN bash /builddeps/patroni_wale.sh
+RUN bash /builddeps/patroni.sh
 
 RUN if [ "$COMPRESS" = "true" ]; then bash /builddeps/compress_build.sh; fi
 
@@ -101,7 +98,7 @@ ENV LC_ALL=en_US.utf-8 \
     RW_DIR=/run \
     DEMO=$DEMO
 
-ENV WALE_ENV_DIR=$RW_DIR/etc/wal-e.d/env \
+ENV WALG_ENV_DIR=$RW_DIR/etc/wal-e.d/env \
     LOG_ENV_DIR=$RW_DIR/etc/log.d/env \
     PGROOT=$PGHOME/pgdata/pgroot
 

…s-appliance/bootstrap/clone_with_wale.py …s-appliance/bootstrap/clone_with_walg.pypostgres-appliance/bootstrap/clone_with_wale.py renamed to postgres-appliance/bootstrap/clone_with_walg.py postgres-appliance/bootstrap/clone_with_wale.py renamed to postgres-appliance/bootstrap/clone_with_walg.py

@@ -25,7 +25,7 @@ def read_configuration():
     parser.add_argument('--recovery-target-time',
                         help='the timestamp up to which recovery will proceed (including time zone)',
                         dest='recovery_target_time_string')
-    parser.add_argument('--dry-run', action='store_true', help='find a matching backup and build the wal-e '
+    parser.add_argument('--dry-run', action='store_true', help='find a matching backup and build the wal-g.'
                         'command to fetch that backup without running it')
     args = parser.parse_args()
 
@@ -40,8 +40,8 @@ def read_configuration():
     return options(args.scope, args.datadir, recovery_target_time, args.dry_run)
 
 
-def build_wale_command(command, datadir=None, backup=None):
-    cmd = ['wal-g' if os.getenv('USE_WALG_RESTORE') == 'true' else 'wal-e'] + [command]
+def build_walg_command(command, datadir=None, backup=None):
+    cmd = ['wal-g', command]
     if command == 'backup-fetch':
         if datadir is None or backup is None:
             raise Exception("backup-fetch requires datadir and backup arguments")
@@ -79,7 +79,7 @@ def choose_backup(backup_list, recovery_target_time):
 
 
 def list_backups(env):
-    backup_list_cmd = build_wale_command('backup-list')
+    backup_list_cmd = build_walg_command('backup-list')
     output = subprocess.check_output(backup_list_cmd, env=env)
     reader = csv.DictReader(fix_output(output), dialect='excel-tab')
     return list(reader)
@@ -89,7 +89,7 @@ def get_clone_envdir():
     from spilo_commons import get_patroni_config
 
     config = get_patroni_config()
-    restore_command = shlex.split(config['bootstrap']['clone_with_wale']['recovery_conf']['restore_command'])
+    restore_command = shlex.split(config['bootstrap']['clone_with_walg']['recovery_conf']['restore_command'])
     if len(restore_command) > 4 and restore_command[0] == 'envdir':
         return restore_command[1]
     raise Exception('Failed to find clone envdir')
@@ -117,10 +117,9 @@ def get_possible_versions():
     return [ver for _, ver in sorted(versions.items(), reverse=True)]
 
 
-def get_wale_environments(env):
-    use_walg = env.get('USE_WALG_RESTORE') == 'true'
-    prefix = 'WALG_' if use_walg else 'WALE_'
-    # len('WALE__PREFIX') = 12
+def get_walg_environments(env):
+    prefix = 'WALG_'
+    # len('WALG_PREFIX') = 12
     names = [name for name in env.keys() if name.endswith('_PREFIX') and name.startswith(prefix) and len(name) > 12]
     if len(names) != 1:
         raise Exception('Found find {0} {1}*_PREFIX environment variables, expected 1'
@@ -141,7 +140,7 @@ def get_wale_environments(env):
 
 def find_backup(recovery_target_time, env):
     old_value = None
-    for name, value in get_wale_environments(env):
+    for name, value in get_walg_environments(env):
         logger.info('Trying %s for clone', value)
         if not old_value:
             old_value = env[name]
@@ -164,12 +163,12 @@ def run_clone_from_s3(options):
 
     backup_name, update_envdir = find_backup(options.recovery_target_time, env)
 
-    backup_fetch_cmd = build_wale_command('backup-fetch', options.datadir, backup_name)
+    backup_fetch_cmd = build_walg_command('backup-fetch', options.datadir, backup_name)
     logger.info("cloning cluster %s using %s", options.name, ' '.join(backup_fetch_cmd))
     if not options.dry_run:
         ret = subprocess.call(backup_fetch_cmd, env=env)
         if ret != 0:
-            raise Exception("wal-e backup-fetch exited with exit code {0}".format(ret))
+            raise Exception("wal-g backup-fetch exited with exit code {0}".format(ret))
 
         if update_envdir:  # We need to update file in the clone envdir or restore_command will fail!
             envdir = get_clone_envdir()

postgres-appliance/bootstrap/maybe_pg_upgrade.py

@@ -40,7 +40,7 @@ def perform_pitr(postgresql, cluster_version, bin_version, config):
     except Exception:
         logs = tail_postgres_logs()
         # Spilo has no other locales except en_EN.UTF-8, therefore we are safe here.
-        if int(cluster_version) >= 13 and 'recovery ended before configured recovery target was reached' in logs:
+        if 'recovery ended before configured recovery target was reached' in logs:
             # Starting from version 13 Postgres stopped promoting when recovery target wasn't reached.
             # In order to improve the user experience we reset all possible recovery targets and retry.
             recovery_conf = config[config['method']].get('recovery_conf', {})
@@ -103,7 +103,7 @@ def main():
     except Exception as e:
         logger.error('Failed to update extensions: %r', e)
 
-    upgrade.analyze()
+    upgrade.analyze(bin_version)
 
 
 def call_maybe_pg_upgrade():