Skip to content

Commit c2806e6

Browse files
ianrobertsianroberts
authored
Merge pull request #12 from GateNLP/teamware-backup-rclone
feat: allow backups to other storage services besides S3
2 parents 0e20a44 + 38494ec commit c2806e6

5 files changed

Lines changed: 128 additions & 35 deletions

File tree

‎gate-teamware/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ type: application
1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
1717
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18-
version: 2.3.0
18+
version: 2.3.1
1919

2020
# This is the version number of the application being deployed. This version number should be
2121
# incremented each time you make changes to the application. Versions are not expected to

‎gate-teamware/README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ By default the chart uses the publicly available Teamware images published on `g
103103

104104
The images to be run are specified in three parts, the top-level `imageRegistry` key in the values file is the registry prefix (by default `ghcr.io/gatenlp/`) which _must_ end with a slash, then `backend` and `staticFiles` each have `image.repository` for the image name (default "teamware-backend" and "teamware-static" respectively) and `image.tag` for the tag, which defaults to match the chart "appVersion" number, plus `pullPolicy` (default "IfNotPresent") and `pullSecrets` (if you are using a private registry whose credentials are not already configured on the default ServiceAccount for this namespace). So if you store your images in a private registry but still name them `teamware-backend` and `teamware-static` then the only thing you should need to override is the `imageRegistry`.
105105

106-
The chart also supports running regular backups of the database to S3 (or a compatible storage system), these can be configured using the settings under the `backup` section, see [`values.yaml`](gate-teamware/values.yaml) for more details.
106+
The chart also supports running regular backups of the database to S3 (or a compatible storage system), Azure Blob Storage, an SFTP server, or any other datastore supported by the [rclone](https://rclone.org) tool, these can be configured using the settings under the `backup` section, see [`values.yaml`](gate-teamware/values.yaml) for more details.
107107

108108
## Install/upgrade
109109

@@ -156,6 +156,12 @@ and the deployments may need to be manually updated using `kubectl rollout resta
156156

157157
## Changelog
158158

159+
### Version 2.3.1
160+
161+
Now supports backups to storage providers other than S3 - the backup cronjob has been switched to use the GateNLP [postgresql-backup-rclone](https://github.com/GateNLP/db-backup-rclone) image, which uploads the database backup files using [rclone](https://rclone.org), which supports many other storage providers besides S3, as well as other S3 authentication mechanisms besides just a static access key and secret key pair.
162+
163+
This change is backwards compatible, existing local `values` files should continue to work unchanged with the new method.
164+
159165
### Version 2.3.0
160166

161167
Upgraded `appVersion` to [GATE Teamware 2.3.0](https://github.com/GateNLP/gate-teamware/releases/tag/v2.3.0).

‎gate-teamware/templates/_helpers.tpl‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,19 @@ specified existing secret or one generated by this chart.
7171
{{- printf "%s-superuser" (include "gate-teamware.fullname" .) -}}
7272
{{- end -}}
7373
{{- end }}
74+
75+
{{/*
76+
Determine the upload prefix for backup jobs (possibly empty).
77+
*/}}
78+
{{- define "gate-teamware.backupPrefix" -}}
79+
{{- if .Values.backup.uploadPrefix -}}
80+
{{- .Values.backup.uploadPrefix -}}
81+
{{- if not (.Values.backup.uploadPrefix | hasSuffix "/") }}/{{- end -}}
82+
{{- else if eq .Values.backup.provider "s3" -}}
83+
{{- printf "%s/" (.Values.backup.bucketName | required "bucketName is required if uploadPrefix is not set") -}}
84+
{{- if .Values.backup.keyPrefix -}}
85+
{{- .Values.backup.keyPrefix -}}
86+
{{- if not (.Values.backup.keyPrefix | hasSuffix "/") }}/{{- end -}}
87+
{{- end -}}
88+
{{- end -}}
89+
{{- end }}

‎gate-teamware/templates/cronjob-backup.yaml‎

Lines changed: 61 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,13 @@ spec:
2626
{{- include "gate-teamware.labels" $ | nindent 12 }}
2727
{{- include "gate-teamware.selectorLabels" $ | nindent 12 }}
2828
app.kubernetes.io/component: backup-{{ $name }}
29+
{{- with $.Values.backup.podLabels }}
30+
{{- toYaml . | nindent 12 }}
31+
{{- end }}
32+
{{- with $.Values.backup.podAnnotations }}
33+
annotations:
34+
{{- toYaml . | nindent 12 }}
35+
{{- end }}
2936
spec:
3037
serviceAccountName: {{ include "gate-teamware.serviceAccountName" $ }}
3138
restartPolicy: Never
@@ -34,41 +41,72 @@ spec:
3441
image: {{ $.Values.backup.image | quote }}
3542
imagePullPolicy: {{ $.Values.backup.pullPolicy }}
3643
env:
37-
- name: S3_ACCESS_KEY_ID
44+
- name: PGHOST
45+
value: {{ include "postgresql.primary.fullname" $.Subcharts.postgresql | quote }}
46+
- name: PGPORT
47+
value: {{ include "postgresql.service.port" $.Subcharts.postgresql | quote }}
48+
- name: BACKUP_DATABASES
49+
value: {{ $.Values.postgresql.auth.database | quote }}
50+
- name: PGUSER
51+
value: {{ $.Values.postgresql.auth.username | quote }}
52+
- name: PGPASSWORD
3853
valueFrom:
3954
secretKeyRef:
40-
name: {{ $.Values.backup.credentialsSecret | required "backup.credentialsSecret is required" | quote }}
55+
name: {{ include "postgresql.secretName" $.Subcharts.postgresql }}
56+
key: password
57+
{{- if eq $.Values.backup.provider "s3" }}
58+
- name: RCLONE_CONFIG_STORE_TYPE
59+
value: s3
60+
- name: RCLONE_CONFIG_STORE_ENV_AUTH
61+
value: "true"
62+
- name: RCLONE_S3_NO_CHECK_BUCKET
63+
value: "true"
64+
# Don't double check uploads after completion - this is necessary if
65+
# our credentials are "write-only" (allowing put but not get)
66+
- name: RCLONE_S3_NO_HEAD
67+
value: "true"
68+
# Skip checksum checks, since these are not compatible with S3 server-side encryption
69+
- name: RCLONE_IGNORE_CHECKSUM
70+
value: "true"
71+
{{- if $.Values.backup.credentialsSecret }}
72+
- name: AWS_ACCESS_KEY_ID
73+
valueFrom:
74+
secretKeyRef:
75+
name: {{ $.Values.backup.credentialsSecret | quote }}
4176
key: access-key-id
42-
- name: S3_SECRET_ACCESS_KEY
77+
- name: AWS_SECRET_ACCESS_KEY
4378
valueFrom:
4479
secretKeyRef:
45-
name: {{ $.Values.backup.credentialsSecret | required "backup.credentialsSecret is required" | quote }}
80+
name: {{ $.Values.backup.credentialsSecret | quote }}
4681
key: secret-key
47-
- name: S3_BUCKET
48-
value: {{ $.Values.backup.bucketName | required "backup.bucketName is required" | quote }}
49-
- name: S3_PREFIX
50-
value: {{ with $.Values.backup.keyPrefix }}{{ . }}/{{ end }}{{ $name }}
82+
{{- end }}
5183
{{- if $.Values.backup.endpoint }}
52-
- name: S3_ENDPOINT
84+
- name: AWS_ENDPOINT_URL_S3
5385
value: {{ $.Values.backup.endpoint | quote }}
5486
{{- end }}
5587
{{- if $.Values.backup.region }}
56-
- name: S3_REGION
88+
- name: AWS_REGION
5789
value: {{ $.Values.backup.region | quote }}
5890
{{- end }}
59-
- name: POSTGRES_HOST
60-
value: {{ include "postgresql.primary.fullname" $.Subcharts.postgresql | quote }}
61-
- name: POSTGRES_PORT
62-
value: {{ include "postgresql.service.port" $.Subcharts.postgresql | quote }}
63-
- name: POSTGRES_DATABASE
64-
value: {{ $.Values.postgresql.auth.database | quote }}
65-
- name: POSTGRES_USER
66-
value: {{ $.Values.postgresql.auth.username | quote }}
67-
- name: POSTGRES_PASSWORD
68-
valueFrom:
69-
secretKeyRef:
70-
name: {{ include "postgresql.secretName" $.Subcharts.postgresql }}
71-
key: password
91+
{{- end }}
92+
- name: UPLOAD_PREFIX
93+
value: {{ printf "%s%s" (include "gate-teamware.backupPrefix" $) $name | quote }}
94+
{{- with $.Values.backup.extraEnv }}
95+
{{- toYaml . | nindent 14 }}
96+
{{- end }}
97+
{{- with $.Values.backup.extraArgs }}
98+
args:
99+
{{- toYaml . | nindent 14 }}
100+
{{- end }}
101+
{{- with $.Values.backup.extraVolumeMounts }}
102+
volumeMounts:
103+
{{- toYaml . | nindent 14 }}
104+
{{- end }}
105+
106+
{{- with $.Values.backup.extraVolumes }}
107+
volumes:
108+
{{- toYaml . | nindent 10 }}
109+
{{- end }}
72110

73111
---
74112
{{- end }}{{/* range over schedule */}}

‎gate-teamware/values.yaml‎

Lines changed: 43 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -352,22 +352,37 @@ postgresql:
352352

353353
# Additional useful settings include persistence.storageClass and persistence.size
354354

355-
# Configure regular database backups to S3 (or compatible) storage
355+
# Configure regular database backups to any rclone-supported storage provider.
356356
backup:
357357
# Image for the backup cronjob
358-
image: "schickling/postgres-backup-s3:latest"
358+
image: "ghcr.io/gatenlp/postgresql-backup-rclone:latest"
359359
# Pull policy for the backup image
360360
pullPolicy: Always
361-
# API endpoint, if not the public Amazon S3
361+
# Backup storage provider. Default is "s3" for backwards compatibility, but
362+
# you can set this value to "custom" and provide your own rclone configuration
363+
# in extraEnv, extraVolumes and extraVolumeMounts to use any other provider
364+
# supported by rclone.
365+
#
366+
# See https://github.com/GateNLP/db-backup-rclone for full details.
367+
provider: s3
368+
# API endpoint, if not the public Amazon S3. Ignored if provider is not s3.
362369
endpoint: ""
363-
# S3 region e.g. us-east-1
370+
# S3 region e.g. us-east-1. Ignored if provider is not s3.
364371
region: ""
365-
# Bucket to which the backups should be stored
372+
# Prefix within the configured rclone remote where the backup files should
373+
# be placed - exactly how this is interpreted depends on the configuration,
374+
# for S3 the first path segment should be the bucket name.
375+
#
376+
# If this value is set, bucketName and keyPrefix are ignored.
377+
uploadPrefix: ""
378+
# Bucket to which the backups should be stored. Ignored if provider is not
379+
# s3, or if uploadPrefix has been configured directly.
366380
bucketName: ""
367-
# common key prefix for all the backups
381+
# common key prefix for all the backups. Ignored if provider is not
382+
# s3, or if uploadPrefix has been configured directly.
368383
keyPrefix: ""
369384
# Kubernetes secret with keys "access-key-id" and "secret-key" that have
370-
# write access to the specified bucket
385+
# write access to the specified bucket. Ignored if provider is not s3
371386
credentialsSecret: ""
372387

373388
concurrencyPolicy: ""
@@ -377,13 +392,31 @@ backup:
377392
# Backup schedule. Keys in the map are the name of the backup schedule
378393
# (e.g. daily, weekly), values are the cron expression defining the schedule.
379394
# The actual backup files will be placed at
380-
# s3://<bucketName>/<keyPrefix>/<schedule-name>/<database-name>_<timestamp>.sql.gz
395+
# store:<uploadPrefix>/<schedule-name>/<database-name>_<timestamp>.sql.gz
381396
#
382397
# Note that the cron job creates new backups but does not expire old ones -
383398
# if you want old backups to expire (or be moved to cheaper storage like
384-
# infrequent access or Glacier) then you should configure that at the bucket
385-
# level using S3 lifecycle rules.
399+
# infrequent access or Glacier) then you should configure that using the
400+
# storage provider's own mechanisms, e.g. S3 lifecycle rules.
386401
schedule: {}
387402
# daily: "13 01 * * *"
388403
# weekly: "13 02 * * 0"
389404
# monthly: "13 00 1 * *"
405+
406+
# extra environment variable settings to be passed to the backup container
407+
extraEnv: []
408+
# extra command line arguments to be passed to the backup container (and
409+
# from there through to the pg_dump command)
410+
extraArgs: []
411+
# extra volume definitions added to the backup pod, e.g. an rclone
412+
# configuration file, private key for authentication to an sftp datastore,
413+
# TLS CA certificates, etc. etc.
414+
extraVolumes: []
415+
# volumeMounts entries to mount the extra volumes into the backup container
416+
extraVolumeMounts: []
417+
418+
# Additional labels to apply to the backup pods
419+
podLabels: {}
420+
# Annotations to apply to the backup pods
421+
podAnnotations: { }
422+

0 commit comments

Comments
 (0)