diff --git a/charts/db-sync/Chart.yaml b/charts/db-sync/Chart.yaml new file mode 100644 index 00000000000..4658cdc1eee --- /dev/null +++ b/charts/db-sync/Chart.yaml @@ -0,0 +1,14 @@ +apiVersion: v2 +name: db-sync +description: Helm chart for govuk-db-sync CronJobs (backup/restore for PostgreSQL, MySQL, DocumentDB) +type: application +version: 0.1.0 +appVersion: "0.1.0" +keywords: + - backup + - restore + - database + - cronjob +maintainers: + - name: GOV.UK Platform Engineering + email: govuk-platform-engineering@digital.cabinet-office.gov.uk diff --git a/charts/db-sync/README.md b/charts/db-sync/README.md new file mode 100644 index 00000000000..b6e4bcdf7c5 --- /dev/null +++ b/charts/db-sync/README.md @@ -0,0 +1,172 @@ +# db-sync Helm Chart + +Helm chart for deploying `govuk-db-sync` CronJobs to manage database backups and restores across PostgreSQL, MySQL, and DocumentDB. + +## Installation + +```bash +helm install db-sync ./charts/db-sync \ + -f values-staging.yaml \ + -n database-sync +``` + +## Configuration + +### Basic Job Configuration + +Each job in `values.yaml` requires: + +- **schedule**: Cron schedule (e.g., `"0 2 * * *"`) +- **operation**: `backup` or `restore` +- **dbType**: `postgres`, `mysql`, or `documentdb` +- **dbName**: Target database name +- **s3Bucket**: S3 bucket for backups (e.g., `s3://govuk-prod-backups`) +- **s3Path**: S3 path prefix (e.g., `myapp/db`) + +### Example Configuration + +```yaml +cronjobs: + production: + # PostgreSQL backup + account-api-postgres: + schedule: "37 23 * * *" + operation: backup + dbType: postgres + dbName: account-api_production + s3Bucket: s3://govuk-prod-database-backups + s3Path: account-api/db + + # PostgreSQL restore with transformation + email-alert-api-postgres: + schedule: "54 3 * * 1" + operation: restore + dbType: postgres + dbName: email-alert-api_production + s3Bucket: s3://govuk-staging-database-backups + s3Path: email-alert-api/db + extraEnv: + - name: DB_OWNER + value: email-alert-api + + # MySQL backup + release-mysql: + schedule: "11 3 * * 1" + operation: backup + dbType: mysql + dbName: release_production + s3Bucket: s3://govuk-prod-database-backups + s3Path: release/db + + # DocumentDB backup + publisher-documentdb: + schedule: "13 3 * * 1" + operation: backup + dbType: documentdb + dbName: publisher_production + s3Bucket: s3://govuk-prod-database-backups + s3Path: publisher/docdb +``` + +### Optional Job Configuration + +- **suspend**: Set to `true` to disable a job (default: `false`) +- **maxJobRuntimeSeconds**: Maximum job runtime in seconds (default: 43200 = 12 hours) +- **transformScript**: Name of a transform script including path. +- **extraEnv**: Job-specific environment variables +- **resources**: Override default resource limits/requests + +## Secrets Management + +This chart uses AWS Secrets Manager via the ExternalSecrets operator. + +**Expected Secret Structure:** + +Create a secret at `govuk/db-sync/passwd` with key/value pairs mapping job names to passwords: + +```json +{ + "account-api-postgres": "password123", + "email-alert-api-postgres": "password456", + "release-mysql": "password789", + "publisher-documentdb": "mongodb-connection-string" +} +``` + +The chart loads these credentials into Kubernetes Secret `db-sync-passwd`, which is referenced by CronJobs. + +## IAM Permissions + +The ServiceAccount uses AWS IAM Roles for Service Accounts (IRSA). Configure the annotation: + +```yaml +serviceAccount: + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/db-sync-role +``` + +**Required IAM Permissions:** +- S3: `s3:GetObject`, `s3:PutObject` on backup buckets +- RDS (for backups): Database instance access +- Secrets Manager: `secretsmanager:GetSecretValue` for `govuk/db-sync/passwd` +- CloudWatch (optional): Metrics publishing + +## Transform Scripts + +Place SQL/JavaScript transformation scripts in the `scripts/` directory: + +``` +charts/db-sync/ +├── scripts/ +│ ├── email-alert-api.sql +│ ├── content-store.sql +│ └── custom-transform.js +``` + +Reference them in job config: + +```yaml +transformScript: email-alert-api.sql +``` + +## Environment-Specific Values + +Create separate values files for each environment: + +```bash +values-production.yaml +values-staging.yaml +values-integration.yaml +``` + +Then deploy with: + +```bash +helm install db-sync ./charts/db-sync -f values-production.yaml +``` + +## Monitoring + +Each CronJob logs to stdout. Monitor failures via: +- Kubernetes Events: `kubectl describe cronjob db-sync-` +- Pod Logs: `kubectl logs -l app.kubernetes.io/component=` +- Failed Jobs: `kubectl get jobs --failed` + +## Troubleshooting + +**Job not running:** +- Check if suspended: `kubectl get cronjob db-sync-` +- Verify schedule is valid: https://crontab.guru/ + +**Secret not found:** +- Ensure ExternalSecret is synced: `kubectl describe externalsecret db-sync-passwd` +- Check AWS Secrets Manager: `aws secretsmanager get-secret-value --secret-id govuk/db-sync/passwd` + +**Pod crash:** +- Check pod logs: `kubectl logs -p` (for previous pod) +- Verify S3 bucket access and IAM permissions +- Check database connectivity and credentials + +## Values Reference + +See `values.yaml` for all available configuration options. diff --git a/charts/db-sync/templates/_helpers.tpl b/charts/db-sync/templates/_helpers.tpl new file mode 100644 index 00000000000..9977fb0c717 --- /dev/null +++ b/charts/db-sync/templates/_helpers.tpl @@ -0,0 +1,168 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "db-sync.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +*/}} +{{- define "db-sync.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "db-sync.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.AppVersion | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "db-sync.labels" -}} +helm.sh/chart: {{ include "db-sync.chart" . }} +{{ include "db-sync.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "db-sync.selectorLabels" -}} +app.kubernetes.io/name: {{ include "db-sync.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "db-sync.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "db-sync.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +Default DB port for a dbType. +Usage: include "db-sync.defaultDbPort" "postgres" +*/}} +{{- define "db-sync.defaultDbPort" -}} +{{- if eq . "postgres" -}} +5432 +{{- else if eq . "mysql" -}} +3306 +{{- else if eq . "documentdb" -}} +27017 +{{- end -}} +{{- end }} + +{{/* +URI scheme for a dbType. +Usage: include "db-sync.dbUriScheme" "postgres" +*/}} +{{- define "db-sync.dbUriScheme" -}} +{{- if eq . "postgres" -}} +postgres +{{- else if eq . "mysql" -}} +mysql +{{- else if eq . "documentdb" -}} +mongodb +{{- end -}} +{{- end }} + +{{/* +Transform DB user default by engine. +Expected dict keys: job +*/}} +{{- define "db-sync.transformUser" -}} +{{- $job := .job -}} +{{- if eq $job.dbType "documentdb" -}} +{{- default "mongoadmin" $job.transformUsername -}} +{{- else if eq $job.dbType "mysql" -}} +{{- default "root" $job.transformUsername -}} +{{- else -}} +{{- default "postgres" $job.transformUsername -}} +{{- end -}} +{{- end }} + +{{/* +Build source URI from components unless sourceUri is explicitly set. +Expected dict keys: root, job, jobName +*/}} +{{- define "db-sync.sourceUri" -}} +{{- $root := .root -}} +{{- $job := .job -}} +{{- $jobName := .jobName -}} +{{- if $job.sourceUri -}} +{{- $job.sourceUri -}} +{{- else -}} + {{- $hostname := default $jobName $job.sourceDbHostname -}} + {{- if $hostname -}} + {{- $port := $job.sourceDbPort -}} + {{- if not $port -}} + {{- $port = include "db-sync.defaultDbPort" $job.dbType | trim -}} + {{- end -}} + {{- $username := default $root.Values.defaultDbUsername $job.sourceDbUsername -}} + {{- $scheme := include "db-sync.dbUriScheme" $job.dbType | trim -}} + {{- printf "%s://%s@%s:%s/%s" $scheme $username $hostname $port $job.dbName -}} + {{- end -}} +{{- end -}} +{{- end }} + +{{/* +Build destination URI from components unless destUri is explicitly set. +Expected dict keys: root, job, jobName +*/}} +{{- define "db-sync.destUri" -}} +{{- $root := .root -}} +{{- $job := .job -}} +{{- $jobName := .jobName -}} +{{- if $job.destUri -}} +{{- $job.destUri -}} +{{- else if ne $job.dbType "documentdb" -}} + {{- $hostname := default $jobName $job.destDbHostname -}} + {{- if $hostname -}} + {{- $port := $job.destDbPort -}} + {{- if not $port -}} + {{- $port = include "db-sync.defaultDbPort" $job.dbType | trim -}} + {{- end -}} + {{- $username := default $root.Values.defaultDbUsername $job.destDbUsername -}} + {{- $scheme := include "db-sync.dbUriScheme" $job.dbType | trim -}} + {{- printf "%s://%s@%s:%s/%s" $scheme $username $hostname $port $job.dbName -}} + {{- end -}} +{{- end -}} +{{- end }} + +{{/* +Build transform URI from components unless transformUri is explicitly set. +Expected dict keys: job, transformUser +*/}} +{{- define "db-sync.transformUri" -}} +{{- $job := .job -}} +{{- $transformUser := .transformUser -}} +{{- if $job.transformUri -}} +{{- $job.transformUri -}} +{{- else -}} + {{- $scheme := include "db-sync.dbUriScheme" $job.dbType | trim -}} + {{- $port := include "db-sync.defaultDbPort" $job.dbType | trim -}} + {{- $transformDbName := default $job.dbName $job.transformDbName -}} + {{- printf "%s://%s@127.0.0.1:%s/%s" $scheme $transformUser $port $transformDbName -}} +{{- end -}} +{{- end }} diff --git a/charts/db-sync/templates/cronjob.yaml b/charts/db-sync/templates/cronjob.yaml new file mode 100644 index 00000000000..c53605a2fdb --- /dev/null +++ b/charts/db-sync/templates/cronjob.yaml @@ -0,0 +1,269 @@ +{{- $_ := .Values.govukEnvironment | required "govukEnvironment is required" }} +{{- $fullName := include "db-sync.fullname" . }} + +{{- range $jobName, $job := get .Values.cronjobs .Values.govukEnvironment }} + {{- $jobFullName := printf "%s-%s" $fullName $jobName }} + {{- $isBackup := eq $job.operation "backup" }} + {{- $isRestore := eq $job.operation "restore" }} + {{- $usesSource := $isBackup }} + {{- $usesDest := $isRestore }} + {{- $usesTransform := and $isBackup $job.transformScript }} + {{- $sourcePasswordKey := $job.sourcePasswordSecretKey | default $jobName }} + {{- $destPasswordKey := $job.destPasswordSecretKey | default $jobName }} + {{- $transformUser := include "db-sync.transformUser" (dict "job" $job) | trim }} + {{- $transformUri := include "db-sync.transformUri" (dict "job" $job "transformUser" $transformUser) | trim }} + {{- $sourceUri := include "db-sync.sourceUri" (dict "root" $ "job" $job "jobName" $jobName) | trim }} + {{- $destUri := include "db-sync.destUri" (dict "root" $ "job" $job "jobName" $jobName) | trim }} + {{- /* Ensure required fields are present */}} + {{- $_ := $job.schedule | required (printf "schedule is required for job %s" $jobName) }} + {{- $_ := $job.operation | required (printf "operation is required for job %s" $jobName) }} + {{- $_ := $job.dbType | required (printf "dbType is required for job %s" $jobName) }} + {{- $_ := $job.dbName | required (printf "dbName is required for job %s" $jobName) }} + {{- $s3Bucket := $job.s3Bucket | default $.Values.defaultS3Bucket }} + {{- $_ := $s3Bucket | required (printf "s3Bucket is required for job %s (set per-job or via defaultS3Bucket)" $jobName) }} + {{- $_ := $job.s3Path | required (printf "s3Path is required for job %s" $jobName) }} + {{- if and $isBackup (not $sourceUri) }} + {{- fail (printf "backup job %s requires either sourceUri or a resolvable source hostname" $jobName) }} + {{- end }} + {{- if and $isRestore (ne $job.dbType "documentdb") (not $destUri) }} + {{- fail (printf "restore job %s requires either destUri or a resolvable destination hostname" $jobName) }} + {{- end }} +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: {{ $jobFullName }} + annotations: + argocd.argoproj.io/ignore-healthcheck: "true" + labels: + {{- include "db-sync.labels" $ | nindent 4 }} + app.kubernetes.io/component: {{ $jobName }} +spec: + schedule: {{ $job.schedule | quote }} + concurrencyPolicy: Forbid + startingDeadlineSeconds: 28800 # 8 hours + suspend: {{ or $.Values.suspendAllCronJobs $job.suspend | default false }} + failedJobsHistoryLimit: 2 + successfulJobsHistoryLimit: 2 + jobTemplate: + metadata: + name: {{ $jobFullName }} + labels: + {{- include "db-sync.selectorLabels" $ | nindent 8 }} + app.kubernetes.io/component: {{ $jobName }} + spec: + activeDeadlineSeconds: {{ $job.maxJobRuntimeSeconds | default 43200 }} # Default: 12 hours + backoffLimit: 0 + # Ignore pod disruptions + podFailurePolicy: + rules: + - action: Ignore + onPodConditions: + - type: DisruptionTarget + status: "True" + template: + metadata: + name: {{ $jobFullName }} + labels: + {{- include "db-sync.selectorLabels" $ | nindent 12 }} + app.kubernetes.io/component: {{ $jobName }} + annotations: + cluster-autoscaler.kubernetes.io/safe-to-evict: "false" + spec: + enableServiceLinks: false + restartPolicy: Never + dnsConfig: + searches: + - blue.{{ $.Values.govukEnvironment }}.govuk-internal.digital + - {{ $.Values.govukEnvironment }}.govuk-internal.digital + serviceAccountName: {{ include "db-sync.serviceAccountName" $ }} + securityContext: + {{- toYaml $.Values.podSecurityContext | nindent 12 }} + volumes: + - name: tmp + emptyDir: {} + {{- if $usesTransform }} + initContainers: + {{ if eq $job.dbType "documentdb" }} + - name: transform-db + image: "{{ $.Values.transformImage.docdb.repository }}:{{ $.Values.transformImage.docdb.tag }}" + imagePullPolicy: {{ $.Values.transformImage.docdb.pullPolicy }} + restartPolicy: Always + command: + - /app/transform-db + ports: + - containerPort: 27017 + name: documentdb + env: + - name: GOVUK_ENVIRONMENT + value: {{ $.Values.govukEnvironment }} + - name: MONGO_INITDB_ROOT_PASSWORD + value: {{ $job.transformPassword | default "secret" | quote }} + - name: MONGO_INITDB_ROOT_USERNAME + value: {{ $transformUser | quote }} + resources: + {{- toYaml $job.transformResources | default $.Values.transformResources | nindent 16 }} + securityContext: + {{- toYaml $.Values.transformSecurityContext | nindent 16 }} + {{ else if eq $job.dbType "mysql" }} + - name: transform-db + image: "{{ $.Values.transformImage.mysql.repository }}:{{ $.Values.transformImage.mysql.tag }}" + imagePullPolicy: {{ $.Values.transformImage.mysql.pullPolicy }} + restartPolicy: Always + ports: + - containerPort: 3306 + name: mysql + env: + - name: GOVUK_ENVIRONMENT + value: {{ $.Values.govukEnvironment }} + - name: MYSQL_ROOT_PASSWORD + value: {{ $job.transformPassword | default "secret" | quote }} + resources: + {{- toYaml $job.transformResources | default $.Values.transformResources | nindent 16 }} + securityContext: + {{- toYaml $.Values.transformSecurityContext | nindent 16 }} + {{ else if eq $job.dbType "postgres" }} + - name: transform-db + image: "{{ $.Values.transformImage.postgres.repository }}:{{ $.Values.transformImage.postgres.tag }}" + imagePullPolicy: {{ $.Values.transformImage.postgres.pullPolicy }} + restartPolicy: Always + command: + - /app/transform-db + ports: + - containerPort: 5432 + name: postgres + env: + - name: GOVUK_ENVIRONMENT + value: {{ $.Values.govukEnvironment }} + - name: POSTGRES_PASSWORD + value: {{ $job.transformPassword | default "secret" | quote }} + - name: POSTGRES_USER + value: {{ $transformUser | quote }} + resources: + {{- toYaml $job.transformResources | default $.Values.transformResources | nindent 16 }} + securityContext: + {{- toYaml $.Values.transformSecurityContext | nindent 16 }} + {{ else }} + {{- fail (printf "Unsupported dbType '%s' for job %s" $job.dbType $jobName) }} + {{- end }} + {{- end }} + containers: + - name: db-sync + image: "{{ $.Values.appImage.repository }}:{{ $.Values.appImage.tag }}" + imagePullPolicy: {{ $.Values.appImage.pullPolicy }} + command: + - /app/db-sync + args: + - {{ $job.operation }} + {{- if $usesTransform }} + - --transform-script + - /scripts/{{ $job.transformScript }} + {{- end }} + env: + - name: GOVUK_ENVIRONMENT + value: {{ $.Values.govukEnvironment }} + - name: PUSHGATEWAY_URL + valueFrom: + configMapKeyRef: + name: govuk-apps-env + key: PROMETHEUS_PUSHGATEWAY_URL + {{- if eq $job.dryRun "true" }} + - name: DRY_RUN + value: {{ $job.dryRun }} + {{- end }} + {{- if $job.appName }} + - name: APP_NAME + value: {{ $job.appName }} + {{- end }} + - name: DB_TYPE + value: {{ $job.dbType }} + - name: DB_NAME + value: {{ $job.dbName }} + {{- if $job.dbOwner }} + - name: DB_OWNER + value: {{ $job.dbOwner }} + {{- end }} + {{- if $job.threads }} + - name: THREADS + value: {{ $job.threads | quote }} + {{- end }} + {{- if $usesSource }} + {{- if $sourceUri }} + - name: SOURCE_URI + value: {{ $sourceUri }} + {{- end }} + {{- if ne $job.dbType "documentdb" }} + - name: SOURCE_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $fullName }}-passwd + key: {{ $sourcePasswordKey }} + {{- else if $job.sourcePasswordSecretKey }} + - name: SOURCE_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $fullName }}-passwd + key: {{ $job.sourcePasswordSecretKey }} + {{- end }} + {{- end }} + {{- if $usesDest }} + {{- if $destUri }} + - name: DEST_URI + value: {{ $destUri }} + {{- else if eq $job.dbType "documentdb" }} + - name: DEST_URI + valueFrom: + secretKeyRef: + name: {{ $fullName }}-passwd + key: {{ $jobName }}-dest + {{- end }} + {{- if and $isRestore (eq $job.dbType "documentdb") }} + {{- if $job.docdbSourceDb }} + - name: DOCDB_SOURCE_DB + value: {{ $job.docdbSourceDb }} + {{- end }} + {{- end }} + {{- if ne $job.dbType "documentdb" }} + - name: DEST_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $fullName }}-passwd + key: {{ $destPasswordKey }} + {{- else if $job.destPasswordSecretKey }} + - name: DEST_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $fullName }}-passwd + key: {{ $job.destPasswordSecretKey }} + {{- end }} + {{- end }} + - name: S3_REGION + value: {{ $job.s3Region | default "eu-west-1" }} + - name: S3_BUCKET + value: {{ $s3Bucket }} + {{- if $job.s3Path }} + - name: S3_PATH + value: {{ $job.s3Path }} + {{- end }} + {{- if $usesTransform }} + - name: TRANSFORM_SCRIPT + value: {{ $job.transformScript }} + - name: TRANSFORM_URI + value: {{ $transformUri }} + - name: TRANSFORM_PASSWORD + value: {{ $job.transformPassword | default "secret" | quote }} + {{- end }} + {{- with $.Values.extraEnv }} + {{- toYaml . | nindent 16 }} + {{- end }} + {{- with $job.extraEnv }} + {{- toYaml . | nindent 16 }} + {{- end }} + resources: + {{- toYaml $job.appResources | default $.Values.appResources | nindent 16 }} + securityContext: + {{- toYaml $.Values.appSecurityContext | nindent 16 }} + volumeMounts: + - name: tmp + mountPath: /tmp +{{- end }} diff --git a/charts/db-sync/templates/externalsecrets.yaml b/charts/db-sync/templates/externalsecrets.yaml new file mode 100644 index 00000000000..77fe92d739a --- /dev/null +++ b/charts/db-sync/templates/externalsecrets.yaml @@ -0,0 +1,27 @@ +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: {{ include "db-sync.fullname" . }}-passwd + labels: + {{- include "db-sync.labels" . | nindent 4 }} + annotations: + kubernetes.io/description: > + Map of hostname to aws_db_admin password for backup/restore of databases + hosted in RDS. The required key is the one used by the consuming job, + which defaults tothe pub name unless overridden by chart values. + TODO: stop using passwords. +spec: + refreshInterval: {{ .Values.externalSecrets.refreshInterval }} + secretStoreRef: + name: aws-secretsmanager + kind: ClusterSecretStore + target: + deletionPolicy: {{ .Values.externalSecrets.deletionPolicy }} + name: {{ include "db-sync.fullname" . }}-passwd + dataFrom: + - extract: + key: {{ .Values.externalSecrets.secretKey }} + conversionStrategy: Default + decodingStrategy: None + metadataPolicy: None diff --git a/charts/db-sync/templates/serviceaccount.yaml b/charts/db-sync/templates/serviceaccount.yaml new file mode 100644 index 00000000000..b7f7043380d --- /dev/null +++ b/charts/db-sync/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "db-sync.serviceAccountName" . }} + labels: + {{- include "db-sync.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/db-sync/values-production.yaml b/charts/db-sync/values-production.yaml new file mode 100644 index 00000000000..0b4257626a9 --- /dev/null +++ b/charts/db-sync/values-production.yaml @@ -0,0 +1,236 @@ +# Production values for db-sync Helm Chart +# Overrides govukEnvironment and provides production-specific job configuration + +govukEnvironment: production + +defaultS3Bucket: s3://govuk-production-rds-dumps + +cronjobs: + production: + # PostgreSQL backups + account-api-postgres: + schedule: "37 23 * * *" + operation: backup + dbType: postgres + dbName: account-api_production + s3Path: account-api_postgres/account-api_production + + authenticating-proxy-postgres: + schedule: "23 23 * * *" + operation: backup + dbType: postgres + dbName: authenticating_proxy_production + s3Path: authenticating-proxy_postgres/authenticating_proxy_production + + chat-postgres: + schedule: "47 23 * * *" + operation: backup + dbType: postgres + dbName: govuk_chat_production + s3Path: chat_postgres/govuk_chat_production + + ckan-postgres: + schedule: "43 4 * * *" + operation: backup + dbType: postgres + dbName: ckan_production_eks + s3Path: ckan_postgres/ckan_production_eks + + content-block-manager-postgres: + schedule: "43 0 * * *" + operation: backup + dbType: postgres + dbName: content_block_manager_production + s3Path: content-block-manager_postgres/content_block_manager_production + + content-data-admin-postgres: + schedule: "4 23 * * *" + operation: backup + dbType: postgres + dbName: content_data_admin_production + s3Path: content-data-admin_postgres/content_data_admin_production + + content-data-api-postgres: + schedule: "35 23 * * *" + operation: backup + dbType: postgres + dbName: content_performance_manager_production + s3Path: content-data-api_postgres/content_performance_manager_production + + content-store-postgres: + schedule: "06 21 * * *" + operation: backup + dbType: postgres + dbName: content_store_production + s3Path: content-store_postgres/content_store_production + transformScript: content-store_postgres.sql + maxJobRuntimeSeconds: 86400 # 24 hours for large DB + + draft-content-store-postgres: + schedule: "16 21 * * *" + operation: backup + dbType: postgres + dbName: draft_content_store_production + s3Path: draft-content-store_postgres/draft_content_store_production + maxJobRuntimeSeconds: 86400 # 24 hours for large DB + + content-tagger-postgres: + schedule: "31 23 * * *" + operation: backup + dbType: postgres + dbName: content_tagger_production + s3Path: content-tagger_postgres/content_tagger_production + + email-alert-api-postgres: + schedule: "54 23 * * *" + operation: backup + dbType: postgres + dbName: email-alert-api_production + transformScript: email-alert-api_postgres.sql + s3Path: email-alert-api_postgres/email-alert-api_production + + places-manager-postgres: + schedule: "18 23 * * *" + operation: backup + dbType: postgres + dbName: imminence_production + s3Path: places-manager_postgres/imminence_production + + link-checker-api-postgres: + schedule: "43 23 * * *" + operation: backup + dbType: postgres + dbName: link_checker_api_production + s3Path: link-checker-api_postgres/link_checker_api_production + + local-links-manager-postgres: + schedule: "8 23 * * *" + operation: backup + dbType: postgres + dbName: local-links-manager_production + s3Path: local-links-manager_postgres/local-links-manager_production + + locations-api-postgres: + schedule: "32 23 * * *" + operation: backup + dbType: postgres + dbName: locations_api_production + s3Path: locations-api_postgres/locations_api_production + + publishing-api-postgres: + schedule: "36 21 * * *" + operation: backup + dbType: postgres + dbName: publishing_api_production + s3Path: publishing-api_postgres/publishing_api_production + transformScript: publishing-api_postgres.sql + maxJobRuntimeSeconds: 86400 # 24 hours for large DB + dbOwner: publishing-api-owner-role + + publisher-postgres: + schedule: "41 23 * * *" + operation: backup + dbType: postgres + dbName: publisher_production + s3Path: publisher_postgres/publisher_production + + service-manual-publisher-postgres: + schedule: "49 23 * * *" + operation: backup + dbType: postgres + dbName: service-manual-publisher_production + s3Path: service-manual-publisher_postgres/service-manual-publisher_production + + support-api-postgres: + schedule: "38 23 * * *" + operation: backup + dbType: postgres + dbName: support_contacts_production + s3Path: support-api_postgres/support_contacts_production + + transition-postgres: + schedule: "24 3 * * *" + operation: backup + dbType: postgres + dbName: transition_production + s3Path: transition_postgres/transition_production + + # MySQL backups + collections-publisher-mysql: + schedule: "21 23 * * *" + operation: backup + dbType: mysql + dbName: collections_publisher_production + s3Path: collections-publisher_mysql/collections_publisher_production + + release-mysql: + schedule: "11 23 * * *" + operation: backup + dbType: mysql + dbName: release_production + s3Path: release_mysql/release_production + + search-admin-mysql: + schedule: "56 23 * * *" + operation: backup + dbType: mysql + dbName: search_admin_production + s3Path: search-admin_mysql/search_admin_production + + signon-mysql: + schedule: "3 23 * * *" + operation: backup + dbType: mysql + dbName: signon_production + s3Path: signon_mysql/signon_production + + whitehall-mysql: + schedule: "28 0 * * *" + operation: backup + dbType: mysql + dbName: whitehall_production + s3Path: whitehall_mysql/whitehall_production + transformScript: whitehall_mysql.sql + maxJobRuntimeSeconds: 86400 # 24 hours for large DB + + # DocumentDB backups (shared instance, multiple databases) + publisher-documentdb: + schedule: "13 0 * * *" + operation: backup + dbType: documentdb + dbName: publisher_production + sourcePasswordSecretKey: shared-documentdb + s3Path: publisher_documentdb/publisher_production + + short-url-manager-documentdb: + schedule: "13 2 * * *" + operation: backup + dbType: documentdb + dbName: short_url_manager_production + sourcePasswordSecretKey: shared-documentdb + s3Path: short-url-manager_documentdb/short_url_manager_production + + travel-advice-publisher-documentdb: + schedule: "13 4 * * *" + operation: backup + dbType: documentdb + dbName: travel_advice_publisher_production + sourcePasswordSecretKey: shared-documentdb + s3Path: travel-advice-publisher_documentdb/travel_advice_publisher_production + + govuk-content-documentdb: + schedule: "13 6 * * *" + operation: backup + dbType: documentdb + dbName: govuk_content_production + sourcePasswordSecretKey: shared-documentdb + s3Path: govuk-content_documentdb/govuk_content_production + + govuk-assets-documentdb: + schedule: "13 8 * * *" + operation: backup + dbType: documentdb + dbName: govuk_assets_production + sourcePasswordSecretKey: shared-documentdb + transformScript: asset-manager_documentdb.js + s3Path: govuk-assets_documentdb/govuk_assets_production diff --git a/charts/db-sync/values.yaml b/charts/db-sync/values.yaml new file mode 100644 index 00000000000..a5a5a21bb7d --- /dev/null +++ b/charts/db-sync/values.yaml @@ -0,0 +1,133 @@ +# db-sync Helm Chart Configuration + +nameOverride: "" +fullnameOverride: "" + +# Suspend all CronJobs globally +suspendAllCronJobs: false + +# External Secrets configuration for AWS Secrets Manager +externalSecrets: + refreshInterval: 1h + deletionPolicy: Delete + # AWS Secrets Manager secret key containing database passwords + # Expected format: {hostname: password, ...} + secretKey: "govuk/db-backup/passwd" + +# Image configuration for the main db-sync app container +appImage: + repository: "172025368201.dkr.ecr.eu-west-1.amazonaws.com/github/alphagov/govuk/govuk-db-sync" + tag: "latest" + pullPolicy: IfNotPresent + +# Image configuration for transform sidecar containers by database engine +transformImage: + postgres: + repository: "postgres" + tag: "17-alpine" + pullPolicy: IfNotPresent + mysql: + repository: "mysql" + tag: "8.4" + pullPolicy: IfNotPresent + docdb: + repository: "mongo" + tag: "3.6" + pullPolicy: IfNotPresent + +# Resource limits and requests +appResources: + limits: + cpu: 2000m + memory: 1024Mi + requests: + cpu: 400m + memory: 512Mi + +transformResources: + limits: + cpu: 2000m + memory: 1024Mi + requests: + cpu: 400m + memory: 512Mi + +# Pod security context +podSecurityContext: + fsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + runAsGroup: 1001 + seccompProfile: + type: RuntimeDefault + +# Container security context +appSecurityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + +# Sidecar container security context for transform database engines +transformSecurityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + +# Service account configuration +serviceAccount: + create: true + name: db-sync + annotations: + eks.amazonaws.com/role-arn: "" # Overridden in app-config (Argo CD) + +# Default S3 bucket for all jobs in this release (can be overridden per-job with s3Bucket) +defaultS3Bucket: "" + +# Default database connection settings (can be overridden per-job) +defaultDbUsername: aws_db_admin + +# Global extra environment variables +extraEnv: [] + +# govukEnvironment determines which set of jobs to use from `cronjobs` below +# (production, staging, integration, etc.) +govukEnvironment: staging # Overridden in app-config (Argo CD) + +# CronJobs configuration +# Structure: +# cronjobs: +# : +# : +# schedule: "0 2 * * *" # Cron schedule (required) +# operation: backup # Operation: backup or restore (required) +# dbType: postgres # Database type: postgres, mysql, documentdb (required) +# dbName: mydb # Database name (required) +# s3Bucket: s3://my-bucket # S3 bucket override (falls back to defaultS3Bucket) +# s3Path: app/db # S3 path prefix (required for backup/restore) +# sourceDbHostname: db-prod.internal # Source DB hostname (defaults to job name) +# sourceDbPort: "5432" # Source DB port (overrides default for dbType) +# sourceDbUsername: aws_db_admin # Source DB username (overrides defaultDbUsername) +# sourceUri: postgres://... # Full source URI (overrides constructed from components) +# destDbHostname: db-staging.internal # Dest DB hostname (defaults to job name) +# destDbPort: "5432" # Dest DB port (overrides default for dbType) +# destDbUsername: aws_db_admin # Dest DB username (overrides defaultDbUsername) +# destUri: postgres://... # Full dest URI (overrides constructed from components) +# suspend: false # Suspend this job (optional) +# maxJobRuntimeSeconds: 43200 # Max runtime in seconds (default: 12 hours) +# transformScript: script.sql # Transform script filename from ConfigMap (optional, backup only) +# threads: 4 # Concurrent threads for dump/restore (optional) +# dbOwner: role_name # Database owner role (optional, restore+postgres only) +# docdbSourceDb: db_name # Source database name (optional, restore+documentdb only) +# extraEnv: [] # Job-specific env vars (optional) + +cronjobs: + production: {} + # See values-production.yaml for production job definitions + + staging: {} + # See values-staging.yaml for staging job definitions + + integration: {} + # See values-integration.yaml for integration job definitions