Merge pull request wger-project#70 from bbkz/dev

add rc.6

Author: bbkz

RELEASENOTES.md

@@ -9,8 +9,16 @@ file and update your own.
 
 First make a backup of your database and media files.
 
-If you update a existing installation you need to set the following,
-as the key already exists from the previous installation, alternativly you can delete the secret:
+Compared to the docker compose setup the helm chart takes care of the following tasks
+automatically, so don't get confused with the documentation:
+
+* creates a jwt private and public key
+* setups the powersync database
+
+If you upgrade a existing installation the jwt key already exists
+from the previous installation. You have to the jwt secret, alternatively
+you can set `update: true`, this will force the private and public key to
+be regenerated on every install and upgrade:
 
 ```yaml
 app:
@@ -22,26 +30,39 @@ app:
 
 The now unused signing key remains in the secret, but serves no purpose.
 
-This chart also takes care of the JWT and powersync setup no manual task is required there.
-
 The first start and even restarting the wger container takes a long time as we now use a
-post-install hook the helm command can timeout, use `--timeout 15m`.
+post-install hook the helm command can timeout, you have to use `--timeout 15m` on the helm
+command.
 
 * upgrade to wger 2.6
 * minor upgrade postgres to 15.18
 * minor upgrade redis to 8.8.0
 * new powersync service for offline sync for the mobile app introduced
 * new service accounts introduced for jwt autogeneration and powersync database initialization
 * JWT signing key has been removed
-* autogenerated JWT keys with a pre-install, pre-update and pre-rollback hook
-* appends to the current jwt secret
+* autogenerated JWT keys with a pre-install and pre-update hook
+* can append to the current jwt secret
 * nginx and persistent storage is now mandatory
 * nginx get's it's own deployment
 * clean up unused volume definitions
-* service and target ports changes
-* add resource settings for most containers
-* separate service settings
-* path in nginx for the static and media files changed
+* service and target ports changed
+* add resource setting possibility for most containers
+* reorganize yamls
+* path in nginx for the static and media files changed, to match docker compose setup
+* celery enabled by default
+* REFRESH_TOKEN_LIFETIME default changed from 24 to 2880
+* EXERCISE_CACHE_TTL default changed from 18000 to 2419200
+* AXES_IPWARE_PROXY_COUNT default changed from 0 to 1
+* CELERY_WORKER_CONCURRENCY added with default 4
+* CACHE_API_EXERCISES_CELERY added with default True
+* CACHE_API_EXERCISES_CELERY_FORCE_UPDATE added with default True
+* replaced .Values.app.axes.ipwareProxyCount with .Values.app.proxyCount
+* NUMBER_OF_PROXIES added with default 1 for REST Framework
+
+### Post Install Tasks
+
+* Some unused thumbnail sizes have been deleted, run `./manage.py prune-thumbnails` to delete dangling files
+* The default location for ingredient images has changed. Please run `./manage migrate-ingredient-image-paths` to migrate existing entries. Note that this is technically optional, as the old paths will continue working, but it is advised for consistency.
 
 ## 0.3.0
 

charts/wger/Chart.yaml

@@ -1,6 +1,6 @@
 ---
 apiVersion: v2
-version: 1.0.0-rc.4
+version: 1.0.0-rc.6
 appVersion: "2.6"
 name: wger
 description: A Helm chart for wger installation on Kubernetes

charts/wger/README.md

@@ -14,9 +14,9 @@ For a more productive environment you have to enable nginx as a reverse proxy. T
 
 ## Prerequisites
 
-* Kubernetes 1.15+
+* Kubernetes 1.33+
 * Helm 3.0+
-* PV infrastructure on the cluster persistence is needed
+* PV infrastructure (with ReadWriteMany) on the cluster persistence is needed
 * Ingress infrastructure for exposing the installation
 
 
@@ -41,6 +41,7 @@ There is a example of the `values.yaml` in the [example folder](/example/) with
 
 Please see the [parameters section](#parameters).
 
+**The startup phase of the wger app container is running database migrations, it is crucial that it runs properly.**
 
 ## Parameters
 
@@ -59,7 +60,7 @@ For additional configuration of the Groundhog2k's PostgreSQL and Redis charts, p
 | `app.global.annotations`      | Annotations to attach to each resource, apart from the ingress and the persistence objects | Dictionary | `{}` |
 | `app.global.replicas`         | Number of webserver instances that should be running. | Integer | `1`           |
 | `app.global.securityContext`  | Pod security context                                  | Object  | see [values.yaml](charts/wger/values.yaml) |
-
+| `app.global.proxyCount`       | Count of proxies                                      | String  | `1`           |
 
 ### Mail
 
@@ -109,20 +110,23 @@ For additional configuration of the Groundhog2k's PostgreSQL and Redis charts, p
 
 Celery requires persistent volumes.
 
-| Name                            | Description                   | Type       | Default Value     |
-|---------------------------------|-------------------------------|------------|-------------------|
-| `celery.enabled`                | Enable celery for sync        | Boolean    | `True`            |
-| `celery.annotations`            | Annotations                   | Dictionary | `{}`              |
-| `celery.replicas`               | Enable celery for sync        | Integer    | `1`               |
-| `celery.replicasWorker`         | Enable celery for sync        | Integer    | `1`               |
-| `celery.securityContext`        | Pod security context          | Object     | see [values.yaml](values.yaml) |
-| `celery.syncExercises`          | sync exercises                | Boolean    | `True`            |
-| `celery.syncImages`             | sync exercise images          | Boolean    | `True`            |
-| `celery.syncVideos`             | sync exercise videos          | Boolean    | `True`            |
-| `celery.ingredientsFrom`        | source for ingredients, possible values `WGER`,`OFF` | String  | `WGER`  |
-| `celery.flower.enabled`         | enable flower webinterface for celery | Boolean    | `False`   |
-| `celery.flower.secret.name`     | Name of the secret            | String     | `flower`          |
-| `celery.flower.secret.password` | Password for the webinterface | String     | `randAlphaNum 50` |
+| Name                             | Description                   | Type       | Default Value     |
+|----------------------------------|-------------------------------|------------|-------------------|
+| `celery.enabled`                 | Enable celery for sync        | Boolean    | `True`            |
+| `celery.annotations`             | Annotations                   | Dictionary | `{}`              |
+| `celery.replicas`                | Enable celery for sync        | Integer    | `1`               |
+| `celery.replicasWorker`          | Enable celery for sync        | Integer    | `1`               |
+| `celery.workerConcurrency`       | Set to one if using sqlite    | Integer    | `4`               |
+| `celery.securityContext`         | Pod security context          | Object     | see [values.yaml](values.yaml) |
+| `celery.syncExercises`           | sync exercises                | Boolean    | `True`            |
+| `celery.syncImages`              | sync exercise images          | Boolean    | `True`            |
+| `celery.syncVideos`              | sync exercise videos          | Boolean    | `True`            |
+| `celery.warmupExercisesCache`    | task to warmup cache 1w       | Boolean    | `True`            |
+| `celery.warmupExercisesCacheAll` | warmup all exercises          | Boolean    | `True`            |
+| `celery.ingredientsFrom`         | source for ingredients, possible values `WGER`,`OFF` | String  | `WGER`  |
+| `celery.flower.enabled`          | enable flower webinterface for celery | Boolean    | `False`   |
+| `celery.flower.secret.name`      | Name of the secret            | String     | `flower`          |
+| `celery.flower.secret.password`  | Password for the webinterface | String     | `randAlphaNum 50` |
 
 
 ## JWT
@@ -145,7 +149,6 @@ Celery requires persistent volumes.
 | `app.axes.lockoutParameters`         | List (comma separated string) | String | `"ip_address"` |
 | `app.axes.failureLimit`              | Limit of failed auth          | String | `10` |
 | `app.axes.cooloffTime`               | in Minutes                    | String | `30` |
-| `app.axes.ipwareProxyCount`          | Count of proxies              | String | `0` |
 | `app.axes.ipwareMetaPrecedenceOrder` | Proxy header magnitude | List (comma separated string) | `"HTTP_X_FORWARDED_FOR,REMOTE_ADDR"` |
 
 

charts/wger/templates/NOTES.txt

@@ -19,22 +19,4 @@ Get the postgres user passwords:
 kubectl -n {{ .Release.Namespace }} get secret {{ .Release.Name }}-postgres -o jsonpath='{.data.USERDB_PASSWORD}' | base64 -d
 ```
 
-After the first installation you need to generate the database setting for powersync, this is used from the wger companion app
-to authenticate for offline synchronisation:
-
-```
-export POD=$(kubectl -n {{ .Release.Namespace }} get pods -l "app.kubernetes.io/name={{ .Release.Name }}-app" -o jsonpath="{.items[0].metadata.name}")
-kubectl -n {{ .Release.Namespace }} exec -it pod/$POD -c wger -- ./manage.py setup-powersync-storage
-```
-
-{{- end }}
-
-{{- $manipulatejwt := (include "manipulatejwt" .) }}
-{{- if eq $manipulatejwt "doit" }}
-The jwt keys have been changed you need to restart the deployment to activate the new keys:
-
-```
-kubectl -n {{ .Release.Namespace }} rollout restart deploy {{ .Release.Name }}-app {{ .Release.Name }}-powersync {{ .Release.Name }}-nginx {{ .Release.Name }}-celery {{ .Release.Name }}-celery-worker
-```
-
 {{- end }}

charts/wger/templates/_helpers.tpl

@@ -44,7 +44,7 @@ environment:
   - name: DJANGO_CACHE_TIMEOUT
     value: {{ int .Values.app.django.cache.timeout | default "1296000" | quote }}
   - name: EXERCISE_CACHE_TTL
-    value: "18000"
+    value: "2419200"
   # django general
   {{- if .Values.ingress.enabled }}
   - name: SITE_URL
@@ -63,6 +63,12 @@ environment:
     value: "False"
   - name: DJANGO_MEDIA_ROOT
     value: "/home/wger/media"
+  # Django Rest Framework
+  # The number of proxies in front of the application. In the default configuration
+  # only nginx is. Change as approtriate if your setup differs. Also note that this
+  # is only used when throttling API requests.
+  - name: NUMBER_OF_PROXIES
+    value: {{ int .Values.app.global.proxyCount | default "1" | quote }}
   # axes
   - name: AXES_ENABLED
   {{- if .Values.app.axes.enabled }}
@@ -77,7 +83,7 @@ environment:
   - name: AXES_COOLOFF_TIME
     value: {{ int .Values.app.axes.cooloffTime | default "30" | quote }}
   - name: AXES_IPWARE_PROXY_COUNT
-    value: {{ int .Values.app.axes.ipwareProxyCount | default "0" | quote }}
+    value: {{ int .Values.app.global.proxyCount | default "1" | quote }}
     # @todo bad default, use the default from axes REMOTE_ADDR only
   - name: AXES_IPWARE_META_PRECEDENCE_ORDER
     value: {{ .Values.app.axes.ipwareMetaPrecedenceOrder | default "HTTP_X_FORWARDED_FOR,REMOTE_ADDR" | quote }}
@@ -87,7 +93,7 @@ environment:
   - name: ACCESS_TOKEN_LIFETIME
     value: {{ int .Values.app.jwt.accessTokenLifetime | default "10" | quote }}
   - name: REFRESH_TOKEN_LIFETIME
-    value: {{ int .Values.app.jwt.refreshTokenLifetime | default "24" | quote }}
+    value: {{ int .Values.app.jwt.refreshTokenLifetime | default "2880" | quote }}
   # gunicorn settings
   - name: WGER_USE_GUNICORN
     value: "True"
@@ -126,6 +132,12 @@ environment:
     value: {{ .Values.celery.syncVideos | default "True" | quote }}
   - name: DOWNLOAD_INGREDIENTS_FROM
     value: {{ .Values.celery.ingredientsFrom | default "WGER" | quote }}
+  - name: CELERY_WORKER_CONCURRENCY
+    value: {{ .Values.celery.workerConcurrency | default "4" | quote }}
+  - name: CACHE_API_EXERCISES_CELERY
+    value: {{ .Values.celery.warmupExercisesCache | default "True" | quote }}
+  - name: CACHE_API_EXERCISES_CELERY_FORCE_UPDATE
+    value: {{ .Values.celery.warmupExercisesCacheAll | default "True" | quote }}
   {{- end }}
 {{- end }}
 
@@ -162,7 +174,7 @@ environment:
 
 {{/*
  database settings
- used for wger-app and celery containers
+ used for wger-app, celery and powersync containers
 */}}
 {{- define "database.settings" }}
   - name: DJANGO_DB_HOST
@@ -211,6 +223,8 @@ environment:
 
 {{/*
  powersync settings
+ requires database.settings
+ used for wger-app, celery and powersync containers
 */}}
 {{- define "powersync.settings" }}
   - name: JWT_PRIVATE_KEY
@@ -285,17 +299,23 @@ environment:
 {{- end }}
 
 {{/*
- initContainer web command
- used for celery containers
+ initContainer app command
+ used for celery and powersync containers
 */}}
-{{- define "initContainer.web.command" }}
+{{- define "initContainer.app.command" }}
 {{- $dbhost := .Values.app.django.existingDatabase.host | default (print .Release.Name "-postgres") | quote }}
 {{- $dbport := .Values.app.django.existingDatabase.port | default .Values.postgres.service.port | int | quote }}
+{{- $svcport := .Values.app.service.port | default 8000 | int | quote }}
 - /bin/sh
 - -c
-- until nc -zvw10 {{ $dbhost }} {{ $dbport }}; do echo "Waiting for postgres service ({{ $dbhost }}:{{ $dbport }}) "; sleep 2; done &&
+# sleep 35; wait for terminationGracePeriodSeconds of the wger-app container
+# this prevents using the wger-app container which are in the process of termination
+# @todo find a better solution to prevent starting powersync
+# on upgrades before the new wger-app container is ready
+# -> this may be only relevant when upgrading from a "non" powersync setup
+- sleep 35; until nc -zvw10 {{ $dbhost }} {{ $dbport }}; do echo "Waiting for postgres service ({{ $dbhost }}:{{ $dbport }}) "; sleep 2; done &&
   until nc -zvw10 {{ .Release.Name }}-redis {{ .Values.redis.service.serverPort }}; do echo "Waiting for redis service ({{ .Release.Name }}-redis:{{ .Values.redis.service.serverPort }})"; sleep 2; done &&
-  until wget --spider http://{{ .Release.Name }}-http:80; do echo "Waiting for nginx service ({{ .Release.Name }}-http:80)"; sleep 2; done
+  until nc -zvw10 {{ .Release.Name }}-app {{ $svcport }}; do echo "Waiting for wger app service ({{ .Release.Name }}-app:{{ $svcport }})"; sleep 2; done
 {{- end }}
 
 {{/*

charts/wger/templates/configmap-powersync.yaml

@@ -76,6 +76,14 @@ data:
       # JWKS audience
       audience: ["powersync"]
 
+    # System-level configuration options
+    system:
+      # Service logging configuration
+      logging:
+        # Log level for the service logs
+        level: info #  'silly', 'debug', 'verbose', 'http', 'info', 'warn', 'error'
+        format: text # 'json' or 'text'
+
   sync_rules.yaml: |
     # Note that changes to this file are not watched.
     # The service needs to be restarted for changes to take effect.

charts/wger/templates/deployment-celery.yaml

@@ -8,6 +8,9 @@ metadata:
     app.kubernetes.io/name: {{ .Release.Name }}
 spec:
   replicas: {{ .Values.celery.replicas }}
+  strategy:
+    # kills all old Pods first
+    type: Recreate
   selector:
     matchLabels:
       app.kubernetes.io/name: {{ .Release.Name }}-celery
@@ -16,6 +19,8 @@ spec:
       labels:
         app.kubernetes.io/name: {{ .Release.Name }}-celery
       annotations:
+        # restart on upgrade
+        rollme: {{ randAlphaNum 5 | quote }}
     {{- with .Values.celery.annotations }}
       {{- range $key, $value := . }}
         {{ $key }}: {{ $value | quote}}
@@ -149,8 +154,9 @@ spec:
       initContainers:
         - name: init-container
           image: docker.io/busybox:latest
+          imagePullPolicy: {{ .Values.app.global.image.PullPolicy }}
           command:
-          {{- include "initContainer.web.command" . | indent 10 }}
+          {{- include "initContainer.app.command" . | indent 10 }}
       volumes:
         - name: wger-celery-beat
           persistentVolumeClaim:
@@ -171,6 +177,9 @@ metadata:
   {{- end }}
 spec:
   replicas: {{ .Values.celery.replicasWorker }}
+  strategy:
+    # kills all old Pods first
+    type: Recreate
   selector:
     matchLabels:
       app.kubernetes.io/name: {{ .Release.Name }}-celery-worker
@@ -179,6 +188,8 @@ spec:
       labels:
         app.kubernetes.io/name: {{ .Release.Name }}-celery-worker
       annotations:
+        # restart on upgrade
+        rollme: {{ randAlphaNum 5 | quote }}
     {{- with .Values.celery.annotations }}
       {{- range $key, $value := . }}
         {{ $key }}: {{ $value | quote}}
@@ -239,6 +250,15 @@ spec:
                   key: "password"
             {{- end }}
           command: ["/start-worker"]
+          readinessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - celery -A wger inspect ping
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: wger-media
               mountPath: /home/wger/media
@@ -252,8 +272,9 @@ spec:
       initContainers:
       - name: init-container
         image: docker.io/busybox:latest
+        imagePullPolicy: {{ .Values.app.global.image.PullPolicy }}
         command:
-        {{- include "initContainer.web.command" . | indent 10 }}
+        {{- include "initContainer.app.command" . | indent 10 }}
       volumes:
         - name: wger-media
           persistentVolumeClaim:

charts/wger/templates/deployment-powersync.yaml

@@ -7,6 +7,9 @@ metadata:
     app.kubernetes.io/name: {{ .Release.Name }}
 spec:
   replicas: {{ .Values.powersync.replicas }}
+  strategy:
+    # kills all old Pods first
+    type: Recreate
   selector:
     matchLabels:
       app.kubernetes.io/name: {{ .Release.Name }}-powersync
@@ -15,6 +18,8 @@ spec:
       labels:
         app.kubernetes.io/name: {{ .Release.Name }}-powersync
       annotations:
+        # restart on upgrade
+        rollme: {{ randAlphaNum 5 | quote }}
     {{- with .Values.powersync.annotations }}
       {{- range $key, $value := . }}
         {{ $key }}: {{ $value | quote}}
@@ -71,8 +76,9 @@ spec:
       initContainers:
         - name: init-container
           image: docker.io/alpine:latest
+          imagePullPolicy: {{ .Values.powersync.image.PullPolicy }}
           command:
-          {{- include "initContainer.web.command" . | indent 10 }}
+          {{- include "initContainer.app.command" . | indent 10 }}
       volumes:
         - name: powersync-conf
           configMap: