|
| 1 | +# Upgrade Temporal Helm chart |
| 2 | + |
| 3 | +#### 1. Sync master branch |
| 4 | + |
| 5 | +On `master` branch just push sync-fork |
| 6 | + |
| 7 | +#### 2. Merge the master branch into fm-devops |
| 8 | + |
| 9 | +Open pull request to merge the `master` branch into `fm-devops`. When resolving conflicts, take into account the following changes that were done to `fm-devops` and that shouldn't be lost: |
| 10 | + |
| 11 | +- [https://github.com/fairmoney/temporal-helm-charts/pull/10/files](https://github.com/fairmoney/temporal-helm-charts/pull/10/files) |
| 12 | +- [https://github.com/fairmoney/temporal-helm-charts/pull/4/files](https://github.com/fairmoney/temporal-helm-charts/pull/4/files) |
| 13 | +- [https://github.com/fairmoney/temporal-helm-charts/pull/18](https://github.com/fairmoney/temporal-helm-charts/pull/18) |
| 14 | + |
| 15 | +Also, on `fm-devops` branch, there are some additional files specific for Fairmoney deployments: |
| 16 | + |
| 17 | +- charts/temporal/templates/certificates.yaml |
| 18 | +- charts/temporal/templates/external-secrets.yaml |
| 19 | +- charts/temporal/README.md |
| 20 | + |
| 21 | +After resolving the conflicts, when prompted, choose to write the changes to a new branch (e.g. `master-upgrade`). Then, a new PR from `master-upgrade` into `fm-devops` will be created. |
| 22 | + |
| 23 | +### 3. Mirror the new `admin-tools` and `ui` images to service ECR |
| 24 | + |
| 25 | +Mirror the tags used by the new temporal-helm-chart version. |
| 26 | + |
| 27 | +### 4. Build new Temporal server image |
| 28 | + |
| 29 | +Follow [these](#upgrade-temporal-server-image) steps to upgrade the temporal server image to the release used by the new temporal-helm-chart version. |
| 30 | + |
| 31 | +## ECR Repositories inventory: |
| 32 | + |
| 33 | +Upstream images are mirrored to: |
| 34 | +``` |
| 35 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-admin-tools |
| 36 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-server |
| 37 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-ui |
| 38 | +``` |
| 39 | + |
| 40 | +Fairmoney custom built images are pushed to: |
| 41 | +``` |
| 42 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-server/server |
| 43 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-server/admin-tools |
| 44 | +409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-server/auto-setup |
| 45 | +``` |
| 46 | +In the Hosted Temporal deployment we currently we use upstream images, except for the server image where we use the Fairmoney one (`409558311582.dkr.ecr.eu-west-1.amazonaws.com/temporalio-server/server`). |
| 47 | + |
| 48 | +# Upgrade Temporal server image |
| 49 | + |
| 50 | +Fairmoney temporal server image build process relies on the following repositories: |
| 51 | + |
| 52 | +- [fairmoney/temporal](https://github.com/fairmoney/temporal) |
| 53 | +- [fairmoney/temporal-docker-build](https://github.com/fairmoney/temporal-docker-builds) |
| 54 | +- [fairmoney/temporal-features](https://github.com/fairmoney/temporal-features) |
| 55 | + |
| 56 | +These repositories have custom workflow changes to fit in the Fairmoney CI pipeline. `fairmoney/temporal` also has the fm-authorizer and fm-claim-mapper custom modules. These changes are in `fm/*` branches. |
| 57 | + |
| 58 | +`fairmoney/temporal` is where the temporal server code reside and the final image is built based on the `fm/main` branch. |
| 59 | + |
| 60 | +Here is how the upgrade process to newer temporal release from upstream would look like: |
| 61 | + |
| 62 | +#### Lets take the example of merging upstream release/v1.25.x into fm/main |
| 63 | + |
| 64 | +``` |
| 65 | +git clone https://github.com/fairmoney/temporal.git # Clone the Fairmoney organization repository |
| 66 | +cd temporal |
| 67 | +git remote add upstream https://github.com/temporalio/temporal.git # Add the upstream repository as a remote |
| 68 | +git fetch upstream release/v1.25.x # Fetch the specific branch from the upstream repo |
| 69 | +git checkout -b release/v1.25.x upstream/release/v1.25.x # Checkout to the fetched branch (optional) |
| 70 | +git cherry-pick a027abccc2dafcdf97e0a0b86fba577194e791ef # Cherry pick the commit that contains the fm claim-mapper and authorizer |
| 71 | +git push origin release/v1.25.x:fm/release/v1.25.x # Push the branch to the organization repository |
| 72 | +
|
| 73 | +git checkout -b fm/ci-workflows # Checkout to a new branch that will include the CI changes |
| 74 | +# Depending on the upstream evolution either cherry pick commit c1f0636d48bff9408262028cc4abf3ce40defddc and commit f3664539c6494813b30127f412a6226b5f593454 or manually change all the needed places to make CI working (see https://github.com/fairmoney/temporal/pull/7/files) |
| 75 | +git push origin fm/ci-workflows |
| 76 | +# On the Fairmoney organization repository create pull request from fm/ci-workflows into fm/release/v1.25.x (see https://github.com/fairmoney/temporal/pull/6) |
| 77 | +# Once there are no CI errors (not neccesarily failed tests, there may be some units failed units tests), merge the pull request opened to fm/release/v1.25.x |
| 78 | +# Now back on your local branch |
| 79 | +git pull --rebase origin fm/release/v1.25.x # Pull changes merged in the organization repository |
| 80 | +git push origin release/v1.25.x:fm/main --force # Ovewrite fm/main branch |
| 81 | +``` |
| 82 | + |
| 83 | +When pull request is merged to `fm/main`, in the `fairmoney/temporal` repository, the following workflows are triggered: `github/workflows/run-tests.yml`, `.github/workflows/features-integration.yml` **AND** `.github/workflows/trigger-publish.yml`. `.github/workflows/trigger-publish.yml` will invoke `update-submodules` (from `fairmoney/temporal-docker-builds`) to update all submodules to their latest commit. |
| 84 | + |
| 85 | +In `fairmoney/temporal-docker-builds`, `.github/workflows/update-submodules.yml` workflow will push a commit to `fm/main` branch. As a result of the push, the following workflow will be triggered: `.github/workflows/docker.yml` |
| 86 | + |
| 87 | +`github/workflows/docker.yml` builds Docker images and publishes to service account's ECR in the following repos: |
| 88 | +- temporalio-server/admin-tools |
| 89 | +- temporalio-server/auto-setup |
| 90 | +- temporalio-server/server |
| 91 | + |
| 92 | + |
| 93 | +After the images are published, pick up the `sha` from service account's ECR repository `temporalio-server/server` of the latest image pushed (just the 7 characters, without the sha prefix). |
| 94 | +Now, to release a new FM temporal server image, from `fairmoney/temporal-docker-builds` manually trigger the following workflow: **Release Temporal Images** (`.github/workflows/release-temporal.yml`) as shown in the image: |
| 95 | +- make sure the branch selected is `fm/main` |
| 96 | +- commit sha picked from ECR |
| 97 | +- set the tag for the image, make sure it starts with **v** (e.g. v1.25-fairmoney) |
| 98 | + |
| 99 | +In a similar way, trigger the **Release Temporal Images to FMB** (`.github/workflows/release-temporal-fmb.yml`) to publish the image to FMB Infra ACR. |
| 100 | + |
| 101 | + |
| 102 | + |
| 103 | +# Build cross cluster CA trust using Pushsecret from ESO and Bundle from trust-manager |
| 104 | + |
| 105 | +Using `Object.kubernetes.crossplane.io` and `watch` feature from provider-kubernetes, a `PushSecret` resource is created each time `temporal-ca-secret` in `cert-manager` namespace has a new `resourceVersion` (meaning the root CA was renewed or changed for some reason). When this happens, |
| 106 | + |
| 107 | +- [AWS implementation] `PushSecret` adds a new key/value in `temporal/non-prod-root-ca-list` secret from AWS Secrets Manager. The key is the `resourceVersion` of the secret and the value is the new root CA (`tls.crt`). |
| 108 | +- [Azure implementation] `PushSecret` adds a new secret in Azure Key Vault. The secret name is the new `resourceVersion` (8-digit name) and the value is the new root CA in PEM format. |
| 109 | + |
| 110 | +Then, using `externalsecret` from ESO, the Secrets Manager secret (or the Azure Key Vault secrets) are imported into `temporal-root-ca-list` secret in `cert-manager` namespace. |
| 111 | + |
| 112 | +Lastly, a `Bundle.trust.cert-manager.io` resource takes all keys from `temporal-root-ca-list` secret and the local `tls.crt` key from `temporal-ca-secret` and bundles them into a trust bundle written to `temporal-trust-bundle` secret in all namespaces. |
| 113 | + |
| 114 | +Detailed diagram [here](https://www.notion.so/fairmoney/Cross-cluster-trust-for-Temporal-root-CA-using-trust-manager-1967f8f1d6868093a4d3f7ed56ba734e). |
0 commit comments