diff --git a/docs/user-guide/ci-cd-integration.md b/docs/user-guide/ci-cd-integration.md index 1a7b6b33..5462271f 100644 --- a/docs/user-guide/ci-cd-integration.md +++ b/docs/user-guide/ci-cd-integration.md @@ -11,69 +11,405 @@ description: Instructions for automating deployments using nctl CLI in CI/CD pip # CI/CD Integration -## Automate Deployments +This guide explains how you can setup a CI/CD pipeline to automate deployments. We'll provide examples for GitHub Actions +and Semaphore. Furthermore, we explain how you could setup review apps. + +## Prerequisites + +Before setting up CI/CD, you'll need: + +- An existing Deploio application linked to a Git repository +- An API Service Account (ASA) for authentication (see [here](#create-an-api-service-account)) + +## Continuous Integration + +### Linters and tests + +The following workflow shows an example CI pipeline with GitHub Actions. It runs linting and tests in parallel. +See [Continuous Deployment](#continuous-deployment) for details on how to setup the deployment step. + +```yaml +name: CI + +on: + push: + branches: [main, develop] + pull_request: + +jobs: + lint: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + - name: Install dependencies + run: npm ci + - name: Run linters + run: npm run lint + + test: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + - name: Install dependencies + run: npm ci + - name: Run tests + run: npm test +``` + +### Review Apps + +Review apps are currently not supported yet in the Deploio cockpit. However, we provide you the commands that you can use +to automate the setup within your CI/CD pipeline. This makes it fully customizable to your use case. -Currently, when we link the GitHub repository and target revision for the application, the application will automatically re-deploy on branch changes. +The integration requires two local scripts. +- `bin/deploy_review_app` to copy the template app and to point the new app to your feature branch +- `bin/delete_review_app` to clean up the created review app and it's accessories (DB, Redis, etc.) -If this is sufficient, the application can remain with this setup pointing to a static branch. +These scripts interact with Deploio in the following way. -Otherwise, we can automate deployments using scripts and integrate deployments with CI/CD pipelines. +```mermaid +sequenceDiagram + actor Dev as Developer + participant CI as CI / CLI + participant Depl as Deploio -##### Install `nctl` in CI pipelines. + Note over Dev, Depl: Deploy -We will need to configure the CI process to install and authenticate the `nctl` CLI. + Dev->>CI: Open pull request + CI->>Depl: Copy template app (if missing) + CI->>Depl: Create accessories (DB, etc.) (if missing) + CI->>Depl: Set review app git revision & ENV variables + Depl-->>Dev: Review app URL -For this, we will need to set the `NCTL_API_TOKEN` variable and `NCTL_ORGANIZATION` variable on the CI environment. We can use the API Service Account (ASA) to create a token. See the next section for instructions on this. + Note over Dev, Depl: Cleanup (on PR close/merge) + Dev->>CI: Close/merge PR + CI->>Depl: Delete review app & accessories +``` + +We recommend that you consider the scripts as "templates". You can copy them to your project and customize them as you want. +You might want to replace the Postgres commands with MySQL, or create additional Redis services for example. -Once this is done, we can install and authenticate `nctl` as below. +`bin/deploy_review_app` +```bash +#!/usr/bin/env bash + +set -e + +BRANCH_NAME="${1:-$(git branch --show-current)}" + +case "$BRANCH_NAME" in + main|master|develop) echo "Skipping deploy for branch $BRANCH_NAME"; exit 0 ;; +esac + +UNIQUE_SUFFIX=$(echo -n "$BRANCH_NAME" | sha1sum | cut -c1-8) +REVIEW_APP_NAME="review-app-${UNIQUE_SUFFIX}" + +echo "Deploying $REVIEW_APP_NAME for branch $BRANCH_NAME..." + +# Copy App if missing +echo "Checking App..." +nctl get app "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" >/dev/null 2>&1 || \ +nctl copy app "$DEPLOIO_TEMPLATE_APP" -p "$DEPLOIO_PROJECT" --start --target-name="$REVIEW_APP_NAME" + +# Init Economy DB if missing +echo "Checking Postgres DB..." +nctl get postgresdatabase "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" >/dev/null 2>&1 || \ +nctl create postgresdatabase "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" --location=nine-es34 --backup-schedule=disabled --wait +DATABASE_URL=$(nctl get postgresdatabase "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" --print-connection-string) + +# Set git revision and ENVs +echo "Updating App..." +nctl update app "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" \ + --git-revision="${BRANCH_NAME}" \ + --env="DATABASE_URL=${DATABASE_URL}" \ + --skip-repo-access-check \ + $ENV_FLAGS + +# Output app URL +APP_URL=$(nctl get app "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" -o json | jq -r '.status.atProvider.defaultURLs[0] // empty') +echo "app-name=$REVIEW_APP_NAME" +if [ -n "$APP_URL" ]; then + echo "app-url=$APP_URL" +fi + +# Wait for release to succeed +echo "Waiting for release..." +for i in $(seq 1 60); do + STATUS=$(nctl get releases -a "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" -o json | jq -r '.[0].status.atProvider.releaseStatus // empty') + case "$STATUS" in + available|superseded) echo "release-status=success"; exit 0 ;; + failed) echo "release-status=error"; exit 1 ;; + esac + sleep 5 +done +echo "release-status=error" +exit 1 +``` +`bin/delete_review_app` ```bash -sudo apt-get update && sudo apt-get install nctl -nctl auth login +#!/usr/bin/env bash + +set -e + +BRANCH_NAME="${1:-$(git branch --show-current)}" + +case "$BRANCH_NAME" in + main|master|develop) echo "Skipping delete for branch $BRANCH_NAME"; exit 0 ;; +esac + +UNIQUE_SUFFIX=$(echo -n "$BRANCH_NAME" | sha1sum | cut -c1-8) +REVIEW_APP_NAME="review-app-${UNIQUE_SUFFIX}" + +echo "Deleting $REVIEW_APP_NAME for branch $BRANCH_NAME..." + +nctl delete app "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" --force --wait || true +nctl delete postgresdatabase "$REVIEW_APP_NAME" -p "$DEPLOIO_PROJECT" --force --wait || true + +echo "Deleted $REVIEW_APP_NAME" ``` -##### Issue tokens for secure deployment automation. +::: info +You might be wondering why we force delete the resources here. This is in place to skip the deletion confirmation. The `--force` +flag still respects the deletion protection mechanism on your main app. You could limit the risk by enabling deletion protection +for your production environments. +::: + +#### GitHub Actions + +To automate the review app creation, you could setup e.g. a GitHub Actions workflow. The following workflow +runs automatically as soon as you open a PR, mark it ready for review or close it. In addition, it can also be triggered +manually in the Actions tab. Once the latest release of the review app is successful, it will display the deployment in your +pull request. + +```yaml +name: Review App + +on: + pull_request: + types: [opened, ready_for_review, closed] + workflow_dispatch: + +jobs: + deploy: + if: github.event.action != 'closed' && !github.event.pull_request.draft + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install nctl + run: | + echo 'deb [trusted=yes] https://repo.nine.ch/deb/ /' | sudo tee /etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get update -qqo Dir::Etc::sourcelist=/etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get install -qq nctl + + - name: Authenticate nctl + run: | + nctl auth login \ + --api-client-id=${{ secrets.NCTL_API_CLIENT_ID }} \ + --api-client-secret=${{ secrets.NCTL_API_CLIENT_SECRET }} \ + --organization=${{ secrets.NCTL_ORGANIZATION }} + + - name: Create deployment + id: deployment + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + DEPLOYMENT_ID=$(gh api repos/${{ github.repository }}/deployments \ + --input - --jq '.id' \ + <<< '{"ref":"${{ github.head_ref || github.ref_name }}","environment":"review-app","auto_merge":false,"required_contexts":[]}') + echo "id=$DEPLOYMENT_ID" >> "$GITHUB_OUTPUT" + + gh api repos/${{ github.repository }}/deployments/$DEPLOYMENT_ID/statuses \ + -f state=pending \ + -f description="Deploying review app..." + + - name: Deploy review app + id: deploy + env: + DEPLOIO_PROJECT: my-project + DEPLOIO_TEMPLATE_APP: main + run: | + OUTPUT=$(bin/deploy_review_app "${{ github.head_ref || github.ref_name }}") + echo "$OUTPUT" + echo "$OUTPUT" | grep "^app-url=" >> "$GITHUB_OUTPUT" || true + echo "$OUTPUT" | grep "^app-name=" >> "$GITHUB_OUTPUT" || true + + - name: Deployment success + if: success() + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api repos/${{ github.repository }}/deployments/${{ steps.deployment.outputs.id }}/statuses \ + -f state=success \ + -f environment_url="${{ steps.deploy.outputs.app-url }}" \ + -f description="Review app deployed" + + - name: Deployment error + if: failure() && steps.deployment.outputs.id + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api repos/${{ github.repository }}/deployments/${{ steps.deployment.outputs.id }}/statuses \ + -f state=error \ + -f description="Review app deployment failed" + + cleanup: + if: github.event.action == 'closed' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install nctl + run: | + echo 'deb [trusted=yes] https://repo.nine.ch/deb/ /' | sudo tee /etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get update -qqo Dir::Etc::sourcelist=/etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get install -qq nctl + + - name: Authenticate nctl + run: | + nctl auth login \ + --api-client-id=${{ secrets.NCTL_API_CLIENT_ID }} \ + --api-client-secret=${{ secrets.NCTL_API_CLIENT_SECRET }} \ + --organization=${{ secrets.NCTL_ORGANIZATION }} + + - name: Delete review app + env: + DEPLOIO_PROJECT: my-project + run: bin/delete_review_app "${{ github.head_ref || github.ref_name }}" + + - name: Deactivate deployment + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + DEPLOYMENT_ID=$(gh api "repos/${{ github.repository }}/deployments?ref=${{ github.head_ref || github.ref_name }}&environment=review-app" \ + --jq '.[0].id' 2>/dev/null || true) + + if [ -n "$DEPLOYMENT_ID" ] && [ "$DEPLOYMENT_ID" != "null" ]; then + gh api repos/${{ github.repository }}/deployments/$DEPLOYMENT_ID/statuses \ + -f state=inactive \ + -f description="Review app deleted" + fi +``` + +::: info +Replace `my-project` with your Deploio project name and `main` with the name of your template application. The template app is copied to create each review app, so it should be configured with your desired defaults. +::: + +## Continuous Deployment + +### Automate deployments + +When you link a GitHub repository and target branch to your Deploio application, the application automatically +re-deploys whenever you push a change to that branch. If this is sufficient for your workflow, no additional CI/CD setup +is needed. + +For more control — such as deploying only after tests pass, deploying specific commits, or getting deployment status +feedback — you can automate deployments using `nctl` in your CI/CD pipeline. The following sections explain how to do this. + +### Revision-based deployments + +The concept behind revision-based deployments is pretty simple. Instead of pointing your Deploio app to a branch, we +point it to a specific git revision (commit SHA). This way, the Deploio build is triggered exactly +for the specified state of the repository. + +#### Flow -For extra security, we should create a token for the CI to use. This token can be created using the API Service Account (ASA) and can be used to authenticate the `nctl` CLI. +The following sequence diagram summarizes the revision-based deployment flow: -We create a token with: +```mermaid +sequenceDiagram + participant Dev as Developer + participant CI as CI/CD Pipeline + participant Depl as Deploio + + Dev->>CI: Push code changes + CI->>CI: optional: Run checks (tests, linters, etc.) + CI->>Depl: Authenticate with nctl + CI->>Depl: Update app to point to specific git revision + Depl->>Depl: Trigger build and release + CI->>Depl: Poll build and release status + Depl-->>CI: Build and release status +``` + +#### Install `nctl` + +Configure your CI process to install and authenticate the `nctl` CLI. You'll need the following secrets set in your CI environment: + +| Secret | Description | +|---|---| +| `NCTL_API_CLIENT_ID` | The client ID from your API Service Account | +| `NCTL_API_CLIENT_SECRET` | The client secret from your API Service Account | +| `NCTL_ORGANIZATION` | Your Nine organization name | + +Install and authenticate `nctl`: + +```bash +# Install nctl (Debian/Ubuntu) +echo 'deb [trusted=yes] https://repo.nine.ch/deb/ /' | sudo tee /etc/apt/sources.list.d/repo.nine.ch.list +sudo apt-get update -qqo Dir::Etc::sourcelist=/etc/apt/sources.list.d/repo.nine.ch.list +sudo apt-get install -qq nctl +``` + +#### Create an API Service Account + +Create an API Service Account (ASA) so your CI pipeline can authenticate without using personal credentials. + +Create a new ASA: ```bash nctl create asa {token_name} ``` -and then we can view the token using: +View the token: ```bash nctl get apiserviceaccount {token_name} --print-token ``` -We can then set this as the `DEPLOIO_API_TOKEN` on the CI environment. +Authenticate `nctl` using the ASA credentials: -##### Trigger deployments on code (ref) updates. +```bash +nctl auth login \ + --api-client-id=$NCTL_API_CLIENT_ID \ + --api-client-secret=$NCTL_API_CLIENT_SECRET \ + --organization=$NCTL_ORGANIZATION +``` + +⚠️ Store the client ID and secret as secrets in your CI environment. Never commit them to your repository. + +#### Update app git revision + +Set the following environment variables in your CI pipeline: + +| Variable | Description | +|---|---| +| `DEPLOIO_PROJECT` | Your Deploio project name | +| `DEPLOIO_APP_NAME` | Your Deploio application name | -To trigger deployments on the CI, we also need to set the `DEPLOIO_PROJECT` and `DEPLOIO_APP_NAME` environment variables on the CI. We can then use the `nctl` CLI to "update" the application with the new git revision. +Then use `nctl` to update the application to a specific git revision: ```bash nctl update app $DEPLOIO_APP_NAME \ --project $DEPLOIO_PROJECT \ --git-revision=$(git rev-parse HEAD) \ - --build-env="RUBY_VERSION=$(cat .ruby-version)" \ --skip-repo-access-check ``` -#### Deployment feedback on the CI +#### Poll build and release status -We can also add a script to check Deploio for the build and release status, and provide feedback to the CI. +After triggering a deployment, you can poll Deploio for the build and release status. This lets your CI pipeline report whether the deployment succeeded or failed. -Below is an example of this in Ruby, but this can be adapted as required. +Below is an example script in Ruby that checks the build and release status. You can adapt this to any language. ```ruby require 'yaml' require 'open3' -TIMEOUT_IN_SECONDS = 300 # 5 minutes (build & release should not take more than 5 minutes each) +TIMEOUT_IN_SECONDS = 300 INTERVAL_IN_SECONDS = 30 PROJECT = ENV['DEPLOIO_PROJECT'] @@ -126,7 +462,6 @@ end puts "(1/2) Checking build status for revision #{REVISION}..." -# Polling mechanism for build status elapsed = 0 build = nil while elapsed < TIMEOUT_IN_SECONDS @@ -157,7 +492,6 @@ if elapsed >= TIMEOUT_IN_SECONDS || build.nil? exit 1 end -# If the build is successful, proceed to check the release status build_name = build.dig('metadata', 'name') puts "(2/2) Checking release status for build #{build_name}..." @@ -169,10 +503,10 @@ while elapsed < TIMEOUT_IN_SECONDS if release case release_status(release) when 'available' - puts "Release is available for build #{build_name}. App has been successfully deployed ✅" + puts "Release is available for build #{build_name}. Deployment successful." exit 0 when 'failed' - puts "Release failed for build #{build_name} ❌" + puts "Release failed for build #{build_name}." exit 1 else puts "Release status is #{release_status(release)}, waiting..." @@ -189,25 +523,97 @@ puts "Release check timed out after #{TIMEOUT_IN_SECONDS} seconds." exit 1 ``` -We want to trigger this script after we trigger the app deployment. With our Ruby script, we would add the below to our CI process script. +Run the script after triggering the deployment: ```bash ruby bin/check_deploio_deployment_status.rb ``` -#### Summary - -Once the above has been put in to place, we should see the following process: - -✅ The tests run in line with your usual CI process. +### GitHub Actions + +Here's an example GitHub Actions workflow that deploys your application after tests pass: + +```yaml +name: Deploy + +on: + push: + branches: [main] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run tests + run: | + # Add your test commands here + echo "Running tests..." + + deploy: + needs: test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install nctl + run: | + echo 'deb [trusted=yes] https://repo.nine.ch/deb/ /' | sudo tee /etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get update -qqo Dir::Etc::sourcelist=/etc/apt/sources.list.d/repo.nine.ch.list + sudo apt-get install -qq nctl + + - name: Authenticate nctl + run: | + nctl auth login \ + --api-client-id=${{ secrets.NCTL_API_CLIENT_ID }} \ + --api-client-secret=${{ secrets.NCTL_API_CLIENT_SECRET }} \ + --organization=${{ secrets.NCTL_ORGANIZATION }} + + - name: Deploy to Deploio + run: | + nctl update app ${{ vars.DEPLOIO_APP_NAME }} \ + --project ${{ vars.DEPLOIO_PROJECT }} \ + --git-revision=$(git rev-parse HEAD) \ + --skip-repo-access-check +``` -✅ Once the tests are successful, this is then promoted to deployment. +### Semaphore + +Create a deployment pipeline file (e.g., `main-deploy.yml`) that installs `nctl`, authenticates, and triggers the deployment: + +```yaml +version: v1.0 +name: Deploy to Deploio + +agent: + machine: + type: f1-standard-4 + os_image: ubuntu2204 + +blocks: + - name: deploy + task: + secrets: + - name: deploio-credentials + jobs: + - name: deploy + commands: + - echo 'deb [trusted=yes] https://repo.nine.ch/deb/ /' | sudo tee /etc/apt/sources.list.d/repo.nine.ch.list + - sudo apt-get update -qqo Dir::Etc::sourcelist=/etc/apt/sources.list.d/repo.nine.ch.list + - sudo apt-get install -qq nctl + - nctl auth login --api-client-id=$NCTL_API_CLIENT_ID --api-client-secret=$NCTL_API_CLIENT_SECRET --organization=$NCTL_ORGANIZATION + - nctl update app $DEPLOIO_APP_NAME --project $DEPLOIO_PROJECT --git-revision=$(git rev-parse HEAD) --skip-repo-access-check +``` -✅ This deployment script runs on the CI, installing and authenticating `nctl` and updating the app running on Deploio to the new git revision, triggering a new deployment on Deploio. +#### Auto-promote after tests -✅ We then trigger the script (in our case a Ruby script) which provides feedback on the release. Once this has completed successfully the CI is finished and we have an updated application running. +In your main pipeline file, add a promotion that triggers the deployment pipeline when tests pass: -[//]: # (## Deployment Strategies) -[//]: # () -[//]: # (##### Rolling deployments.) -[//]: # (##### Blue/green deployments.) +```yaml +promotions: + - name: deploy + deployment_target: production + pipeline_file: main-deploy.yml + auto_promote: + when: result = 'passed' and branch = 'main' +```