Skip to content

deployment_guide.md

Latest commit

 

History

History
133 lines (94 loc) · 4.7 KB

deployment_guide.md

File metadata and controls

133 lines (94 loc) · 4.7 KB
Raw

Deployment Guide

Prerequisites

  • Docker + access to quay.io/earthcode registry
  • kubectl configured against the target cluster
  • Airflow 3+ with S3 DAG sync enabled (s3://airflow-dags-bc/earthcode/)
  • pixi installed (https://pixi.sh) for generating the build context

1. Build and Push the Image

The docker/ directory is the build context, generated by appligator. It contains a two-stage Dockerfile (pixi build stage → lean Debian runtime), a pixi.toml, a pyproject.toml, and a locked pixi.lock.

Why pixi.toml and pyproject.toml are needed alongside pixi.lock:

  • pixi.toml — pixi unconditionally requires a manifest to run; pixi install --locked won't start without one. The docker/pixi.toml is a minimal stub that satisfies this.
  • pyproject.toml — needed by the pip install --no-deps -e . step that installs smos_ard from src/. It cannot be declared in pixi.toml as a path = "." dependency with --locked because pixi hashes local source at lock time; in Docker the path content differs and pixi refuses to install. Installing with pip after pixi install --locked sidesteps this.

procodile and gavicore are installed directly from the eo-tools/eozilla git repo because the latest changes needed are not yet in a published PyPI release.

Since src/ lives at the project root it must be copied into the build context before building (docker/src/ is gitignored):

# Copy smos_ard source into the build context
cp -r src docker/src

# Build and push (bump the tag on each release)
docker build -t quay.io/earthcode/smos-ard-pipeline:0.1.6 docker/
docker push quay.io/earthcode/smos-ard-pipeline:0.1.6

After pushing, update the image field in dags/smos-ard-service-v2.py to the new tag and redeploy the DAG (step 4).

2. Apply Kubernetes RBAC

The Airflow service account (airflow-sa in the airflow namespace) needs permission to create and manage pods in the earthcode namespace. Apply once:

kubectl apply -f k8s/rbac.yaml

3. Create Kubernetes Secret

Every pipeline pod receives credentials from the earthcode-secrets secret:

Variable Required Used by Description
AWS_S3_ACCESS_KEY_ID yes fetch, aggregate, publish AWS credentials for reading/writing S3
AWS_S3_SECRET_ACCESS_KEY yes fetch, aggregate, publish AWS credentials for reading/writing S3
AWS_DEFAULT_REGION no fetch, aggregate, publish AWS region; defaults to eu-central-1
CREODIAS_S3_KEY yes fetch CREODIAS S3 key for the xcube-smos data store
CREODIAS_S3_SECRET yes fetch CREODIAS S3 secret for the xcube-smos data store

GITHUB_USERNAME and GITHUB_TOKEN are not needed by the pipeline pods. They are only required when running publish_trigger.py separately to open the OSC catalogue PR.

kubectl create secret generic earthcode-secrets \
  --namespace=earthcode \
  --from-literal=AWS_S3_ACCESS_KEY_ID=<key> \
  --from-literal=AWS_S3_SECRET_ACCESS_KEY=<secret> \
  --from-literal=AWS_DEFAULT_REGION=eu-central-1 \
  --from-literal=CREODIAS_S3_KEY=<key> \
  --from-literal=CREODIAS_S3_SECRET=<secret>

4. Deploy the DAG

Airflow syncs DAGs from s3://airflow-dags-bc/earthcode/. Copy the active DAG there:

aws s3 cp dags/smos-ard-service-v2.py s3://airflow-dags-bc/earthcode/

Airflow polls the bucket periodically; the DAG will appear in the UI within a minute or two.

5. Scheduled Runs

The DAG runs automatically at 23:00 UTC every day (schedule="0 23 * * *"). start_date and end_date are derived from Airflow's {{ ds }} logical date — no manual input needed for routine updates.

Template Resolves to (example: ds = 2026-05-13)
{{ ds }} 2026-05-13
{{ macros.ds_add(ds, -1) }} 2026-05-12

So each scheduled run fetches [yesterday, today]. The boundary day (yesterday) is already in the cube and is stripped before appending.

Catchup: catchup=False — if the DAG is paused and re-enabled, only the most recent interval runs. Backfill missed days with an explicit manual trigger (see below).

6. Publishing to the OSC Catalogue

After the pipeline runs, catalogue artefacts are staged at:

s3://earthcode-ard-cubes/catalogue-pending/smos-ard-smos-sm-global/

To open the GitHub PR on the OSC metadata repo, run the publish trigger manually (requires GITHUB_USERNAME, GITHUB_TOKEN, and AWS credentials in the environment):

python -m smos_ard.publish_trigger \
    s3://earthcode-ard-cubes/catalogue-pending/smos-ard-smos-sm-global/

On success the artefacts are archived to catalogue-published/{collection_id}/{timestamp}/ and the pending prefix is removed.