Skip to content

Commit bfa185d

Browse files
vancauwevancauwe
authored
Merge pull request #3 from sdsc-ordes/feat/airflow
feat: airflow
2 parents c8f743c + 08841c5 commit bfa185d

9 files changed

Lines changed: 611 additions & 1 deletion

File tree

‎.github/workflows/docker-airflow.yml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
name: Build and Push Docker Image for Airflow
2+
3+
on:
4+
release:
5+
type: [published]
6+
#trigger on each commit of each branch and push with latest tag
7+
push :
8+
pull_request:
9+
paths:
10+
- 'pyproject.toml'
11+
- 'tools/images/**'
12+
- '.github/workflows/**'
13+
14+
env:
15+
REGISTRY: ghcr.io
16+
17+
jobs:
18+
build-image:
19+
runs-on: ubuntu-latest
20+
#if: github.ref != 'refs/heads/main'
21+
permissions:
22+
contents: read
23+
packages: write
24+
25+
steps:
26+
# https://github.com/actions/checkout
27+
- name: checkout repository
28+
uses: actions/checkout@v4
29+
30+
- name: lowercase image name
31+
run: |
32+
echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}-airflow" >> ${GITHUB_ENV}
33+
34+
# https://github.com/docker/setup-qemu-action
35+
- name: Set up QEMU
36+
uses: docker/setup-qemu-action@v3.0.0
37+
38+
# https://github.com/docker/setup-buildx-action
39+
- name: Set up Docker Buildx
40+
id: buildx
41+
uses: docker/setup-buildx-action@v3.0.0
42+
43+
- name: Get current release version
44+
id: release-version
45+
run: |
46+
version=$(grep -E '^version += +' pyproject.toml | sed -E 's/.*= +//' | sed "s/['\"]//g")
47+
echo "version=${version}" >> $GITHUB_OUTPUT
48+
echo "version_build=${version}_"$(git rev-parse --short "$GITHUB_SHA") >> $GITHUB_OUTPUT
49+
50+
# https://github.com/docker/build-push-action
51+
- name: Build Docker image
52+
uses: docker/build-push-action@v5.0.0
53+
with:
54+
context: .
55+
platforms: linux/amd64
56+
file: tools/images/Dockerfile-airflow
57+
push: false
58+
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.release-version.outputs.version_build }}
59+
build-args: PLATFORM=cu124
60+
outputs: type=image,annotation-index.org.opencontainers.image.description=Run a Graph Neural Network.
61+
62+
push-image:
63+
runs-on: ubuntu-latest
64+
#if: github.ref == 'refs/heads/main'
65+
permissions:
66+
contents: read
67+
packages: write
68+
69+
steps:
70+
# https://github.com/actions/checkout
71+
- name: checkout repository
72+
uses: actions/checkout@v4
73+
74+
- name: lowercase image name
75+
run: |
76+
echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}-airflow" >> ${GITHUB_ENV}
77+
78+
# https://github.com/docker/setup-qemu-action
79+
- name: Set up QEMU
80+
uses: docker/setup-qemu-action@v3.0.0
81+
82+
# https://github.com/docker/setup-buildx-action
83+
- name: Set up Docker Buildx
84+
id: buildx
85+
uses: docker/setup-buildx-action@v3.0.0
86+
87+
- name: Get current release version
88+
id: release-version
89+
run: |
90+
version=$(grep -E '^version += +' pyproject.toml | sed -E 's/.*= +//' | sed "s/['\"]//g")
91+
echo "version=${version}" >> $GITHUB_OUTPUT
92+
echo "version_build=${version}_"$(git rev-parse --short "$GITHUB_SHA") >> $GITHUB_OUTPUT
93+
94+
# https://github.com/docker/login-action
95+
- name: Log in to the Container registry
96+
uses: docker/login-action@v3.0.0
97+
with:
98+
registry: ${{ env.REGISTRY }}
99+
username: ${{ github.actor }}
100+
password: ${{ secrets.GITHUB_TOKEN }}
101+
102+
# https://github.com/docker/metadata-action
103+
- name: Extract metadata (tags, labels) for Docker
104+
id: meta
105+
uses: docker/metadata-action@v5.0.0
106+
with:
107+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
108+
tags: |
109+
type=raw,value=latest,enable=${{ github.event_name == 'push' }}
110+
type=raw,value=${{ needs.build-image.outputs.version }},enable=${{ github.event_name == 'release' }}
111+
# for future:
112+
# type=raw,value=${{ needs.build-image.outputs.version_build }},enable=${{ github.event_name == 'push' }}
113+
114+
# https://github.com/docker/build-push-action
115+
- name: Push Docker image
116+
uses: docker/build-push-action@v5.0.0
117+
with:
118+
context: .
119+
platforms: linux/amd64
120+
file: tools/images/Dockerfile-airflow
121+
push: true
122+
tags: ${{ steps.meta.outputs.tags }}
123+
labels: ${{ steps.meta.outputs.labels }}
124+
build-args: PLATFORM=cu124
125+
outputs: type=image,annotation-index.org.opencontainers.image.description=Run a Graph Neural Network.

‎.gitignore‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,13 @@
1-
.env
1+
.env
2+
.DS_Store
3+
4+
# Ignore any file anywhere in the repository called env_variables
5+
src/airflow/src/env_variables.json
6+
7+
# Ignore all Airflow related configurations
8+
**/airflow/config/
9+
!.gitkeep
10+
**/airflow/plugins/
11+
!.gitkeep
12+
**/airflow/logs/
13+
!.gitkeep

‎src/airflow/.env.dist‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
AIRFLOW_UID=VALUE_OF_AIRFLOW_UID

‎src/airflow/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Airflow Deployment
2+
3+
## Linux set-up
4+
5+
```
6+
mkdir -p ./dags ./logs ./plugins ./config
7+
echo -e "AIRFLOW_UID=$(id -u)" > .env
8+
```
9+
(One must run these commands before launching the docker compose)
10+
11+
## Docker compose
12+
13+
```
14+
docker compose up -d
15+
```
16+
(Note: the api server is currently configured for port `8080`.)
17+
18+
## Sending in DAGs
19+
20+
Dags can be directly placed under the dags folder. Note: if airflow needs to be redeployed, then all folders must be deleted and recreated, DAGs should be backed up somewhere else.
21+
22+
For executing a API call:
23+
24+
(if on the machine, keep localhost, else replace with machine IP.)
25+
26+
1. Get your JWT token
27+
```
28+
curl -X POST http://localhost:8080/auth/token -H "Content-Type: application/json" -d '{
29+
"username": "airflow",
30+
"password": "airflow"
31+
}'
32+
```
33+
The response is YOUR_TOKEN mentioned below.
34+
35+
2. Get the DACS
36+
```
37+
curl -X GET "http://localhost:8080/api/v2/dags" \
38+
-H "Accept: application/json" \
39+
-H "Authorization: Bearer YOUR_TOKEN"
40+
```
41+
42+
3. Trigger a DAG - Coming soon
43+
44+

‎src/airflow/dags/dag-inference.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
from airflow import DAG
2+
from airflow.providers.docker.operators.docker import DockerOperator
3+
from datetime import datetime, timedelta
4+
5+
from airflow.models import Variable
6+
7+
default_args = {
8+
'owner': 'airflow',
9+
'retries': 1,
10+
'retry_delay': timedelta(minutes=5),
11+
}
12+
13+
with DAG(
14+
dag_id='inference_dag',
15+
default_args=default_args,
16+
description='Trigger ML inference every week',
17+
start_date=datetime(2024, 1, 1),
18+
catchup=False,
19+
tags=['ml', 'neo4j', 'inference'],
20+
) as dag:
21+
22+
inference_task = DockerOperator(
23+
task_id='call_runai',
24+
image='ghcr.io/sdsc-ordes/open-pulse-airflow:latest',
25+
force_pull=True,
26+
environment={
27+
'INFERENCE_URL': Variable.get("INFERENCE_URL"),
28+
'NEO4J_DATABASE': Variable.get("NEO4J_DATABASE"),
29+
'API_TOKEN': Variable.get("API_TOKEN"),
30+
},
31+
)
32+
33+
inference_task

0 commit comments

Comments
 (0)