Skip to content

Commit e730f1e

Browse files
author
Nissan Pow
committed
ci: fix airflow timeout flakiness with 4 mitigations
1. Reduce airflow workers from 2 to 1 (less k8s resource contention) 2. Increase minikube memory from 6GB to 7GB for airflow backend 3. Increase per-test timeout from 900s to 1200s for airflow 4. Add --reruns=1 across all backends to retry flaky timeouts The airflow deployer tests have high overhead per test (DAG parsing, k8s pod scheduling, conda env setup) that sometimes exceeds 900s in constrained CI environments. Different tests timeout on each run, confirming this is environmental variance not a code bug.
1 parent 45d18c9 commit e730f1e

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

.github/workflows/ux-tests.yml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -100,15 +100,23 @@ jobs:
100100
- backend: local
101101
services: "minio,postgresql,metadata-service"
102102
workers: 4
103+
memory: 6144
104+
timeout: 900
103105
- backend: argo-kubernetes
104106
services: "minio,postgresql,metadata-service,argo-workflows"
105107
workers: 2
108+
memory: 6144
109+
timeout: 900
106110
- backend: airflow-kubernetes
107111
services: "minio,postgresql,metadata-service,airflow"
108-
workers: 2
112+
workers: 1
113+
memory: 7168
114+
timeout: 1200
109115
- backend: sfn-batch
110116
services: "minio,postgresql,metadata-service,localbatch,ddb-local,sfn-local"
111117
workers: 2
118+
memory: 6144
119+
timeout: 900
112120

113121
runs-on: ubuntu-latest
114122

@@ -138,7 +146,7 @@ jobs:
138146
with:
139147
driver: docker
140148
cpus: 2
141-
memory: 6144
149+
memory: ${{ matrix.memory }}
142150

143151
- name: Restore minikube image cache
144152
id: image-cache
@@ -259,7 +267,8 @@ jobs:
259267
-n ${{ matrix.workers }} \
260268
-v \
261269
--tb=short \
262-
--timeout=900 \
270+
--timeout=${{ matrix.timeout }} \
271+
--reruns=1 \
263272
--cov=metaflow \
264273
--cov-report=term-missing \
265274
--cov-report=xml:coverage.xml \

0 commit comments

Comments
 (0)