Skip to content

Commit 6d7dd2d

Browse files
eudelins-zamaeudelins-zama
authored
ci(common): deterministic docker build cancellation (#1864)
* ci(kms-connector): deterministic docker build cancellation * ci(coprocessor): deterministic docker build cancellation * ci(gateway-contracts): deterministic docker build cancellation * ci(host-contracts): deterministic docker build cancellation * ci(test-suite): deterministic docker build cancellation * fix(common): fix github-script version in docs * fix(ci): last commit error handling * fix(ci): actionlint findings * docs(ci): update readme * ci(common): fix after review
1 parent ab21177 commit 6d7dd2d

8 files changed

+638
-312
lines changed

‎.github/README.md‎

Lines changed: 139 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ The repository uses a set of reusable workflows to build and publish Docker imag
88

99
1. **Build images only when relevant files change** - avoiding unnecessary builds
1010
2. **Re-tag existing images when no changes occur** - ensuring every commit on `main` has corresponding Docker image tags without rebuilding
11+
3. **Deterministic build cancellation** - ensuring that when multiple PRs are merged simultaneously on `main`, only the latest commit's workflow runs (see [Deterministic Build Cancellation](#deterministic-build-cancellation))
1112

1213
### Architecture Overview
1314

@@ -17,25 +18,47 @@ The repository uses a set of reusable workflows to build and publish Docker imag
1718
│ - Triggered on push/release │
1819
└───────────────┬─────────────────────┘
1920
│
20-
▼
21-
┌─────────────────────────────────────┐
22-
│ check-changes-for-docker- │ (reusable workflow)
23-
│ build.yml │
24-
│ - Detects file changes │
25-
│ - Finds previous image base commit │
26-
│ - Outputs: changes, base-commit │
27-
└───────────────┬─────────────────────┘
28-
│
29-
┌───────┴──────────────────────┐
30-
▼ ▼
31-
┌───────────────────────────┐ ┌─────────────────────────┐
32-
│ common-docker.yml (build) │ │ re-tag-docker-image.yml │
33-
│ if changes │ │ if no changes │
34-
└───────────────────────────┘ └─────────────────────────┘
21+
┌──────────┴──────────┐
22+
▼ ▼
23+
┌─────────────────┐ ┌─────────────────────────────────────┐
24+
│ is-latest- │ │ check-changes-for-docker- │
25+
│ commit.yml │ │ build.yml │
26+
│ (push only) │ │ - Detects file changes │
27+
│ - Checks if │ │ - Finds previous image base commit │
28+
│ current SHA │ │ - Outputs: changes, base-commit │
29+
│ is latest on │ └───────────────┬─────────────────────┘
30+
│ branch │ │
31+
└────────┬────────┘ │
32+
│ │
33+
└───────────┬───────────────┘
34+
│
35+
▼
36+
┌─────────────────────────────┐
37+
│ build-decisions (optional) │ (job in caller workflow, only for
38+
│ - Centralizes build logic │ multi-image workflows)
39+
│ - Outputs: build/retag/skip│
40+
└───────────────┬─────────────┘
41+
│
42+
┌──────────────┼──────────────┐
43+
▼ ▼ ▼
44+
┌────────────────┐ ┌──────────────┐ ┌──────┐
45+
│ common-docker │ │ re-tag-docker│ │ skip │
46+
│ (build) │ │ -image.yml │ │ │
47+
└────────────────┘ └──────────────┘ └──────┘
3548
```
3649

3750
### Reusable Workflows
3851

52+
#### `is-latest-commit.yml`
53+
54+
Checks whether the current commit is the latest on the target branch. This enables deterministic build cancellation by allowing workflows to skip execution if a newer commit has been pushed.
55+
56+
**How it works:**
57+
58+
1. Uses `git ls-remote` to fetch the latest commit SHA from the remote branch
59+
2. Compares it with the current workflow's commit SHA (`github.sha`)
60+
3. Outputs `is_latest: true` if they match, `false` otherwise
61+
3962
#### `check-changes-for-docker-build.yml`
4063

4164
Determines whether a Docker image needs to be rebuilt by checking if relevant files have changed since the last commit that has a published Docker image.
@@ -50,48 +73,116 @@ Determines whether a Docker image needs to be rebuilt by checking if relevant fi
5073

5174
Creates a new tag for an existing Docker image without rebuilding it.
5275

53-
### Docker Build Workflow Pattern
76+
### Docker Build Workflow Patterns
5477

55-
Each service has its own docker build workflow (e.g., `coprocessor-docker-build.yml`, `kms-connector-docker-build.yml`) that follows this pattern:
78+
Each service has its own docker build workflow. There are two patterns depending on the number of images built:
79+
80+
#### Simple Pattern (Single Image)
81+
82+
Used by workflows that build a single image (e.g., `gateway-contracts-docker-build.yml`, `host-contracts-docker-build.yml`, `test-suite-docker-build.yml`). The decision logic is embedded directly in the job `if` conditions:
5683

5784
```yaml
5885
jobs:
59-
# 1. Check for changes using the reusable workflow
60-
check-changes-my-service:
86+
# 1. Check if this is the latest commit (push events only)
87+
is-latest-commit:
88+
uses: ./.github/workflows/is-latest-commit.yml
89+
if: github.event_name == 'push'
90+
91+
# 2. Check for changes
92+
check-changes:
93+
if: github.event_name == 'push' || inputs.is_workflow_call
6194
uses: ./.github/workflows/check-changes-for-docker-build.yml
62-
secrets:
63-
GHCR_READ_TOKEN: ${{ secrets.GHCR_READ_TOKEN }}
64-
permissions:
65-
actions: 'read'
66-
contents: 'read'
67-
pull-requests: 'read'
68-
with:
69-
caller-workflow-event-name: ${{ github.event_name }}
70-
caller-workflow-event-before: ${{ github.event.before }}
71-
docker-image: fhevm/my-service
72-
filters: |
73-
my-service:
74-
- .github/workflows/my-service-docker-build.yml
75-
- my-service/**
76-
77-
# 2. Build if changes detected (or on release/workflow_dispatch)
78-
build-my-service:
79-
needs: check-changes-my-service
95+
# ... configuration
96+
97+
# 3. Build with inline decision logic
98+
build:
99+
needs: [is-latest-commit, check-changes]
100+
concurrency:
101+
group: my-service-build-${{ github.ref_name }}
102+
cancel-in-progress: true
80103
if: |
81-
github.event_name == 'release'
82-
|| (github.event_name != 'workflow_dispatch' && needs.check-changes-my-service.outputs.changes == 'true')
83-
|| (github.event_name == 'workflow_dispatch' && inputs.build_my_service)
104+
always()
105+
&& (
106+
github.event_name == 'release'
107+
|| github.event_name == 'workflow_dispatch'
108+
|| (github.event_name == 'push' && needs.is-latest-commit.outputs.is_latest == 'true' && needs.check-changes.outputs.changes == 'true')
109+
|| (inputs.is_workflow_call && needs.check-changes.outputs.changes == 'true')
110+
)
84111
uses: zama-ai/ci-templates/.github/workflows/common-docker.yml@<version>
85112
# ... build configuration
86113

87-
# 3. Re-tag if no changes detected (push events only)
88-
re-tag-my-service-image:
89-
needs: check-changes-my-service
114+
# 4. Re-tag with inline decision logic
115+
re-tag-image:
116+
needs: [is-latest-commit, check-changes]
90117
if: |
91-
needs.check-changes-my-service.outputs.changes != 'true' && github.event_name == 'push'
118+
always()
119+
&& (
120+
github.event_name == 'push' && needs.is-latest-commit.outputs.is_latest == 'true' && needs.check-changes.outputs.changes != 'true'
121+
)
92122
uses: ./.github/workflows/re-tag-docker-image.yml
93-
with:
94-
image-name: "fhevm/my-service"
95-
previous-tag-or-commit: ${{ needs.check-changes-my-service.outputs.base-commit }}
96-
new-tag-or-commit: ${{ github.event.after }}
123+
# ... configuration
97124
```
125+
126+
#### Complex Pattern (Multiple Images)
127+
128+
Used by workflows that build multiple images (e.g., `coprocessor-docker-build.yml`, `kms-connector-docker-build.yml`). A centralized `build-decisions` job computes the action for each service to avoid duplicating decision logic:
129+
130+
```yaml
131+
jobs:
132+
# 1. Check if this is the latest commit (push events only)
133+
is-latest-commit:
134+
uses: ./.github/workflows/is-latest-commit.yml
135+
if: github.event_name == 'push'
136+
137+
# 2. Check for changes for each service
138+
check-changes-service-a:
139+
uses: ./.github/workflows/check-changes-for-docker-build.yml
140+
# ... configuration
141+
142+
check-changes-service-b:
143+
uses: ./.github/workflows/check-changes-for-docker-build.yml
144+
# ... configuration
145+
146+
# 3. Centralized decision logic for all services
147+
build-decisions:
148+
runs-on: ubuntu-latest
149+
if: always()
150+
needs: [is-latest-commit, check-changes-service-a, check-changes-service-b]
151+
outputs:
152+
service_a: ${{ steps.decide.outputs.service_a }}
153+
service_b: ${{ steps.decide.outputs.service_b }}
154+
steps:
155+
# ... decide which images need to be built
156+
157+
# 4. Build if decision is "build"
158+
build-service-a:
159+
needs: build-decisions
160+
concurrency:
161+
group: service-a-build-${{ github.ref_name }}
162+
cancel-in-progress: true
163+
if: always() && needs.build-decisions.outputs.service_a == 'build'
164+
uses: zama-ai/ci-templates/.github/workflows/common-docker.yml@<version>
165+
# ... build configuration
166+
167+
# 5. Re-tag if decision is "retag"
168+
re-tag-service-a-image:
169+
needs: [build-decisions, check-changes-service-a]
170+
if: always() && needs.build-decisions.outputs.service_a == 'retag'
171+
uses: ./.github/workflows/re-tag-docker-image.yml
172+
# ... configuration
173+
```
174+
175+
### Deterministic Build Cancellation
176+
177+
When multiple PRs are merged to `main` in quick succession, GitHub's concurrency groups cannot guarantee which workflow will "win" - the ordering is arbitrary. This could result in an older commit's workflow completing while a newer commit's workflow gets cancelled.
178+
179+
To solve this, the workflows now use a **deterministic cancellation** approach:
180+
181+
1. **`is-latest-commit.yml`** checks at runtime if the current commit is still the latest on the branch
182+
2. If the commit is the latest: proceed with build or retag. If a newer commit exists: skip all work.
183+
184+
This is used only if the docker build workflow is triggered by a push on `main`!
185+
186+
This ensures that only the workflow for the most recent commit on `main` will actually build or retag images, regardless of the order in which GitHub starts the workflows.
187+
188+
**Note:** Concurrency groups are still used on individual build jobs to prevent duplicate builds of the same service, but the `is-latest-commit` check handles the cross-workflow coordination.

0 commit comments

Comments
 (0)