Gated Release Pipeline #1205

MrAlias · 2026-01-28T22:08:42Z

Resolve #1200

Update release pipeline to create a draft release instead of immediately publishing it. The publish will be done by a maintainer after review.
Add a verification jobs that call our testing and build checks
Add a way for maintainers to re-trigger the release workflow in case there is a failure that requires a retry.

codecov · 2026-01-28T22:18:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.10%. Comparing base (b97b052) to head (41d3cf0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1205      +/-   ##
==========================================
- Coverage   44.04%   43.10%   -0.95%     
==========================================
  Files         298      299       +1     
  Lines       27097    32120    +5023     
==========================================
+ Hits        11934    13844    +1910     
- Misses      14446    17383    +2937     
- Partials      717      893     +176

Flag	Coverage Δ
integration-test	`21.19% <ø> (?)`
integration-test-arm	`0.00% <ø> (?)`
integration-test-vm-${ARCH}-${KERNEL_VERSION}	`0.00% <ø> (?)`
k8s-integration-test	`2.41% <ø> (?)`
oats-test	`0.00% <ø> (?)`
unittests	`44.03% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add a script to continually poll the CI system status for a commit. It reports if the CI system succeeded for the jobs we care about or not.

Copilot

Pull request overview

This PR implements a gated release pipeline that creates draft releases instead of immediately publishing them, allowing maintainers to review before publication. The changes add CI verification to ensure all required checks pass before creating a release, and provide a manual workflow trigger for re-running failed release attempts.

Changes:

Added CI verification script that polls GitHub's API to check required workflow statuses before allowing release
Updated release workflow to create draft releases only after CI verification passes
Added manual workflow trigger capability for re-running release workflow
Documented the new automatic and manual release processes

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
`.github/workflows/release.yml`	Added verification job with tag validation and CI checks, configured release job to create drafts after verification passes, added workflow_dispatch trigger for manual re-runs
`.github/scripts/verify-ci-status.sh`	New script that verifies required CI checks (test shards, kernel variants, k8s tests, OATS tests, markdown linting) have passed on tagged commit, with polling and timeout logic
`RELEASING.md`	Added documentation for automatic release workflow, manual trigger process, and CI verification requirements

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RELEASING.md

.github/scripts/verify-ci-status.sh

.github/workflows/release.yml

.github/scripts/verify-ci-status.sh

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RELEASING.md

.github/workflows/release.yml

.github/scripts/verify-ci-status.sh

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/scripts/verify-ci-status.sh

.github/workflows/release.yml

.github/scripts/verify-ci-status.sh

Directly call all CI workflows needed to validate the code being released is passing our testing and build validation. This accounts for the release commits most often not containing any code changes that trigger these CI workflows and therefore not something we can check happened successfully before releasing.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/release.yml

RELEASING.md

.github/workflows/release.yml

.github/workflows/pull_request_integration_tests.yml

grcevski

This looks good to me, but I'm not very good with the CI jobs. Probably good to have someone else approve as well.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-30T01:18:17Z

.github/workflows/release.yml

+          # Validate tag format (must be v*.*.* with numeric components, optional pre-release suffix)
+          if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9]+)?$ ]]; then
+            echo "ERROR: Tag must match format v*.*.* or v*.*.*-suffix (e.g., v1.2.3 or v1.2.3-rc1), got: $TAG"


The regex pattern for the pre-release suffix allows only alphanumeric characters ([a-zA-Z0-9]+), which means it would reject valid semantic versioning pre-release identifiers that contain dots or hyphens (e.g., v1.2.3-rc.1, v1.2.3-beta-2). According to semantic versioning (semver), pre-release versions can contain alphanumerics and hyphens, separated by dots. Consider updating the regex to be more permissive, such as ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?$ to support standard semantic versioning formats.

Suggested change

# Validate tag format (must be v*.*.* with numeric components, optional pre-release suffix)

if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9]+)?$ ]]; then

echo "ERROR: Tag must match format v*.*.* or v*.*.*-suffix (e.g., v1.2.3 or v1.2.3-rc1), got: $TAG"

# Validate tag format (must be v*.*.* with numeric components, optional semver-style pre-release suffix)

if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?$ ]]; then

echo "ERROR: Tag must match format v*.*.* or v*.*.*-suffix (e.g., v1.2.3, v1.2.3-rc1, v1.2.3-rc.1, v1.2.3-beta-2), got: $TAG"

Copilot · 2026-01-30T01:18:17Z

.github/workflows/release.yml

      # - publish_dockerhub_main.yml
      # - publish_dockerhub_k8s_cache_main.yml
      - "v*.*.*"
+      - "v*.*.*-*"


There's a mismatch between the tag trigger pattern and the validation regex. The trigger pattern v*.*.*-* uses wildcards and will match tags like v1.2.3-rc.1 (with dots in the suffix), but the validation regex ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9]+)?$ only allows alphanumeric characters in the suffix (no dots or additional hyphens). This means the workflow could be triggered by a tag that then fails validation. Consider either making the trigger pattern more specific or making the validation regex more permissive to handle standard semantic versioning formats.

Suggested change

- "v*.*.*-*"

- "v*.*.*-[0-9A-Za-z]*"

MrAlias added 2 commits January 28, 2026 14:07

Allow manual re-trigger of release workflow

7e63cb1

Make created release a draft

58a375f

Add verify-ci-status.sh script

1b62720

Add a script to continually poll the CI system status for a commit. It reports if the CI system succeeded for the jobs we care about or not.

MrAlias force-pushed the gate-release branch 2 times, most recently from 83a2685 to cf32c77 Compare January 28, 2026 22:28

MrAlias added 2 commits January 28, 2026 14:31

Add verification step to release workflow

128a500

Update release docs

ead6f98

MrAlias force-pushed the gate-release branch from cf32c77 to ead6f98 Compare January 28, 2026 22:31

MrAlias marked this pull request as ready for review January 28, 2026 22:36

MrAlias requested a review from a team as a code owner January 28, 2026 22:36

Copilot AI review requested due to automatic review settings January 28, 2026 22:36

Copilot started reviewing on behalf of MrAlias January 28, 2026 22:36 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

MrAlias and others added 2 commits January 28, 2026 14:50

Apply suggestions from code review

5e1f675

Co-authored-by: Copilot <[email protected]>

Validate pushed tag as well

bb67787

MrAlias requested a review from Copilot January 28, 2026 22:55

Copilot started reviewing on behalf of MrAlias January 28, 2026 22:55 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

RELEASING.md Outdated Show resolved Hide resolved

RELEASING.md Outdated Show resolved Hide resolved

.github/workflows/release.yml Outdated Show resolved Hide resolved

.github/scripts/verify-ci-status.sh Outdated Show resolved Hide resolved

Apply feedback from PR review

194cbe2

MrAlias requested a review from Copilot January 28, 2026 23:28

Copilot started reviewing on behalf of MrAlias January 28, 2026 23:28 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

MrAlias marked this pull request as draft January 29, 2026 00:19

MrAlias added 3 commits January 29, 2026 08:15

Update releasing docs

1212ec9

Pass git-ref to referenced workflows

9930bf0

MrAlias marked this pull request as ready for review January 29, 2026 16:53

MrAlias requested a review from Copilot January 29, 2026 16:53

Copilot started reviewing on behalf of MrAlias January 29, 2026 16:54 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

MrAlias marked this pull request as draft January 29, 2026 17:48

grcevski approved these changes Jan 29, 2026

View reviewed changes

MrAlias added 2 commits January 29, 2026 16:54

Merge branch 'main' into gate-release

851ebee

Apply feedback

41d3cf0

MrAlias force-pushed the gate-release branch from ecf04c2 to 41d3cf0 Compare January 30, 2026 01:12

MrAlias marked this pull request as ready for review January 30, 2026 01:14

MrAlias requested a review from Copilot January 30, 2026 01:14

Copilot started reviewing on behalf of MrAlias January 30, 2026 01:14 View session

Copilot AI reviewed Jan 30, 2026

View reviewed changes

Gated Release Pipeline #1205

Are you sure you want to change the base?

Gated Release Pipeline #1205

Uh oh!

Conversation

MrAlias commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grcevski left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MrAlias commented Jan 28, 2026 •

edited

Loading

codecov bot commented Jan 28, 2026 •

edited

Loading