[Multi_K8s-Plugin] Primary Rollout by mohammedfirdouss · Pull Request #6591 · pipe-cd/pipecd

mohammedfirdouss · 2026-03-14T23:21:04Z

What this PR does
Implements K8S_PRIMARY_ROLLOUT for the kubernetes_multicluster plugin. After a canary is validated, this stage promotes the new version as PRIMARY across all targeted clusters in parallel. It supports addVariantLabelToSelector (inject variant label into workload selectors), createService (generate a PRIMARY-varianta Service), and prune (delete stale primary resources no longer in Git).

Why we need it
This is the promotion step in the canary pipeline lifecycle — K8S_CANARY_ROLLOUT and WAIT_APPROVAL are validation gates, but K8S_PRIMARY_ROLLOUT is what actually delivers the new version to end-users across all clusters.

Which issue(s) this PR fixes
Fixes #6446

Does this PR introduce a user-facing change

How are users affected — Users can now add K8S_PRIMARY_ROLLOUT to their pipeline to promote a validated canary to primary across multiple clusters
Is this breaking change — No
How to migrate — N/A

Primary rollout with service

Screenshot Capture - 2026-03-19 - 14-39-34

Screenshot Capture - 2026-03-19 - 14-40-14

Screenshot Capture - 2026-03-19 - 14-40-23

Screenshot Capture - 2026-03-19 - 14-40-53

Screenshot Capture - 2026-03-19 - 14-40-57

Screenshot Capture - 2026-03-19 - 14-41-08

Screenshot Capture - 2026-03-19 - 14-41-14

Screenshot Capture - 2026-03-19 - 14-39-25

Primary rollout without service

Screenshot Capture - 2026-03-19 - 14-35-06

Screenshot Capture - 2026-03-19 - 14-35-16

Screenshot Capture - 2026-03-19 - 14-36-01

Screenshot Capture - 2026-03-19 - 14-36-18

Screenshot Capture - 2026-03-19 - 14-36-30

Screenshot Capture - 2026-03-19 - 14-36-40

Screenshot Capture - 2026-03-19 - 14-36-44

Screenshot Capture - 2026-03-19 - 14-34-56

codecov · 2026-03-17T09:52:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 23.29%. Comparing base (e740e52) to head (9589e3e).

❗ There is a different number of reports uploaded between BASE (e740e52) and HEAD (9589e3e). Click for more details.

HEAD has 12 uploads less than BASE

Flag BASE (e740e52) HEAD (9589e3e)

.-tool-codegen-protoc-gen-auth 1 0

.-tool-actions-plan-preview 1 0

.-pkg-app-pipedv1-plugin-scriptrun 1 0

.-pkg-app-pipedv1-plugin-analysis 1 0

.-pkg-plugin-sdk 1 0

.-pkg-app-pipedv1-plugin-waitapproval 1 0

.-pkg-app-pipedv1-plugin-ecs 1 0

.-pkg-app-pipedv1-plugin-wait 1 0

.-pkg-app-pipedv1-plugin-terraform 1 0

.-pkg-app-pipedv1-plugin-kubernetes_multicluster 1 0

.-pkg-app-pipedv1-plugin-kubernetes 1 0

.-tool-actions-gh-release 1 0

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6591      +/-   ##
==========================================
- Coverage   28.94%   23.29%   -5.66%     
==========================================
  Files         577      424     -153     
  Lines       61367    47104   -14263     
==========================================
- Hits        17763    10972    -6791     
+ Misses      42250    35228    -7022     
+ Partials     1354      904     -450

Flag	Coverage Δ
.	`23.29% <ø> (+0.02%)`	⬆️
.-pkg-app-pipedv1-plugin-analysis	`?`
.-pkg-app-pipedv1-plugin-ecs	`?`
.-pkg-app-pipedv1-plugin-kubernetes	`?`
.-pkg-app-pipedv1-plugin-kubernetes_multicluster	`?`
.-pkg-app-pipedv1-plugin-scriptrun	`?`
.-pkg-app-pipedv1-plugin-terraform	`?`
.-pkg-app-pipedv1-plugin-wait	`?`
.-pkg-app-pipedv1-plugin-waitapproval	`?`
.-pkg-plugin-sdk	`?`
.-tool-actions-gh-release	`?`
.-tool-actions-plan-preview	`?`
.-tool-codegen-protoc-gen-auth	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mohammedfirdouss · 2026-03-18T23:42:17Z

@Warashi Please when you have time, I need your review on the primary rollout.

Warashi · 2026-03-18T23:47:20Z

@mohammedfirdouss
Thank you for reminding me.
Could you please resolve the CI failure first?
I don't have much time on this weekend, but I prioritize reviewing this PR after the CI passes.

Warashi

This PR seems to contain changes related to the Canary Clean stage. Please take a look?

mohammedfirdouss · 2026-03-20T10:03:18Z

This PR seems to contain changes related to the Canary Clean stage. Please take a look?

Yes, I just need to rebase that is all.

@Warashi all done

Warashi

I haven't seen all of the changes, but there seems to be something wrong with modifying Canary’s code.
Please take a look.

P.S.
I have a family trip from this Friday to Sunday, so I cannot review deeply. Sorry for the inconvenience.

Warashi · 2026-03-20T12:28:37Z

...bernetes_multicluster/deployment/testdata/canary_rollout_with_create_service/app.pipecd.yaml

I think there's something wrong with this file being modified.
This PR is not for canary stages, and this file is.

mohammedfirdouss · 2026-03-20T14:27:04Z

I haven't seen all of the changes, but there seems to be something wrong with modifying Canary’s code. Please take a look.

P.S. I have a family trip from this Friday to Sunday, so I cannot review deeply. Sorry for the inconvenience.

Ohh, I will check that now. Okay noted, no problem at all.

* Add K8S_PRIMARY_ROLLOUT stage definition and pipeline registration * Implement executeK8sMultiPrimaryRolloutStage with multi-cluster support * Add primary rollout tests covering single cluster, multi-cluster, prune, and service creation scenarios * Add testdata configurations for primary rollout stages * Add K8sPrimaryRolloutStageOptions config type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

mohammedfirdouss · 2026-03-24T06:54:02Z

@Warashi hi. Any updates on this? :)

github-actions bot added area/go area/examples area/pipedv1 labels Mar 14, 2026

mohammedfirdouss changed the title ~~[Multi_K8s-Plugin] Multi Primary Rollout~~ [Multi_K8s-Plugin] Primary Rollout Mar 14, 2026

mohammedfirdouss marked this pull request as ready for review March 17, 2026 09:49

mohammedfirdouss requested a review from a team as a code owner March 17, 2026 09:49

mohammedfirdouss requested review from Warashi, ffjlabo and khanhtc1202 March 17, 2026 09:49

mohammedfirdouss added the do-not-merge/wip label Mar 17, 2026

Warashi reviewed Mar 19, 2026

View reviewed changes

github-actions bot removed the area/examples label Mar 20, 2026

mohammedfirdouss force-pushed the feat/k8s-multi-primary-rollout branch from 0a73ef1 to 2d7112f Compare March 20, 2026 10:21

Warashi reviewed Mar 20, 2026

View reviewed changes

mohammedfirdouss force-pushed the feat/k8s-multi-primary-rollout branch from 2d7112f to 15e118c Compare March 20, 2026 14:33

mohammedfirdouss force-pushed the feat/k8s-multi-primary-rollout branch 3 times, most recently from f0621cd to c28a5bb Compare March 20, 2026 14:42

mohammedfirdouss added 4 commits March 20, 2026 14:43

Add K8s primary rollout stage and related configurations

636cd27

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

Implement K8s multi-primary rollout stage

bff0fc2

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

Add K8s multi-primary rollout stage tests

96273bd

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

Add testdata configurations for primary rollout stages

6ea10cd

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

mohammedfirdouss force-pushed the feat/k8s-multi-primary-rollout branch from c28a5bb to 6ea10cd Compare March 20, 2026 14:43

mohammedfirdouss removed the do-not-merge/wip label Mar 21, 2026

mohammedfirdouss added the approval-needed label Mar 21, 2026

mohammedfirdouss added 2 commits March 23, 2026 00:35

Merge branch 'master' into feat/k8s-multi-primary-rollout

d1674b8

Merge branch 'master' into feat/k8s-multi-primary-rollout

7c7bab6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Multi_K8s-Plugin] Primary Rollout #6591

[Multi_K8s-Plugin] Primary Rollout #6591
mohammedfirdouss wants to merge 6 commits intopipe-cd:masterfrom
mohammedfirdouss:feat/k8s-multi-primary-rollout

mohammedfirdouss commented Mar 14, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 17, 2026 •

edited

Loading

Uh oh!

mohammedfirdouss commented Mar 18, 2026

Uh oh!

Warashi commented Mar 18, 2026

Uh oh!

Warashi left a comment

Uh oh!

mohammedfirdouss commented Mar 20, 2026 •

edited

Loading

Uh oh!

Warashi left a comment

Uh oh!

Warashi Mar 20, 2026

Uh oh!

mohammedfirdouss commented Mar 20, 2026

Uh oh!

mohammedfirdouss commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mohammedfirdouss commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mohammedfirdouss commented Mar 18, 2026

Uh oh!

Warashi commented Mar 18, 2026

Uh oh!

Warashi left a comment

Choose a reason for hiding this comment

Uh oh!

mohammedfirdouss commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Warashi left a comment

Choose a reason for hiding this comment

Uh oh!

Warashi Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

mohammedfirdouss commented Mar 20, 2026

Uh oh!

mohammedfirdouss commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mohammedfirdouss commented Mar 14, 2026 •

edited

Loading

codecov bot commented Mar 17, 2026 •

edited

Loading

mohammedfirdouss commented Mar 20, 2026 •

edited

Loading