[backend] Deleting Recurring Runs Forces Running Workflows to be Terminated

[JerT33]

Environment

KFP version: 2.15.0
KFP SDK version: 2.15.0

Steps to reproduce

When deleting a recurring run via the API, the ScheduledWorkflow CRD is deleted with Kubernetes default cascade behavior. This causes any in-progress workflows (which have an ownerReference to the ScheduledWorkflow) to be terminated immediately

This is problematic for CI/CD workflows that need to update recurring runs by deleting and recreating them since the delete kills any currently running pipeline

This is also been brought up in this stale issue #7421 but does not seem to ever be addressed

Expected result

There should be a way to set the deletion policy when calling delete recurring runs, this way users can select to orphan current workflows to keep them running

---

Impacted by this bug? Give it a 👍.

[GKR5413]

/assign

[GKR5413]

@JerT33

Hey! I'd like to take a crack at fixing this issue. Before I dive into coding, I wanted to run my approach by you to make sure I'm headed in the right direction.

The Problem (as I understand it)

Right now when you delete a recurring run, Kubernetes does its default cascade deletion thing - which means it kills all the child workflows that are currently running. This is a pain for CI/CD workflows because if you want to update a recurring run's config, you basically have to delete it and recreate it... which terminates all your in-progress pipelines. Not great!

What I'm thinking

The simplest fix would be to add an optional orphan_dependents parameter to the delete API. When you set it to true, we'd tell Kubernetes to use DeletePropagationOrphan instead of cascade deletion. This way:

• The ScheduledWorkflow gets deleted ✓
• But the workflows that are already running keep going ✓

And if you don't set it (or set it to false), everything works exactly like it does today - fully backward compatible.

I'm thinking:

1. Update the proto files** to add the new parameter to both v1beta1 and v2beta1:

message DeleteJobRequest {
string id = 1;
bool orphan_dependents = 2; // New optional field
}

2. Modify ResourceManager.DeleteJob() to actually use it:

deleteOptions := &v1.DeleteOptions{}
if orphanDependents {
propagationPolicy := v1.DeletePropagationOrphan
deleteOptions.PropagationPolicy = &propagationPolicy
}
err = r.getScheduledWorkflowClient(k8sNamespace).Delete(ctx, job.K8SName, deleteOptions)

3. Update the server handlers to pass the parameter through from the API request.

4. Tests - Update the existing tests to work with the new signature, and add a test that verifies the orphan behavior actually sets the right propagation policy.

Let me know what you think! Happy to adjust the approach based on your feedback before I start writing code.

Thanks for maintaining this awesome project - excited to contribute!

[JerT33]

@GKR5413

Please see #12574 that I am already currently working on