Stack finalizer blocks namespace deletion when workspace is unavailable

[guineveresaenger]

Description

When a namespace containing a Stack CR (with destroyOnFinalize: true) is deleted, Kubernetes cascades the deletion to all resources in the namespace — including the Workspace StatefulSet and its pods. The Stack finalizer (finalizer.stack.pulumi.com) then tries to run pulumi destroy but can't because the workspace is gone. The Stack gets stuck with its finalizer, blocking namespace deletion indefinitely.

Steps to reproduce

1. Create a namespace with a Stack CR that has destroyOnFinalize: true
2. kubectl delete namespace <namespace>
3. The namespace gets stuck in Terminating

Root cause

During finalization, the Stack controller attempts to create/ensure a workspace (line 952 of stack_controller.go) and waits for it to become ready before running the destroy update. But because the namespace is terminating, the workspace pods are being deleted concurrently and can never become ready. The reconciler retries indefinitely.

There is already a special case (lines 960-967) that removes the finalizer when the workspace is stalled during deletion with no prior successful update. But there's no equivalent handling for when the workspace is simply unavailable because the namespace is being torn down.

Workaround

Manually remove the finalizer:

kubectl patch stack <name> -n <namespace> --type=merge -p '{"metadata":{"finalizers":[]}}'

You may also need to patch Update CRs:

kubectl patch update -n <namespace> --type=merge -p '{"metadata":{"finalizers":[]}}' --all

Suggested fix

When the Stack is marked for deletion and the workspace cannot be created or is not ready, the controller should detect that the namespace is terminating (or that the workspace has been deleted and can't be recreated) and remove the finalizer after a timeout or immediately, since the destroy operation cannot proceed.