Stack stuck in StackProcessing after successful update (workspace pod continuously re-runs and leaks secrets)

[aashishshrestha09]

What happened?

Despite the stack completing successfully (result: succeeded), the Stack CR status remains Ready: False and stays in StackProcessing. This causes the workspace pod to continuously re-run, repeatedly syncing outputs and creating a new *-stack-outputs secret on each cycle.

With resyncFrequencySeconds: 60, this results in a new secret every ~60 seconds, unbounded accumulation of secrets, and excessive API calls.

Setting resyncFrequencySeconds: 0 does not fully resolve the issue (new secrets continue to be created), though at a reduced rate. Something is still re-enqueuing the stack for reconciliation even when periodic resyncs are disabled. We suspect this may be related to workspace pod watch events, but have not confirmed the exact trigger.

Example

apiVersion: auto.pulumi.com/v1alpha1
kind: Stack
metadata:
  name: example-stack
  namespace: example-ns
spec:
  resyncFrequencySeconds: 60
  fluxSource:
    sourceRef:
      type: OCIRepository
      name: example-source
  # ... standard stack config

Observed behavior:

# Stack status shows succeeded but keeps processing
$ kubectl get stacks -n example-ns
NAME             AGE   STATE
example-stack    2h    succeeded

# Secrets accumulate continuously (~1 per resync interval)
$ kubectl get secrets -n example-ns | grep -c stack-outputs
74

# Update records accumulate
$ kubectl get updates -n example-ns | wc -l
1132

# New secrets appear every ~60s even after successful completion
$ kubectl get secrets -n example-ns --sort-by=.metadata.creationTimestamp | grep stack-outputs | tail -5
example-stack-19d081acf7d-stack-outputs   Opaque   6   4m38s
example-stack-19d081c2795-stack-outputs   Opaque   6   3m14s
example-stack-19d081d6f5d-stack-outputs   Opaque   6   102s
example-stack-19d081e49ed-stack-outputs   Opaque   6   26s

With resyncFrequencySeconds: 0:

Secrets still appear, though less frequently. The stack is still being re-enqueued by something other than the periodic resync timer.

Output of pulumi about

• PKO version: 2.4.1
• Kubernetes: 1.33
• Pulumi program source: OCI image
• State backend: DIY

Additional context

Impact

• Unbounded *-stack-outputs secret accumulation (etcd pressure, potential quota issues)
• Excessive state backend API calls
• Unnecessary cloud provider API calls on each pulumi up (even though nothing changes)

What we've tried

Workaround	Result
resyncFrequencySeconds: 60	New secret every 60s, unbounded growth
resyncFrequencySeconds: 0	Reduced frequency but secrets still appear (something else triggers reconciliation)
Manual kubectl annotate to force reconcile	Stack completes but returns to StackProcessing
Deleting accumulated secrets	They reappear on next reconcile cycle

Potentially related

• #1105 — Stack processing race condition
• PRs Fix workspace watch spuriously aborting in-flight Updates #1155, Fix Update controller status write conflicts with SSA #1141, Migrate Workspace controller status writes to Server-Side Apply #1147, Fix Stack controller status write conflicts by migrating to Server-Side Apply #1152 — Partial fixes merged but issue persists in v2.4.1

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[aashishshrestha09]

We just tested with v2.5.1 and unfortunately the issue persists. We identified the same problem:

*-stack-outputs secrets are never garbage collected, each 60s resync creates a brand-new secret and old secrets are never cleaned up.

We also found that resyncFrequencySeconds: 0 doesn't disable resyncs (per the API docs it defaults to 60s), so there's no way to prevent the accumulation.

Even when a Stack reaches Ready: True with no pending changes, PKO continues to reconcile every 60 seconds indefinitely. Each reconcile runs a no-op pulumi up and creates a new *-stack-outputs Secret but never cleans up the previous one. Over time, these secrets accumulate unbounded.

What we expected:
Setting resyncFrequencySeconds: 0 on the Stack spec would disable periodic resyncs entirely. Instead, PKO treats 0 as the default (60s), so there's no way to opt out of resyncs.

[guineveresaenger]

Hi @aashishshrestha09 - I've taken a look and I've discovered the following:

You describe your stack as stuck in StackProcessing. I am unable to reproduce a stuck stack, on either 2.5.1, or on 2.6.0. Can you tell me more about your setup/what you see/how you get to a truly stuck stack?

When using a setup I hope is similar to yours, my stack goes Ready: True, stays there for the default 60s, and then briefly flaps to False/StackProcessing. Secrets accumulate in the way you describe. (Note these will get GC'd after 24h, but it's obviously not something you'd see in real time.)

This only happens, however, if I have continueResyncOnCommitMatch: true set. When it is set to true, the resyncFrequencySeconds determine how often resyncs happen.
60s is the minimum period for a resync - anything smaller than that and we'd be resyncing so often that we'd probably get race conditions all over. That's why setting it to 0 does nothing. If you want to resync less often, you'd set this field to a much higher number. If you don't want to resync at all, you'd set continueResyncOnCommitMatch to false (or not set it, false being default).

On my repro, unsetting continueResyncOnCommitMatch stops the perpetual recreation, and stack output secrets do not accumulate. Setting resyncFrequencySeconds to 300 makes my stack only recreate every 5 minutes. This is meant for reconciliation of Pulumi state.

Can you try the following:

1. kubectl get stack <name> -n <ns> -o jsonpath='{.spec.continueResyncOnCommitMatch} {.spec.resyncFrequencySeconds}' - this will tell you if your Stack has continueResyncOnCommitMatch set and you should unset it if so.
2. kubectl get stacks -n <namespace> -w - is your stack stuck in StackProcessing, or does it flap between that and Ready?
3. (confidence check) Check if your workspace pod is healthy: kubectl describe pod <workspace-pod> -n <namespace>

I've opened #1176 to clarify the way in which the resync frequency works.