KeeperCluster StatefulSet infinite restart loop due to K8s API server default field drift

[brightsparc]

Company or project name

Introspection (introspection.dev) — AI observability platform using self-hosted ClickHouse via the operator.

Describe what's wrong

The KeeperCluster reconciler enters an infinite restart loop because the desired StatefulSet spec omits fields that the Kubernetes API server fills with defaults. On each reconcile cycle, the operator detects a diff between the desired state (nil/zero values) and the actual state (K8s-defaulted values), concludes config has changed, and force-restarts the keeper pod via the kubectl.kubernetes.io/restartedAt annotation. This annotation change itself creates a new diff on the next reconcile, making the loop self-reinforcing.

Does it reproduce on the most recent release?

Yes

How to reproduce

Keeper pod is killed and recreated every ~10-30 seconds indefinitely.

The operator logs show:

  INFO  keeper  forcing Pod restart, because of config changes
  INFO  keeper  updating replica StatefulSet

Expected behavior

Once the keeper is running and config hasn't actually changed, the pod should remain stable.

Error message and/or stacktrace

The operator's templateStatefulSet() and templatePodSpec() functions build a desired spec with several fields left as nil/zero:

Field	Desired (operator)	Actual (K8s-defaulted)
spec.template.spec.terminationGracePeriodSeconds	nil	30
spec.template.spec.schedulerName	""	"default-scheduler"
spec.template.spec.securityContext	nil	{}
spec.updateStrategy.rollingUpdate.partition	nil	0
spec.updateStrategy.rollingUpdate.maxUnavailable	nil	1
spec.persistentVolumeClaimRetentionPolicy	nil	{whenDeleted: Retain, whenScaled: Retain}
Liveness probe successThreshold	0	1

When DeepHashObject() hashes the desired spec vs the actual spec from K8s, the hashes never match. This triggers the config-change detection at resources.go:371, which sets restartedAt to time.Now(), which changes the pod template hash, which triggers another update on the next reconcile.

Additional context

Affected files

• internal/controller/keeper/templates.go — templateStatefulSet() and templatePodSpec()
• internal/controller/clickhouse/templates.go — same functions (same pattern)
• internal/controller/constants.go — DefaultLivenessProbeSettings missing SuccessThreshold: 1
• internal/controller/resources.go — ReconcileReplicaResources() where the diff is detected

Suggested fix

Explicitly set K8s-defaulted fields in the desired spec so they match what the API server returns:

1. Set SuccessThreshold: 1 on DefaultLivenessProbeSettings
2. In templatePodSpec(), default terminationGracePeriodSeconds to 30, schedulerName to "default-scheduler", and securityContext to &PodSecurityContext{}
3. In templateStatefulSet(), set RollingUpdate.Partition to 0, RollingUpdate.MaxUnavailable to 1, and PersistentVolumeClaimRetentionPolicy to Retain/Retain

[GrigoryPervakov]

@brightsparc Can you show your KeeperCluster spec?
Fields filled by the STS controller should not affect it, as the operator stores the computed spec hash in the annotations.
I can't reproduce it with the default settings

[svils]

@GrigoryPervakov I can confirm this. You're right that checksum/spec works — in my case it matched statefulSetRevision fine. The problem is checksum/configuration which gets stuck at a stale value and never converges.

Steps to reproduce

1. Deploy a KeeperCluster, let it stabilize (all conditions True, ConfigurationInSync: True)
2. Delete the controller-manager pod (or scale the deployment to 0 and back to 1)
3. Wait for the new controller-manager to start reconciling
4. The keeper pod enters an infinite restart loop — operator keeps killing it every ~20-30s

The trigger is the controller-manager restart. The new instance computes a configurationRevision that differs from the checksum/configuration annotation stored on the StatefulSet. From that point, the operator force-restarts the pod on every reconciliation and the annotation never converges.

What's actually happening — two bugs compounding

Bug 1: Spec defaulting causes an STS update on every reconciliation

templateStatefulSet() / templatePodSpec() produce a desired spec with nil/zero fields. After Update(), the API server fills defaults. Next reconcile, operator sees a diff, calls Update() again. Every cycle. The diff looks like this:

- SuccessThreshold:              1,       (K8s default)
+ SuccessThreshold:              0,       (operator sets)

- TerminationGracePeriodSeconds: &30,
+ TerminationGracePeriodSeconds: nil,

- SchedulerName:         "default-scheduler",
+ SchedulerName:         "",

- PersistentVolumeClaimRetentionPolicy: &{Retain, Retain},
+ PersistentVolumeClaimRetentionPolicy: nil,

This alone doesn't cause restarts — checksum/spec is annotation-based and stable. But it creates constant STS churn.

Bug 2: Config hash annotation update gets lost in 409 Conflicts

When ReconcileReplicaResources() detects a config hash mismatch (resources.go:371), it does all of this in one Update() call:

1. Sets restartedAt to time.Now() (line 377)
2. Updates checksum/configuration annotation (line 379)
3. Replaces ExistingSTS.Spec with desired spec (line 436) — reintroducing the diffs from Bug 1
4. Calls Update() (line 441)

Because the STS is being modified so frequently (operator + StatefulSet controller both touching it), step 4 often fails with:

ERROR: Operation cannot be fulfilled on statefulsets.apps: the object has been modified

The checksum/configuration update from step 2 is lost. Next reconciliation reads the STS fresh, sees the stale annotation, tries again, conflicts again. Forever.

Operator logs confirm:

INFO   forcing Pod restart, because of config changes
INFO   updating replica StatefulSet
ERROR  update conflict for resource, reschedule to retry

Workaround

Patch the STS annotation manually:

VALUE=$(kubectl get keepercluster <name> -n <ns> -o jsonpath='{.status.configurationRevision}')
kubectl annotate sts <name>-keeper-0 -n <ns> checksum/configuration=$VALUE --overwrite

Loop breaks immediately.

Suggested fix

1. Bug 1 — Set K8s-defaulted fields explicitly in templates: SuccessThreshold: 1, TerminationGracePeriodSeconds: 30, SchedulerName: "default-scheduler", etc.

2. Bug 2 — Don't bundle the config hash annotation update with the full spec replacement in one Update(). Either separate them, or use server-side apply.

[GrigoryPervakov]

The initial description is also because of config changes.

2. should be fixed by wrapping the Update call into a RetryOnConflict

I'll check if the generated config hash is idempotent, as it may cause frequent Pod restarts.

I don't like the approach of including defaults in the initial spec. If future k8s versions introduce new default fields, we'll face the same issue, and it'll require k8s version-specific logic in the operator.

[brightsparc]

The initial description is also because of config changes.

2. should be fixed by wrapping the Update call into a RetryOnConflict

I'll check if the generated config hash is idempotent, as it may cause frequent Pod restarts.

I don't like the approach of including defaults in the initial spec. If future k8s versions introduce new default fields, we'll face the same issue, and it'll require k8s version-specific logic in the operator.

I hit this issue initially when attempting to trigger a clickhouse pod restart based on a change to an annotation for checksum/config and/or secret changes

[GrigoryPervakov]

@brightsparc, @svils, can you check if the issue still reproduces with a build after the mentioned PR