Bug: Bootstrap taking longer than the startup probe failure threshold

[nkolev-mparticle]

Summary

The startup probe configuration is currently hardcoded in the operator and cannot be customized through the ScyllaCluster CRD. This causes issues during node replacement operations on large clusters where the default timeout (~6.7 minutes) is insufficient.

Current Behavior

The startup probe is hardcoded in pkg/controller/scylladbdatacenter/resource.go:

StartupProbe: &corev1.Probe{
    TimeoutSeconds:   int32(30),
    FailureThreshold: int32(40),   // 40 × 10s = 400s = ~6.7 min
    PeriodSeconds:    int32(10),
    // ...
}

Problem

During node replacement (--replace-node-first-boot) on clusters with large data volumes (e.g., 17TB+), the node needs to:

1. Scan and verify existing data directories
2. Reshape SSTables if needed
3. Join gossip
4. Stream data from replicas

This process can take significantly longer than 6.7 minutes, causing the startup probe to fail and the pod to restart repeatedly in a crash loop.

Proposed Solution

Expose startup probe configuration in the ScyllaCluster CRD, similar to how resources and other pod-level settings are configurable. For example:

apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
  datacenter:
    racks:
      - name: rack1
        startupProbe:
          failureThreshold: 360    # 1 hour
          periodSeconds: 10
          timeoutSeconds: 30

Or at the cluster level:

spec:
  startupProbe:
    failureThreshold: 360

Workaround

Currently, the only workaround is to manually patch the StatefulSet after deployment:

kubectl patch statefulset <sts-name> -n <namespace> --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/startupProbe/failureThreshold", "value": 360}]'

This is error-prone and may be overwritten by the operator during reconciliation.

Environment

• Operator version: v1.18.0
• Scylla version: 2025.4.5-enterprise
• Cluster size: 24 nodes, ~17TB per node
• Use case: Node replacement after hardware failure

Additional Context

Related issue: #844 (probe timeouts during overload)

Thank you for considering this feature request!

[ylebi]

Hey, thanks for the detailed report, very helpful to have the environment details and the concrete data size.

Before we go the configurable route, I'd like to understand this a bit more. Could you collect and share the must-gather output from when the startup probe was failing? You can find the instructions here. That would help us figure out if we're looking at a genuine timing issue or something else going on.

Also, what values did you have in mind? You mentioned failureThreshold: 360 (1 hour) in the example, is that what worked for you in practice, or is it a rough estimate?

The reason I'm asking before committing to the API approach is that we might be better off just raising the hardcoded defaults to something more reasonable for large clusters. Exposing these as user-configurable fields adds surface area we'd need to maintain long-term, and we have some upcoming work that may affect how the sidecar and probes interact internally. If the right answer is "the default window should be 2 hours" rather than "every user should tune it themselves", that's a simpler fix with less risk.

[nkolev-mparticle]

Before we go the configurable route, I'd like to understand this a bit more. Could you collect and share the must-gather output from when the startup probe was failing? You can find the instructions here. That would help us figure out if we're looking at a genuine timing issue or something else going on.

Unfortunately, at this point all those logs, pods, nodes are gone (I have been working on resizing this cluster for 6 weeks now, a lot has changed). I can tell you that after the pod getting killed a couple of times after the 6.something minutes, and manually editing the parameters to 1 then 3 hours, it started up fine (with 3 hours). It took much less than 3 hours, I was just overshooting bc I got tired of retrying and waiting. If I am not mistaken, it took about an hour to boot up.

The reason I'm asking before committing to the API approach is that we might be better off just raising the hardcoded defaults to something more reasonable for large clusters. Exposing these as user-configurable fields adds surface area we'd need to maintain long-term, and we have some upcoming work that may affect how the sidecar and probes interact internally. If the right answer is "the default window should be 2 hours" rather than "every user should tune it themselves", that's a simpler fix with less risk.

This makes sense, 100%. Perhaps some dynamic way of detecting it would be a good middle ground? I don't know what else could cause slow startup, but if it is just how it manifested with me, perhaps basing it on the "table count + data size on disk"-based formula would address it. If not, 1 hour is probably a safe enough value that would not cause too much of a delay in unrelated cases.

[mflendrich]

Thanks @nkolev-mparticle, this is very helpful.

Streaming can definitely take longer than several minutes. I am a bit confused because we run node replace operations with streaming taking hours on a regular basis, and we would have run into this problem many many times in other environments. Which makes me a bit suspicious about the startup probe waiting for the end of streaming - will need to check if it is actually the case, and understand why we're getting inconsistent behavior across deployments in this regard.

[mflendrich]

Tracked in the internal Jira as OPERATOR-72.

@nkolev-mparticle we'll get back to you with the result of our internal investigation.

[nkolev-mparticle]

@mflendrich apologies for the confusion - when I mentioned the delay causing the pod to get killed, I did not mean it regarding streaming, but rather the initialization process, during which the node "discovers"/reads its data on disk, before it starts doing any remote operations. My suspicion there is that it has to do with the number of tables we have in the user keyspace in that datastore (close to 1500), though this is not based on any research, just a hunch.

As far as streaming, I don't have stability/aliveness issues - those nodes (a i7ie.24xlarge) take about 3 hours to join the 20-ish node ring, and the old ones (i3en.12xlarge) take anywhere from 12 to 36 hours to leave (without capping streaming or compaction throughput), and do not get killed.

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out

/lifecycle rotten