Skip to content

feat(chart): add optional startupProbe for web and worker - #371

Open
PramaYudhistira wants to merge 1 commit into
langfuse:mainfrom
PramaYudhistira:feat/optional-startup-probe
Open

feat(chart): add optional startupProbe for web and worker#371
PramaYudhistira wants to merge 1 commit into
langfuse:mainfrom
PramaYudhistira:feat/optional-startup-probe

Conversation

@PramaYudhistira

Copy link
Copy Markdown

Problem

langfuse-web (and langfuse-worker) run only a livenessProbe + readinessProbe. If the container takes longer to boot than the liveness budget — initialDelaySeconds + failureThreshold × periodSeconds, ~50s with the chart defaults (20 + 3×10) — the kubelet SIGTERM-kills it mid-boot (exit code 143) before /api/public/health can answer. It restarts, boots slowly again, gets killed again → a permanent CrashLoopBackOff, never reaching Ready.

This is easy to hit after an app-version bump that lengthens startup (e.g. a longer DB/ClickHouse migration or a cold start under node contention). The chart exposes no startupProbe, so today the only workaround is loosening the livenessProbe — which also weakens genuine hang detection.

Change

Add an opt-in startupProbe (default enabled: false, so this is a no-op for existing installs) to both the web and worker deployments. When enabled, Kubernetes runs only the startup probe until it first succeeds and does not run liveness/readiness (and cannot restart the pod) until then — giving a slow first boot a generous window while keeping the liveness probe tight.

  • values.yaml: langfuse.web.startupProbe and langfuse.worker.startupProbe. Defaults give a ~5-minute startup budget when enabled (failureThreshold: 30 × periodSeconds: 10).
  • templates/web/deployment.yaml and templates/worker/deployment.yaml: render the probe only when enabled.
  • tests/startup-probe_test.yaml: helm-unittest suite covering default-off and enabled-on for both deployments.
  • README regenerated via helm-docs; chart version 1.5.35 → 1.6.0.

Backward compatibility

Default-disabled → rendered output is unchanged for existing values. Verified with helm template (no startupProbe by default; present with the configured values when enabled).

Testing

  • helm lint charts/langfuse/ -f charts/langfuse/values.lint.yaml → passes
  • helm unittest charts/langfuse/ → 12 suites / 65 tests pass (incl. the new suite)
  • helm-docs → README in sync

Example usage:

langfuse:
  web:
    startupProbe:
      enabled: true
  worker:
    startupProbe:
      enabled: true

When langfuse-web or langfuse-worker takes longer to boot than the liveness
probe budget (initialDelaySeconds + failureThreshold * periodSeconds), the
kubelet SIGTERM-kills the container mid-boot before it can become ready, so it
never starts and lands in a permanent CrashLoopBackOff. The chart exposes no
startupProbe, so the only workaround is loosening the livenessProbe, which
weakens hang detection.

Add an opt-in startupProbe (default disabled, so no behavior change) for both
the web and worker deployments. When enabled, Kubernetes gates the liveness and
readiness probes until startup first succeeds, giving a slow first boot
(database/ClickHouse migrations, cold start) a generous window without
compromising the liveness probe.

- values.yaml: langfuse.web.startupProbe and langfuse.worker.startupProbe
  (default disabled; ~5 min budget via failureThreshold 30 x periodSeconds 10)
- web/worker deployment templates render the probe only when enabled
- add a helm-unittest suite and regenerate the README via helm-docs
- bump chart version 1.5.35 -> 1.6.0
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jun 22, 2026
@CLAassistant

CLAassistant commented Jun 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@PramaYudhistira

Copy link
Copy Markdown
Author

For context on how this surfaced — we hit it in production right after bumping the Langfuse app version (3.137.0 → 3.185.0). One langfuse-web pod went into a permanent CrashLoopBackOff (~1,400 restarts over ~3 days) while the Deployment sat at 1/2 ready and only paged when the surviving replica briefly dropped too.

What the data showed:

  • Last State: Terminated, Reason: Error, Exit Code: 143 → SIGTERM (kubelet kill), not OOM (that would be 137).
  • The container lived ~48s each time — right at the liveness deadline (initialDelaySeconds 20 + failureThreshold 3 × periodSeconds 10 ≈ 50s).
  • kubectl logs --previous showed a successful boot — Prisma "No pending migrations to apply", Next.js "Ready", init scripts, ClickHouse connected — and then death.

So nothing was actually wrong with the app. The newer version just boots a bit slower than the (unchanged) liveness window, and the kubelet was killing it mid-boot every single time — the timeout had never changed, the slower boot simply crossed it.

Our short-term fix was to widen the liveness window, but that weakens genuine hang detection, and there was no startupProbe to reach for — which is what motivated this PR. With an opt-in startupProbe, startup gets its own generous budget and the liveness probe can stay tight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants