feat(chart): add optional startupProbe for web and worker - #371
feat(chart): add optional startupProbe for web and worker#371PramaYudhistira wants to merge 1 commit into
Conversation
When langfuse-web or langfuse-worker takes longer to boot than the liveness probe budget (initialDelaySeconds + failureThreshold * periodSeconds), the kubelet SIGTERM-kills the container mid-boot before it can become ready, so it never starts and lands in a permanent CrashLoopBackOff. The chart exposes no startupProbe, so the only workaround is loosening the livenessProbe, which weakens hang detection. Add an opt-in startupProbe (default disabled, so no behavior change) for both the web and worker deployments. When enabled, Kubernetes gates the liveness and readiness probes until startup first succeeds, giving a slow first boot (database/ClickHouse migrations, cold start) a generous window without compromising the liveness probe. - values.yaml: langfuse.web.startupProbe and langfuse.worker.startupProbe (default disabled; ~5 min budget via failureThreshold 30 x periodSeconds 10) - web/worker deployment templates render the probe only when enabled - add a helm-unittest suite and regenerate the README via helm-docs - bump chart version 1.5.35 -> 1.6.0
|
For context on how this surfaced — we hit it in production right after bumping the Langfuse app version (3.137.0 → 3.185.0). One What the data showed:
So nothing was actually wrong with the app. The newer version just boots a bit slower than the (unchanged) liveness window, and the kubelet was killing it mid-boot every single time — the timeout had never changed, the slower boot simply crossed it. Our short-term fix was to widen the liveness window, but that weakens genuine hang detection, and there was no |
Problem
langfuse-web(andlangfuse-worker) run only alivenessProbe+readinessProbe. If the container takes longer to boot than the liveness budget —initialDelaySeconds + failureThreshold × periodSeconds, ~50s with the chart defaults (20 + 3×10) — the kubeletSIGTERM-kills it mid-boot (exit code 143) before/api/public/healthcan answer. It restarts, boots slowly again, gets killed again → a permanentCrashLoopBackOff, never reaching Ready.This is easy to hit after an app-version bump that lengthens startup (e.g. a longer DB/ClickHouse migration or a cold start under node contention). The chart exposes no
startupProbe, so today the only workaround is loosening thelivenessProbe— which also weakens genuine hang detection.Change
Add an opt-in
startupProbe(defaultenabled: false, so this is a no-op for existing installs) to both the web and worker deployments. When enabled, Kubernetes runs only the startup probe until it first succeeds and does not run liveness/readiness (and cannot restart the pod) until then — giving a slow first boot a generous window while keeping the liveness probe tight.values.yaml:langfuse.web.startupProbeandlangfuse.worker.startupProbe. Defaults give a ~5-minute startup budget when enabled (failureThreshold: 30×periodSeconds: 10).templates/web/deployment.yamlandtemplates/worker/deployment.yaml: render the probe only whenenabled.tests/startup-probe_test.yaml: helm-unittest suite covering default-off and enabled-on for both deployments.1.5.35 → 1.6.0.Backward compatibility
Default-disabled → rendered output is unchanged for existing values. Verified with
helm template(nostartupProbeby default; present with the configured values when enabled).Testing
helm lint charts/langfuse/ -f charts/langfuse/values.lint.yaml→ passeshelm unittest charts/langfuse/→ 12 suites / 65 tests pass (incl. the new suite)helm-docs→ README in syncExample usage: