Skip to content

Commit 96562d4

Browse files
authored
fix/indexed-search: add livenessProbe to zoekt-webserver to prevent hung pods (#850)
zoekt-webserver was unhealthy for long time ``` NAME READY STATUS RESTARTS AGE indexed-search-0 1/2 Running 0 6h18m indexed-search-1 1/2 Running 0 6h19m indexed-search-2 1/2 Running 0 6h21m indexed-search-3 1/2 Running 0 6h23m indexed-search-4 1/2 Running 0 6h25m indexed-search-5 1/2 Running 0 6h27m indexed-search-6 1/2 Running 0 6h29m indexed-search-7 1/2 Running 0 6h31m ``` ``` Warning Unhealthy 4m20s (x3300 over 4h39m) kubelet Readiness probe failed: Get "http://192.168.11.20:6070/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ``` ref https://linear.app/sourcegraph/issue/PLAT-509/incident-indexed-search-pods-were-unhealthy-for-long-time PLAT-509 ### Checklist - [x] Follow the [manual testing process](https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/TEST.md) - [x] Update [changelog](https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/CHANGELOG.md) - [ ] Update [Kubernetes update doc](https://docs.sourcegraph.com/admin/updates/kubernetes) ### Test plan - [x] Manual Verification - local with kind cluster <img width="1920" height="1080" alt="Screenshot 2026-03-27 at 11 52 41 AM" src="https://github.com/user-attachments/assets/173876e7-8ca3-476b-ac6b-63cc1c29a76c" /> - [x] Tested against cloud-dev-qa with mi2 generate kustomize and kustomize apply
1 parent 1b6f92b commit 96562d4

2 files changed

Lines changed: 10 additions & 0 deletions

File tree

charts/sourcegraph/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Use `**BREAKING**:` to denote a breaking change
88

99
## Unreleased
1010

11+
- Added livenessProbe to zoekt-webserver in indexed-search to detect and restart hung pods
1112
- Fix Pod Disruption Budget for sourcegraph-frontend
1213
- Added a startup probe to the gitserver statefulset to give it time to run the on-disk migration from repo names to repo IDs
1314
- The repo-updater service is no longer needed and has been removed from the chart.

charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,15 @@ spec:
6363
ports:
6464
- name: grpc
6565
containerPort: 6070
66+
livenessProbe:
67+
httpGet:
68+
path: /healthz
69+
port: grpc
70+
scheme: HTTP
71+
initialDelaySeconds: 120
72+
timeoutSeconds: 5
73+
periodSeconds: 60
74+
failureThreshold: 10
6675
readinessProbe:
6776
failureThreshold: 3
6877
httpGet:

0 commit comments

Comments
 (0)