feat: Updating datanode and graylog health check and starup probes

[alix-graylog]

Summary

Replaces the chart's TCP-only probes with checks that answer "this pod can serve" instead of "a port is open". The readiness probe is a StatefulSet's only rollout throttle, so a probe that goes green too early lets a rolling update move on to the next pod while the previous one is still recovering. Closes the G-02 half of #150; G-03 already landed in #147.

Details

• New templates/workload/containers/_probes.tpl renders every probe for both tiers. Any probe block may carry its own httpGet, tcpSocket, exec or grpc handler, which replaces the chart default for that probe. Two handlers on one probe fail the render rather than silently picking one.
• Graylog readiness and startup use httpGet /api/system/lbstatus, with scheme following graylog.config.tls.enabled. Liveness stays a TCP connect on the app port on purpose: DEAD means "stop sending me traffic", so liveness on lbstatus would have kubelet restart a pod that is draining or parked for maintenance.
• Data Node readiness and startup move to a TCP connect on the OpenSearch port (9200). Liveness stays on the Data Node API port (8999).
• startupProbe added to both tiers at failureThreshold: 30 x periodSeconds: 10, so a first-node MongoDB migration or a large journal replay cannot eat the liveness budget.
• datanode.podManagementPolicy exposed, defaulting to OrderedReady.
• successThreshold is no longer rendered on liveness or startup probes, because Kubernetes requires 1 and rejects any other value. The values key still validates, so existing values files keep working.
• 15 new tests in tests/probes_test.yaml. Suite is now 29 suites / 368 tests.
• Docs: a Health Probes section in the chart README, and the lb_status drain note in docs/graylog-message-handling.md updated now that readiness actually acts on it.

Two things the implementation revised

• There is no Data Node health endpoint to probe. Both 8999 and 9200 serve HTTPS with authentication (401 on /, 404 elsewhere), and the image ships no HTTP client for an exec probe. Readiness proves OpenSearch is listening; it cannot gate on shard recovery or cluster-green. Documented as a limitation.
• podManagementPolicy is immutable. Defaulting it to Parallel would make helm upgrade fail on every existing release, so it ships opt-in with the --cascade=orphan recreate procedure documented.

Linked issues

This fixes #150 (G-02 half; G-03 landed in #147). Also covers U-01 and U-04 from
the production readiness review.

PR Checklist

Please check the items that apply to your change.

• Tests added/updated
• Documentation updated
• This PR includes a new feature
• This PR includes a bugfix
• This PR includes a refactor

Testing Checklist

Static Validation

• Linter check passes: helm lint ./charts/graylog
• Helm renders local template sucessfully: helm template graylog ./charts/graylog --validate

Installation

• Fresh installation completes successfully: helm install graylog ./charts/graylog
• All pods reach Running state: kubectl rollout status statefulset/graylog
• Helm tests pass: helm test graylog

Functional (if applicable)

• Web UI accessible and login works
• DataNodes visible in System > Cluster Configuration
• Inputs can be created and receive data

Upgrade (if applicable)

• Upgrade from previous release succeeds
• Scaling up/down works correctly
• Configuration changes apply correctly

Specific to this PR

• Measured the Data Node probe gap on a live cluster. Before: Kubernetes marked the pod Ready at t=21s with only 8999 bound, while OpenSearch did not bind 9200 until t=55s, so the probe went green 34 seconds before OpenSearch was listening. After: the same restart stayed Ready=false through t=58s and went Ready at t=70s with 9200 serving.
• Verified the lb_status drain end to end. PUT .../override/dead made readiness fail and the pod left the Service endpoints in about 50 seconds with 0 restarts, confirming the deliberate TCP-liveness split keeps kubelet from killing a parked pod. override/alive returned it to Ready within 10 seconds.
• Confirmed podManagementPolicy is rejected on a live StatefulSet, which is why Parallel is not the default.
• Confirmed the two-handler guard fails the render.

Notes for reviewers

• Verify all applicable tests above pass
• Validate that the linked issues are no longer reproducible, if applicable
• Sync up with the author before merging
• The commit history should be preserved - use rebase-merge or standard merge options when applicable

One open question from #150

The issue asked what to do about THROTTLED. Graylog answers lbstatus with 429 when it is throttled, usually under journal pressure, and Kubernetes counts only 200-399 as success. So a throttled node now fails readiness and leaves the Service. That is correct backpressure for ingest, but the same Service carries the web port, so a cluster-wide throttle also takes the UI down. This PR keeps the plain httpGet and does not treat 429 as ready. Worth a decision before merge: accept it, or add the exec wrapper the issue floated as an opt-in.

Once the drain is active the pod is no longer reachable through the Service, so the override/alive call has to be addressed to the pod directly. That is documented, but it is the kind of thing worth knowing before you try it on a real
cluster.