fix: adopt custom-runner v1.0.0 - #2311
Merged
Merged
Conversation
custom-runner v1.0 matches methods; mutating verbs are POST and GET /exit now returns 405, so the drain never shut the node exporter down. Signed-off-by: Bence Csati <bence.csati@axoflow.com>
v1.0 serves metrics on their own listener, defaulting to 9533. Every pod that runs a runner already has something on that port, so each container would fail to bind and crash-loop: syslog-ng config-reloader + buffer-metrics-sidecar, both runners fluentd buffer-metrics-sidecar alongside config-reloader, which owns :9533 fluentbit the same pairing as fluentd The buffer-metrics sidecars disable runner metrics, matching the previous behaviour where nothing scraped them. The syslog-ng config-reloader publishes on ConfigReloaderMetricsPort, so the ServiceMonitor that already targets 9533 finally receives data — it previously scraped a port the runner never served. Signed-off-by: Bence Csati <bence.csati@axoflow.com>
Signed-off-by: Bence Csati <bence.csati@axoflow.com>
v1.0 exposes /readyz, which reports 503 naming the path when a configured watch fails to register. That is the failure this container is prone to: if the config volume is not mounted where it expects, it stays Running and healthy while silently never reloading again. Wiring it to the readiness probe turns that into a visible pod condition. Signed-off-by: Bence Csati <bence.csati@axoflow.com>
OverOrion
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts custom-runner v1.0.0,
which changes the HTTP contract and moves metrics to their own listener.
Note
The operator now passes
-metrics-port, which only exists in custom-runnerv1.0 — Go rejects unknown flags, so an older runner exits immediately with
flag provided but not defined. CI rebuilds and republishes the sidecar imagesfrom the merge commit (
type=raw,value=latest,enable={{is_default_branch}}), sothe operator and its images move together automatically. This only bites someone
who pins a sidecar image below
v1.0.0by hand.What breaks without this
Pods running two listeners on 9533 crash-loop. v1.0 serves metrics on a
separate listener defaulting to
9533::9533listenerconfig-reloader,buffer-metrics-sidecarbuffer-metrics-sidecarconfig-reloader(always present)buffer-metrics-sidecarconfig-reloader(opt-in,configHotReload)configHotReloadThe fluentd drain never stopped the node exporter.
drain-watch.shcallsGET /exit; v1.0 matches methods, so that returns405.Changes
drain-watch.shuses-X POSTfor/exit.--metrics-port 0. Nothing scraped the runner'sown metrics there before, so this preserves current behaviour.
config-reloaderpublishes metrics onConfigReloaderMetricsPort.This fixes an existing bug:
syslogng/service.goalready points aServiceMonitor at 9533, but the runner served metrics on its command port, so
that target has never returned data.
config-reloadergains a readiness probe on/readyz, new inv1.0. It returns 503 naming the path when a configured watch fails to register
— the failure where the container stays Running while silently never reloading
again. Behaviour change: a config volume mounted somewhere unexpected now
blocks pod readiness instead of degrading silently.
v1.0.0.Compatibility audit
Every breaking change in the v1.0
MIGRATION.md:
POSTdrain-watch/exit, fixed. No other mutating callsbuffer-metrics(9200, node_exporter), and 7357/7358 appear in no Service or manifestnodeexporter,buffersize,info,reloadall matchUSERdrain-watchnever parses a body-cfgjsonfromgenerateConfigReloaderConfig/configreturns typed outputVerification
On a KIND cluster, operator built from this branch, sidecar images rebuilt on
custom-runner v1.0:
go build ./...,go test ./pkg/...cr-test-syslog-ng-03/3 Running, 0 restarts; fluentbit2/2 with
buffer-metrics-sidecar, 0 restartsconfig-reloader: [-metrics-port 9533 -cfgjson ...],buffer-metrics-sidecar: [--metrics-port 0 --exec ...]/metricson 9533 servessidecar_reloader_*from the liveconfig-reloader — the ServiceMonitor target that previously returned nothing
/readyzreturns 200 on the live config-reloader; the probe passesnetwork namespace fails with
bind: address already in use; with--metrics-port 0it starts clean-cfgjson(negative control: a typo is rejected)drain-watch.shagainst a livev1.0 runner with a stubbed fluentd RPC: the script polls the runner, posts
/exit, kills workers and exits 0, and the runner exits 0 with it.Negative control: the previous
GET /exitreturns405and leaves therunner running, i.e. the node exporter was never being stopped