Skip to content

fix: adopt custom-runner v1.0.0 - #2311

Merged
csatib02 merged 4 commits into
masterfrom
feat/custom-runner-v1
Aug 10, 2026
Merged

fix: adopt custom-runner v1.0.0#2311
csatib02 merged 4 commits into
masterfrom
feat/custom-runner-v1

Conversation

@csatib02

@csatib02 csatib02 commented Aug 7, 2026

Copy link
Copy Markdown
Member

Adopts custom-runner v1.0.0,
which changes the HTTP contract and moves metrics to their own listener.

Note

The operator now passes -metrics-port, which only exists in custom-runner
v1.0 — Go rejects unknown flags, so an older runner exits immediately with
flag provided but not defined. CI rebuilds and republishes the sidecar images
from the merge commit (type=raw,value=latest,enable={{is_default_branch}}), so
the operator and its images move together automatically. This only bites someone
who pins a sidecar image below v1.0.0 by hand.

What breaks without this

Pods running two listeners on 9533 crash-loop. v1.0 serves metrics on a
separate listener defaulting to 9533:

Pod custom-runner containers other :9533 listener collides
syslog-ng config-reloader, buffer-metrics-sidecar — (with each other) when buffer metrics are enabled
fluentd buffer-metrics-sidecar config-reloader (always present) when buffer metrics are enabled
fluentbit buffer-metrics-sidecar config-reloader (opt-in, configHotReload) only with configHotReload
metrics server: listen tcp 0.0.0.0:9533: bind: address already in use

The fluentd drain never stopped the node exporter. drain-watch.sh calls
GET /exit; v1.0 matches methods, so that returns 405.

Changes

  • drain-watch.sh uses -X POST for /exit.
  • Buffer-metrics sidecars pass --metrics-port 0. Nothing scraped the runner's
    own metrics there before, so this preserves current behaviour.
  • The syslog-ng config-reloader publishes metrics on ConfigReloaderMetricsPort.
    This fixes an existing bug: syslogng/service.go already points a
    ServiceMonitor at 9533, but the runner served metrics on its command port, so
    that target has never returned data.
  • The syslog-ng config-reloader gains a readiness probe on /readyz, new in
    v1.0. It returns 503 naming the path when a configured watch fails to register
    — the failure where the container stays Running while silently never reloading
    again. Behaviour change: a config volume mounted somewhere unexpected now
    blocks pod readiness instead of degrading silently.
  • Image pins moved to v1.0.0.

Compatibility audit

Every breaking change in the v1.0
MIGRATION.md:

Breaking change Impact here
Mutating verbs require POST drain-watch /exit, fixed. No other mutating calls
Metrics on their own listener Fixed in all three workloads
Command API binds loopback Safe. Only reached from inside the pod; probes target buffer-metrics (9200, node_exporter), and 7357/7358 appear in no Service or manifest
Process keys validated nodeexporter, buffersize, info, reload all match
Images run as uid 65534 N/A — the binary is copied into images that set their own USER
Single-object API responses N/A — drain-watch never parses a body
Config rejects unknown keys v1.0 accepts the exact -cfgjson from generateConfigReloaderConfig
/config returns typed output Not used

Verification

On a KIND cluster, operator built from this branch, sidecar images rebuilt on
custom-runner v1.0:

  • go build ./..., go test ./pkg/...
  • No restarts. cr-test-syslog-ng-0 3/3 Running, 0 restarts; fluentbit
    2/2 with buffer-metrics-sidecar, 0 restarts
  • Rendered args are correctconfig-reloader: [-metrics-port 9533 -cfgjson ...],
    buffer-metrics-sidecar: [--metrics-port 0 --exec ...]
  • /metrics on 9533 serves sidecar_reloader_* from the live
    config-reloader — the ServiceMonitor target that previously returned nothing
  • /readyz returns 200 on the live config-reloader; the probe passes
  • Collision and fix reproduced directly: a second runner in the same
    network namespace fails with bind: address already in use; with
    --metrics-port 0 it starts clean
  • v1.0 accepts the generated -cfgjson (negative control: a typo is rejected)
  • fluentd drain end to end — ran the real drain-watch.sh against a live
    v1.0 runner with a stubbed fluentd RPC: the script polls the runner, posts
    /exit, kills workers and exits 0, and the runner exits 0 with it.
    Negative control: the previous GET /exit returns 405 and leaves the
    runner running, i.e. the node exporter was never being stopped

custom-runner v1.0 matches methods; mutating verbs are POST and GET /exit now
returns 405, so the drain never shut the node exporter down.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
v1.0 serves metrics on their own listener, defaulting to 9533. Every pod that
runs a runner already has something on that port, so each container would fail
to bind and crash-loop:

  syslog-ng  config-reloader + buffer-metrics-sidecar, both runners
  fluentd    buffer-metrics-sidecar alongside config-reloader, which owns :9533
  fluentbit  the same pairing as fluentd

The buffer-metrics sidecars disable runner metrics, matching the previous
behaviour where nothing scraped them. The syslog-ng config-reloader publishes on
ConfigReloaderMetricsPort, so the ServiceMonitor that already targets 9533
finally receives data — it previously scraped a port the runner never served.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
Signed-off-by: Bence Csati <bence.csati@axoflow.com>
@csatib02 csatib02 self-assigned this Aug 7, 2026
@csatib02 csatib02 added the enhancement New feature or request label Aug 7, 2026
v1.0 exposes /readyz, which reports 503 naming the path when a configured watch
fails to register. That is the failure this container is prone to: if the config
volume is not mounted where it expects, it stays Running and healthy while
silently never reloading again.

Wiring it to the readiness probe turns that into a visible pod condition.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
@csatib02
csatib02 merged commit 36db72a into master Aug 10, 2026
33 checks passed
@csatib02
csatib02 deleted the feat/custom-runner-v1 branch August 10, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants