Skip to content

Make enabledIPv6 work without breaking single-stack clusters - #2310

Merged
csatib02 merged 13 commits into
masterfrom
fix/ipv6-service-families
Aug 10, 2026
Merged

Make enabledIPv6 work without breaking single-stack clusters#2310
csatib02 merged 13 commits into
masterfrom
fix/ipv6-service-families

Conversation

@csatib02

@csatib02 csatib02 commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

enabledIPv6 produced Services the cluster could not accept, and listeners that did not
match those Services. Three regressions landed after 6.7.0, all from one shared helper.

0321c1d2 changed the requested primary IP family from IPv4 to IPv6. Kubernetes does not
allow the primary family of an existing Service to change. It reports the refusal as
invalid, not as immutable, so operator-tools does not recognise it and never recreates the
Service. Upgrading a cluster with enabledIPv6 left the fluentd or syslog-ng Service
failing to reconcile forever, which stalled every reconciler behind it.

The same helper also names a family outright. A cluster with no range for that family
rejects the Service at create, where there is no recovery path. This branch asks the API
server which families it can allocate, with a dry-run Service, and names only those. The
probe needs no new RBAC and has no version floor.

syslog-ng was affected differently. Its network() source never set ip(), so it listened
on IPv4 only while its Service became IPv6-primary. Fluent Bit resolved the Service to an
AAAA record and the connection reached a pod address with no listener. That took ingestion
down, not just metrics.

The last commits remove the cause. A bind address on the Metrics type decouples the
metrics endpoint from the Service IP families, so wanting IPv6 metrics no longer forces an
IPv6-primary Service. Three smaller defects found on the way are fixed too: the Fluent Bit
config used an invalid Listen key, the syslog-ng config-reloader listened on a port
nothing scraped, and the syslog-ng metrics service overrides were never read.

Context: #2100, #2238, #1835 and #2090 are all closed. #2100 was closed by the commit that
caused two of the regressions above.

Test plan

Verified on a KIND cluster (v1.36.1, service CIDR 10.96.0.0/16) and on envtest API
servers driving the real reconcilers.

  • make lint && make test
  • Upgrade a dual-stack cluster running 6.7.0 with fluentd.enabledIPv6: true. The
    Service reconciles and keeps its allocated addresses.
  • Install on a single-stack IPv4 cluster with enabledIPv6: true. Services are created
    and the Logging converges. The families the previous code asked for are rejected with
    spec.ipFamilies[0]: Invalid value: "IPv6": not configured on this cluster.
  • Install on an IPv6-only cluster with enabledIPv6: true. Services are created.
  • Send logs through syslog-ng with syslogNG.enabledIPv6: true. A message reaches the
    ip("::") ip-protocol(6) listener through an IPv4 Service.
  • Scrape the syslog-ng config-reloader on 9533.
  • Set metrics.bind and confirm it reaches the rendered aggregator config.

Not covered: there is no e2e suite for enabledIPv6, so the dual-stack IPv6-primary
ingestion path is verified only at the layers below a cluster. The bind was run against
ghcr.io/axoflow/axosyslog:4.25.0 and the Services against envtest.

@csatib02
csatib02 requested a balanced review from Copilot August 7, 2026 19:01
@csatib02 csatib02 self-assigned this Aug 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves IPv6 handling across aggregator listeners, metrics endpoints, and Kubernetes Services while preserving existing allocations.

Changes:

  • Detects allocatable Service IP families and preserves assigned families during reconciliation.
  • Adds configurable metrics bind addresses and corrects Fluent Bit/syslog-ng listeners.
  • Fixes syslog-ng metrics overrides and config-reloader scraping.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pkg/sdk/logging/model/syslogng/config/source.go Adds source IP protocol rendering.
pkg/sdk/logging/model/syslogng/config/config.go Configures the IPv6 syslog-ng listener.
pkg/sdk/logging/model/syslogng/config/config_test.go Tests source listener rendering.
pkg/sdk/logging/api/v1beta1/common_types.go Adds metrics binding and Service family helpers.
pkg/sdk/logging/api/v1beta1/common_types_test.go Tests metrics bind resolution.
pkg/resources/syslogng/syslogng.go Carries detected families into reconciliation.
pkg/resources/syslogng/statefulset.go Corrects config-reloader metrics port.
pkg/resources/syslogng/service.go Applies family detection and Service overrides.
pkg/resources/syslogng/service_test.go Tests syslog-ng Service fixes.
pkg/resources/ipfamily/detector.go Introduces Service IP-family detection.
pkg/resources/ipfamily/detector_test.go Tests detection and retry behavior.
pkg/resources/fluentd/service.go Preserves and applies Service families.
pkg/resources/fluentd/service_test.go Tests Service upgrade behavior.
pkg/resources/fluentd/fluentd.go Stores detected cluster families.
pkg/resources/fluentd/configsecret.go Resolves the metrics bind address.
pkg/resources/fluentd/config.go Renders the configured metrics bind.
pkg/resources/fluentd/appconfigmap_test.go Updates reconciler construction.
pkg/resources/fluentbit/service.go Applies detected Service families.
pkg/resources/fluentbit/service_test.go Tests family-aware metrics Services.
pkg/resources/fluentbit/fluentbit.go Stores detected cluster families.
pkg/resources/fluentbit/configsecret.go Resolves the metrics bind address.
pkg/resources/fluentbit/config.go Fixes Fluent Bit HTTP listener rendering.
docs/configuration/crds/v1beta1/common_types.md Documents metrics.bind.
controllers/logging/logging_controller.go Integrates family detection into reconcilers.
config/crd/bases/logging.banzaicloud.io_syslogngconfigs.yaml Adds syslog-ng bind schema.
config/crd/bases/logging.banzaicloud.io_loggings.yaml Adds Logging bind schemas.
config/crd/bases/logging.banzaicloud.io_fluentdconfigs.yaml Adds Fluentd bind schema.
config/crd/bases/logging.banzaicloud.io_fluentbitagents.yaml Adds Fluent Bit bind schema.
CLAUDE.md Documents allocated Service field handling.
charts/logging-operator/crds/logging.banzaicloud.io_syslogngconfigs.yaml Synchronizes syslog-ng chart CRD.
charts/logging-operator/crds/logging.banzaicloud.io_loggings.yaml Synchronizes Logging chart CRD.
charts/logging-operator/crds/logging.banzaicloud.io_fluentdconfigs.yaml Synchronizes Fluentd chart CRD.
charts/logging-operator/crds/logging.banzaicloud.io_fluentbitagents.yaml Synchronizes Fluent Bit chart CRD.
charts/logging-operator/charts/logging-operator-crds/templates/logging.banzaicloud.io_syslogngconfigs.yaml Updates packaged syslog-ng CRD template.
charts/logging-operator/charts/logging-operator-crds/templates/logging.banzaicloud.io_loggings.yaml Updates packaged Logging CRD template.
charts/logging-operator/charts/logging-operator-crds/templates/logging.banzaicloud.io_fluentdconfigs.yaml Updates packaged Fluentd CRD template.
charts/logging-operator/charts/logging-operator-crds/templates/logging.banzaicloud.io_fluentbitagents.yaml Updates packaged Fluent Bit CRD template.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/sdk/logging/api/v1beta1/common_types.go
Comment thread pkg/resources/ipfamily/detector.go
Comment thread pkg/sdk/logging/model/syslogng/config/config.go
@csatib02 csatib02 mentioned this pull request Aug 7, 2026
8 tasks
@csatib02
csatib02 requested a balanced review from Copilot August 7, 2026 19:17
@csatib02
csatib02 force-pushed the fix/ipv6-service-families branch from 37d248c to 8d5f9ea Compare August 7, 2026 19:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/resources/ipfamily/detector.go:63

  • nil means both “IPv4-only” and “the probe failed.” The syslog-ng config path converts it to ClusterHasIPv6=false, while Service generation still sets PreferDualStack without pinning a family. On an IPv6-only cluster, a transient dry-run failure can therefore render an IPv4-only network() listener and then create an IPv6 Service, recreating the ingestion outage. Propagate the probe error (or model the unknown state separately) and abort/requeue instead of continuing with nil.
	hasIPv6, err := d.allocatable(ctx, namespace, corev1.IPv6Protocol)
	if err != nil {
		return nil
	}
	hasIPv4, err := d.allocatable(ctx, namespace, corev1.IPv4Protocol)
	if err != nil {
		return nil

pkg/sdk/logging/api/v1beta1/common_types.go:78

  • Metrics is also used by SyslogNGSpec.Metrics, so this field is accepted in the generated syslog-ng CRDs, but syslogNGMetricsSidecarContainer only passes --socket.path and never consumes Bind/BindAddress. Consequently spec.metrics.bind is silently ignored for syslog-ng and the documented default is not applied. Wire the address into the exporter’s listen option, or avoid exposing this field for syslog-ng if the exporter cannot support it.

Kubernetes does not allow the primary IP family of an existing Service to
change, and rejects the attempt as a validation error rather than an
immutable-field one, so operator-tools cannot recognise it and the
delete/recreate fallback never engages. Since 0321c1d asks for an
IPv6-primary Service where 6.7.0 asked for IPv4-primary, upgrading a cluster
running with enabledIPv6 left the fluentd and syslog-ng Services permanently
failing to reconcile, which stalled the rest of the chain behind them.

The before-update hook already pins clusterIP and clusterIPs to whatever the
API server handed out; ipFamilies now follows the same rule. The policy is
pinned only when it would narrow an existing multi-family Service, so
enabling IPv6 on an existing Service still widens it to dual-stack.

Only the two aggregator Services are non-headless and take this path.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
The failure mode is silent: operator-tools only recreates on messages that
read as immutable-field errors, and neither an IP family flip on an existing
Service nor a family the cluster has no range for produces one.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
EnableIPv6Options hard-coded ipFamilies to [IPv6, IPv4] whenever enabledIPv6
was set. The API server rejects a family it has no service range for, and it
does so at create, where there is no recovery path at all: on a single-stack
IPv4 cluster the Service is never allocated and the reconcile stalls behind
it. 2777a11 extended that reach to the Fluent Bit metrics Services, which
worked at 6.7.0.

The cluster is now asked directly, by dry-run creating a probe Service pinned
to IPv6. A dry-run leaves nothing behind, needs no RBAC beyond the services
create the operator already holds, and has no version floor. An IPv4 control
probe separates a missing IPv6 range from an admission webhook that rejects
every Service, and an unresolved answer is retried rather than cached, so a
cluster that gains an IPv6 range is picked up without a restart.

Where the cluster has IPv6 the primary family is unchanged. Where it does not,
the families are left to the cluster and PreferDualStack still applies.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
syslogNG.enabledIPv6 only ever changed the Services. syslog-ng itself has no
ip() on its network source, so it keeps the upstream default of ip(0.0.0.0)
ip-protocol(4) and listens on IPv4 alone. Once 0321c1d made the shared helper
ask for an IPv6 primary, the port 601 Service resolved to an AAAA that Fluent
Bit's forward output prefers, and kube-proxy sent the connection to a pod
address with no listener behind it. That took ingestion down, not just metrics.

The source now binds "::" with ip-protocol(6) under the same flag, which still
accepts IPv4-mapped clients wherever bindv6only is off, so the two sides of the
flag finally agree.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
The single-stack branch emitted "Listen", which is not a [SERVICE] key, so
Fluent Bit ignored it and fell back to its own default. The default happened to
be the same address, which is why nothing looked wrong.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
metricsService and bufferVolumeMetricsService have been on the CRD since
syslog-ng support landed, but nothing ever read them, so whatever a user put
there was dropped without a word.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
enabledIPv6 was the only way to move the metrics listener, and it moves it to
[::], which fluent-plugin-prometheus serves through WEBrick with IPV6_V6ONLY
set. Reaching that listener then needs an IPv6 primary on the Service, which is
what tied metrics scraping to the cluster's IP families in the first place.

A bind address on the Metrics type breaks that coupling: enabledIPv6 now only
picks the default, and a user who needs a specific address names it.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
The probe asked whether IPv6 was allocatable and nothing else, while the
Service always named IPv4 alongside it. On an IPv6-only cluster the probe
succeeded and every Service then carried an IPv4 entry the API server had no
range for, so it was rejected at create exactly as an IPv6 entry is on an
IPv4-only cluster. That is the failure this package exists to prevent, left
open in the mirror direction.

The detector now reports the allocatable families and the Service names those.
A plain IPv4 cluster still names nothing, which is what it did before.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
Pinning the allocated families overrides a single-stack policy the user asked
for through serviceOverrides, so a deliberate downgrade to one family never
happened and nothing said so. The reconcilers now log it.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
Probing each family on its own merit lost the control the previous shape had.
A cluster serves at least one family, so two rejections mean something refuses
every Service, not that the cluster has no ranges. The detector treated that as
a resolved answer of "no families" and kept it for the life of the process, so
an admission webhook that was later fixed left the operator on the wrong answer
until a restart.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
… metrics bind

Three points from review.

The syslog-ng source bound "::" whenever enabledIPv6 was set. That reaches IPv4
clients only through v4-mapped addresses, which net.ipv6.bindv6only turns off.
On a node with that setting and a cluster without IPv6, the Service is IPv4 and
the source had no IPv4 listener, so ingestion failed. The bind now follows the
same cluster answer the Services use.

The buffer-metrics sidecars hard-coded their listen address, so a bind set on
bufferVolumeMetrics was accepted and dropped. They pass it through now. An unset
bind still yields the wildcard address.

A cluster with no IPv6 range is re-probed instead of cached, so a range added to
a running cluster is picked up without restarting the operator.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
…ermined

Families returned nil both for a plain IPv4 cluster and for a probe that could
not run. The syslog-ng config read that as "no IPv6" and rendered an IPv4
listener, while Service generation kept PreferDualStack and let the API server
pick. On an IPv6-only cluster a single failed probe therefore produced an IPv6
Service in front of an IPv4 listener, which is the outage this branch exists to
remove.

The probe now returns an error and the reconcile stops, so the Service and the
listener are always built from the same answer.

Also drops the config-reloader port test, which moved to another pull request
with the change it covered.

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
…ress

Signed-off-by: Bence Csati <bence.csati@axoflow.com>
@csatib02
csatib02 force-pushed the fix/ipv6-service-families branch from 7c9cd10 to 03f0a28 Compare August 10, 2026 14:33
@csatib02
csatib02 merged commit 39f0d27 into master Aug 10, 2026
33 checks passed
@csatib02
csatib02 deleted the fix/ipv6-service-families branch August 10, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants