Beyla discovery config uses regex syntax in glob fields

Chart: k8s-monitoring v4.0.0 (feature-auto-instrumentation sub-chart)
File: charts/k8s-monitoring/charts/feature-auto-instrumentation/templates/_beyla-config.tpl
Beyla version: v3.1.2 (also affects earlier versions)
Status: Confirmed bug — no existing issue filed
Reported by: Prospect during POV (EKS, Bottlerocket, containerd, cgroup v2)

Summary

The default Beyla discovery config generated by the k8s-monitoring Helm chart uses regex syntax for fields that Beyla interprets as glob patterns (gobwas/glob). This causes two problems:

1. discovery.instrument matches zero namespaces, so Beyla never discovers or instruments any services. Result: zero traces.
2. discovery.exclude_instrument matches zero executables, so the intended exclusion of Alloy, OTel Collector, and Beyla itself is silently ineffective.

Bug 1 — k8s_namespace: "." matches nothing

Location

_beyla-config.tpl, line 25:

{{- $services := list (dict "k8s_namespace" ".") }}

Rendered output

discovery:
  instrument:
    - k8s_namespace: .

Problem

The Beyla docs define k8s_namespace as string (glob). In glob syntax, . is a literal dot character. It does not mean "match any character" as it does in regex. No real Kubernetes namespace is named ., so this pattern matches nothing. Beyla discovers zero services and produces zero traces.

Fix

{{- $services := list (dict "k8s_namespace" "*") }}

This renders as k8s_namespace: "*", which is the glob wildcard for "match everything."

Bug 2 — exe_path uses regex alternation in a glob field

Location

_beyla-config.tpl, line 26:

{{- $excludeServices := list (dict "exe_path" ".*alloy.*|.*otelcol.*|.*beyla.*") }}

Rendered output

discovery:
  exclude_instrument:
    - exe_path: .*alloy.*|.*otelcol.*|.*beyla.*

Problem

exe_path is also a string (glob) field. The pattern .*alloy.*|.*otelcol.*|.*beyla.* uses regex constructs:

• .* in glob means "literal dot followed by zero-or-more characters," not "zero-or-more of any character."
• | in glob is a literal pipe character, not alternation.

The pattern effectively matches nothing (or only paths containing literal dots and pipes in those positions).

Fix (option A — glob alternation with braces)

{{- $excludeServices := list (dict "exe_path" "{*alloy*,*otelcol*,*beyla*}") }}

Fix (option B — multiple entries)

{{- $excludeServices := list (dict "exe_path" "*alloy*") (dict "exe_path" "*otelcol*") (dict "exe_path" "*beyla*") }}

Mitigating factor

Starting from recent Beyla versions, Beyla ships a built-in default_exclude_instrument that already excludes *beyla, *alloy, *otelcol, *otelcol-contrib, and several Kubernetes system namespaces (kube-system, kube-node-lease, monitoring, GKE namespaces, etc.). So even with the chart's broken exclude_instrument, Beyla's own defaults provide coverage. Bug 2 is still incorrect and should be fixed, but its practical impact is lower than Bug 1.

Origin

The k8s_namespace: "." pattern has existed since the v1 chart:

{{- $preset = dict "discovery" (dict "services" (list (dict "k8s_namespace" ".")))}}

When PR #2350 (commit 92cbe2119, authored by Pete Wall) updated the v4 chart to rename services/exclude_services to instrument/exclude_instrument to match Beyla's API change, the pattern values were carried forward without correcting the glob vs. regex mismatch.

Why it hasn't been caught

1. Tests assert the broken output. All rendered test snapshots (for example, tests/feature_beyla_config_test.yaml) contain the incorrect patterns, so tests pass.
2. Most real users override the discovery block. The buggy defaults only apply when preset: application is set and no custom discovery block is provided in values. Integration test examples like auto-instrumentation and split-destinations supply their own discovery config, which completely bypasses the defaults (guarded by not (hasKey $config "discovery") on line 24).
3. Beyla's built-in default_exclude_instrument masks Bug 2 silently in newer versions.

Evidence from Beyla documentation

The Beyla service discovery docs define the field types in the instrument / exclude_instrument schema:

All Kubernetes attribute selectors and exe_path are glob fields, not regex.

Affected rendered outputs

The broken defaults appear in every rendered example and test snapshot that uses preset: application without a custom discovery block:

• tests/feature_beyla_config_test.yaml
• tests/platform/openshift/.rendered/output.yaml
• tests/platform/grafana-cloud/k8s-monitoring/.rendered/output.yaml
• docs/examples/tolerations/output.yaml
• docs/examples/resource-requests-and-limits/output.yaml
• docs/examples/private-image-registries/individual/output.yaml
• docs/examples/images/images-by-digest/output.yaml
• docs/examples/features/auto-instrumentation/beyla-metrics/output.yaml
• docs/examples/features/auto-instrumentation/span-metrics-only/output.yaml
• docs/examples/exclude-by-namespace/output.yaml

Recommendation

1. File a bug on grafana/k8s-monitoring-helm.
2. Patch _beyla-config.tpl — two-line fix (lines 25–26).
3. Regenerate all rendered test snapshots to match the corrected output.
4. Review the exclude-by-namespace example (docs/examples/exclude-by-namespace/values.yaml), which also uses k8s_namespace: . as a "match all" pattern in user-facing documentation. That example should be updated to k8s_namespace: "*".
5. Page the author of PR Update Beyla auto-instrumentation config to use instrument/exclude_instrument #2350 (Pete Wall) for review.

Prospect workaround

The prospect resolved the issue by overriding the discovery block in their Helm values:

autoInstrumentation:
  beyla:
    config:
      data:
        discovery:
          instrument:
            - k8s_namespace: "*"
          exclude_instrument:
            - exe_path: "*alloy*"
            - exe_path: "*otelcol*"
            - exe_path: "*beyla*"

This is a valid workaround because the template only injects defaults when no discovery key exists in the user-provided config data.