feat(cse): emit kubelet active settings as a structured JSON Kusto event

[abigailliang-aks-sig-node]

Summary

Emit kubelet's effective runtime configuration (CLI flags + config file content) as a structured JSON event in GuestAgentGenericLogs, queryable via parse_json() from Kusto.

Motivation

There is currently no easy way to query kubelet's active settings across nodes in Kusto. Debugging requires SSH-ing into individual nodes. This event enables a single Kusto query to surface kubelet configuration for all nodes in a cluster.

Design: one event per boot is sufficient

• Kubelet's startup flags/config are constant within a single boot — unlike memory usage, they don't drift, so there is no meaningful timechart to draw over a constant.
• Configuration only changes on restart/reconfig — and every restart re-emits the event. So "one event per boot" naturally covers every real configuration change point.

Architecture: async oneshot systemd service

The telemetry runs as an independent systemd oneshot service (emit-kubelet-active-flags.service), completely decoupled from the CSE critical path:

• WantedBy=kubelet.service, enabled at VHD build time — systemd automatically starts it whenever kubelet starts; no systemctl start call needed in cse_config.sh
• After=kubelet.service — guarantees /etc/default/kubelet and /etc/default/kubeletconfig.json are already written
• ConditionPathExists=!/run/emit-kubelet-active-flags.done + ExecStartPost=/bin/touch /run/emit-kubelet-active-flags.done — limits to one event per boot (prevents flooding if kubelet crashloops; /run is tmpfs, cleared on reboot)
• No CSE changes needed — cse_config.sh is not modified; the service is triggered entirely by systemd dependency ordering

This follows the same pattern as measure-tls-bootstrapping-latency.service.

Why not inline in CSE?

Even with & and FD redirection, a background subshell inherits the CSE pipe FDs. In scriptless phase 2, Go's cmd.Wait() blocks until all pipe write ends close — so the background process still delays CSE completion. A systemd oneshot service runs in its own scope with independent FDs, fully decoupled.

Why not a generic node-telemetry service?

This unit is deliberately kubelet-scoped rather than a shared "node config telemetry" service, because
the repo already establishes one unit per telemetry domain, each with a different trigger:

Unit	Trigger
cgroup-memory-telemetry	.timer — OnBootSec=0 + every 5 min
cgroup-pressure-telemetry	.timer — periodic
aks-check-network	After=network-online.target
emit-kubelet-active-flags	WantedBy=kubelet.service

Folding other domains (e.g. OS configuration: sysctl, THP, cgroup version, ulimits) into this unit would
be wrong on two counts:

• Trigger — OS settings have no kubelet dependency and are ready at boot. Gating them on
  kubelet.service means a node whose kubelet fails to start emits no OS telemetry, which is exactly
  when it is most needed for diagnosis.
• Size — Context1 is hard-truncated at 3072 bytes and this payload already drops config_file when it
  exceeds 3000. A second domain in the same Message would crowd out the kubelet data, so it has to be a
  separate event (separate TaskName) anyway.

Separate event + separate trigger implies a separate unit. What is worth sharing when a second consumer
appears is the event-writing boilerplate (the jq -n envelope, the size cap, the once-per-boot /run
marker) — currently duplicated across ~8 scripts in parts/linux/cloud-init/artifacts/. Note this script
is VHD-baked and intentionally does not source cse_helpers.sh, which is what keeps it off the CSE path.

Data sources: files, not journalctl

Reads directly from kubelet's config files on disk (instantaneous, no polling needed):

File	Contains
/etc/default/kubelet	KUBELET_FLAGS=--cloud-provider=external --max-pods=110 ... (command-line flags)
/etc/default/kubeletconfig.json	KubeletConfiguration JSON (39 translated flags: maxPods, clusterDNS, featureGates, etc.)

Both files are written by ensureKubelet() before kubelet starts, so they are guaranteed to exist when this service runs (After=kubelet.service).

Together these cover all RP-configurable kubelet settings: the 39 flags translated to config file + the command-line flags (notably --cloud-provider).

Size guard

GuestAgentGenericLogs.Context1 is hard-truncated at 3072 bytes (3 KiB), verified empirically (16M events, max Context1 length = 3072). We cap at 3000 bytes. If exceeded, config_file is dropped and replaced with "truncated:exceeded_size_cap", with a log warning.

Files

File	Purpose
parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh	Standalone script: reads config files, assembles JSON, writes guest agent event
parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.service	Systemd oneshot unit: After=kubelet.service, WantedBy=kubelet.service, once-per-boot guard
vhdbuilder/packer/packer_source.sh	Installs both files to VHD paths
vhdbuilder/packer/vhd-image-builder-*.json	Uploads both files during VHD bake
vhdbuilder/packer/pre-install-dependencies.sh	systemctl enable emit-kubelet-active-flags.service — creates the kubelet.service.wants/ symlink that makes WantedBy= take effect
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Same install + enable for the OSGuard image

No changes to cse_config.sh or app.go — the service is fully self-contained.

Event payload

{
  "uses_config_file": true,
  "config_path": "/etc/default/kubeletconfig.json",
  "kubelet_flags_count": 5,
  "kubelet_flags": { "cloud-provider": "external", "node-ip": "10.224.0.5" },
  "config_file": { "maxPods": 110, "clusterDNS": ["10.0.0.10"] }
}

kubelet_flags comes from /etc/default/kubelet, config_file from /etc/default/kubeletconfig.json — so
kubelet_flags_count doubles as a migration-progress metric (how many settings have not moved into the config file yet).

Kusto query example

GuestAgentGenericLogs
| where TaskName == "AKS.CSE.ensureKubelet.kubeletActiveFlags"
| extend payload = parse_json(Context1)
| project PreciseTimeStamp, RoleInstanceName, Cluster,
    usesConfigFile = payload.uses_config_file,
    remainingFlags = payload.kubelet_flags_count,
    maxPods = payload.config_file.maxPods,
    clusterDNS = payload.config_file.clusterDNS,
    featureGates = payload.config_file.featureGates,
    cloudProvider = payload.kubelet_flags.["cloud-provider"],
    allFlags = payload.kubelet_flags

Test plan

• ShellSpec tests pass (819 examples, 0 failures)
• make generate produces no diff
• E2E via [Unofficial][Orchestration] AgentBaker + Linux VHD Full E2E Tests pipeline (bake + apply with current branch)
• SSH into node → systemctl status emit-kubelet-active-flags shows completed
• Event file exists in /var/log/azure/Microsoft.Azure.Extensions.CustomScript/events/ with kubeletActiveFlags TaskName

[abigailliang-aks-sig-node]

Tracking work item: AB#38936519

[github-actions]

Windows Unit Test Results

  3 files   11 suites   47s ⏱️
381 tests 381 ✅ 0 💤 0 ❌
384 runs  384 ✅ 0 💤 0 ❌

Results for commit d95ede2.

♻️ This comment has been updated with latest results.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173691175
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_AzureLinuxV3_MANA validating traffic through MANA VF via debug pod exec

Detective summary: VMSS/node provisioning, SSH, validation pod, MANA PCI, driver, and VF bonding all passed; the failure occurred when Kubernetes exec from the debug pod returned apiserver-to-kubelet proxy 502 Bad Gateway before the curl traffic check could prove a datapath issue.

Likely cause / signature: Transient kubelet streaming/exec transport failure; reusing existing wiki signature wireserver-exec-context-deadline-kubelet-streaming.
Wiki signature: wireserver-exec-context-deadline-kubelet-streaming

Confidence: High

Recommended owner/action: E2E/test-infra owner should track under the kubelet-streaming flake and consider retrying exec on 502/500/context-deadline validation failures; no PR-author action recommended from this signal alone.

Strongest alternative: Real MANA datapath regression, but less likely because MANA setup validations passed and the observed error is the Kubernetes exec proxy transport failing before curl results were available.

Evidence: timeline failed task exit 1; failed test run 545377067; task log 562 around line 799; build metadata and PR #9034 metadata.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173784179
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_Ubuntu2204_ArtifactStreaming_ARM64 kernel-log validation

Detective summary: Ubuntu 22.04 ArtifactStreaming ARM64 provisioned successfully, reached VM validation, then kernel-log validation flagged an early boot Call trace/PANIC_CRASH after node Ready, SSH, validation pod, KSCR/bootstrap, and NIC/rx checks passed.

Likely cause / signature: Likely transient/platform kernel call-trace validation noise; the kubelet-active-flags JSON PR is unlikely causal because the failure is in kernel log scan after unrelated provisioning/validation succeeded.
Wiki signature: ubuntu2204-artifactstreaming-arm64-kernel-calltrace-validation

Confidence: Medium

Recommended owner/action: Node Lifecycle/E2E owners should track as a new Ubuntu2204 ArtifactStreaming ARM64 kernel-calltrace signature and inspect scenario kernel logs if it recurs.

Strongest alternative: A PR-induced kubelet/config output instability, but less likely because the PR surface is telemetry/config formatting and the node was otherwise healthy through validation.

Evidence: timeline failed task exit 1; failed test run 546207590; task log 562 around line 772; build/PR metadata.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

aks-node-controller/helpers/kubelet_flags.go:18

• The TaskName used for the scriptless kubelet-active-flags event ("AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags") does not match the TaskName referenced in the PR description/query example ("ensureKubelet.kubeletActiveFlags"), so the documented Kusto query will miss events emitted by aks-node-controller. Either update the query/docs everywhere this is consumed, or align the TaskName here.

	// kubeletActiveFlagsTaskName is the event TaskName for the kubelet active flags telemetry.
	// Scriptless (aks-node-controller) and shell CSE use different TaskName values; keep this in sync
	// with the query patterns documented in the PR description.
	kubeletActiveFlagsTaskName = "AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags"
	// kubeletFlagsPollMaxAttempts is how many times to poll journalctl for FLAG lines.

aks-node-controller/app.go:193

• EmitKubeletActiveFlagsEvent() can block for up to ~20s (10 attempts × 2s interval) while polling journalctl, but it is called synchronously on the success path. This contradicts the PR description’s claim that the scriptless implementation runs as a goroutine and can also delay the provision command/service completion unnecessarily. Consider moving this to an asynchronous execution path (or otherwise shortening/bounding the poll) while ensuring the process lifetime still allows the event to be written.

		a.eventLogger.LogEvent("Provision", "Completed", helpers.EventLevelInformational, startTime, endTime)
		slog.Info("aks-node-controller finished successfully.")
		// Emit kubelet active flags as a structured event for Kusto querying.
		// Best-effort: telemetry failures do not fail provisioning.
		a.eventLogger.EmitKubeletActiveFlagsEvent()
	}

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/cse_config.sh:1016

• The PR description states the Go/aks-node-controller path is the “primary” implementation (with TaskName AKS.AKSNodeController.ensureKubelet.kubeletActiveFlags and helper code under aks-node-controller/helpers/kubelet_flags.go), but the current changeset only adds the shell fallback (AKS.CSE.ensureKubelet.kubeletActiveFlags). This mismatch makes it unclear whether the scriptless/phase-2 path is actually covered by this PR or whether the description needs updating.

    # Emit kubelet's effective startup configuration as a structured JSON event so it is queryable
    # per node from Kusto (rollout verification for the flags-to-config-file migration, and config/
    # version tracking dashboards). The JSON lands verbatim in GuestAgentGenericLogs.Context1 and is
    # consumable with parse_json(). Best-effort: never fail node provisioning on this.
(emitKubeletActiveFlagsEvent || true) &

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:52

• Flag parsing only handles --key=value tokens. Several kubelet flags are passed in --key value form (e.g. --config /etc/default/kubeletconfig.json, --kubeconfig ..., --bootstrap-kubeconfig ...), so they’ll be silently dropped from the emitted JSON.

        # Parse --key=value pairs from the flags string
        local key value
        for token in ${kubelet_flags}; do
            case "${token}" in
                --*=*)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:28

• The script currently only reads KUBELET_FLAGS from /etc/default/kubelet, but kubelet’s effective startup args also include flags coming from other env vars and systemd drop-ins (e.g. KUBELET_CONTAINERD_FLAGS, KUBELET_CONTAINER_RUNTIME_FLAG, KUBELET_CGROUP_FLAGS, KUBELET_TLS_BOOTSTRAP_FLAGS, and --node-labels). As-is, the emitted event won’t reflect the full effective kubelet startup configuration.

    # Read KUBELET_FLAGS from /etc/default/kubelet
    if [ -f "${KUBELET_DEFAULT_FILE}" ]; then
        kubelet_flags="$(grep '^KUBELET_FLAGS=' "${KUBELET_DEFAULT_FILE}" | sed 's/^KUBELET_FLAGS=//')"
    fi

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:80

• The size guard only drops config_file once; if the payload is still > 3000 bytes (e.g., unusually long KUBELET_FLAGS values), GuestAgentGenericLogs.Context1 will still be truncated and parse_json() will fail. Consider re-checking size after dropping config_file, and if still oversized, also drop (or truncate) flags to guarantee the emitted payload remains valid JSON.

    if [ "${payload_bytes}" -gt "${max_message_bytes}" ]; then
        echo "kubelet active settings payload exceeded ${max_message_bytes} bytes (${payload_bytes}), dropping config_file content" >&2
        payload="$(jq -cn \
            --argjson uses_config_file "${uses_config_file}" \
            --arg config_path "${config_path}" \

[awesomenix]

This is oneshot service even if it crash loops, it will still be executed only once isnt it? since you also set Restart=no, may be you can get rid of it.

[Devinwong]

seems we also need After=kubelet.service

Without After=kubelet.service, systemd may start both services concurrently:
kubelet ───────► starting
collector ─────► reads settings immediately

With After=:
kubelet ───────► start job finishes
└──► collector starts

You can verify if this is really needed.

[Devinwong]

But since kubelet configs is always ready before kubelet starts, this is probably fine. Just food for thought.