[BUG] checkKnownKey emits false-positive WARN for valid system-probe.yaml keys when read by the core Agent process

[siebrand]

Agent version

7.82.1

Bug Report

The core Agent process logs checkKnownKey WARN messages for config keys that are fully valid and
schema-registered, but happen to be defined only in the system-probe config setup path
(pkg/config/setup/system_probe.go and related files) rather than in the core Agent setup path.

Example warnings observed on a host running Agent 7.82.1 with no system-probe.yaml present:

WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.enable_oom_kill is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.enable_co_re is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.enable_runtime_compiler is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.enable_kernel_header_download is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.allow_prebuilt_fallback is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.telemetry_enabled is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key system_probe_config.max_conns_per_message is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.collect_tcp_v4 is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.collect_tcp_v6 is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.collect_udp_v4 is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.collect_udp_v6 is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.enable_protocol_classification is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.enable_gateway_lookup is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key network_config.enable_root_netns is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key runtime_security_config.enabled is unknown
WARN | (pkg/config/nodetreemodel/config.go:524 in checkKnownKey) | config key service_monitoring_config.enabled is unknown

Every one of these keys is present in pkg/config/schema/yaml/system-probe_schema.yaml at 7.82.1,
confirmed by parsing the schema directly:

system_probe_config properties include: enable_oom_kill, enable_co_re, enable_runtime_compiler,
  enable_kernel_header_download, allow_prebuilt_fallback, telemetry_enabled, max_conns_per_message
network_config properties include: collect_tcp_v4, collect_tcp_v6, collect_udp_v4, collect_udp_v6,
  enable_protocol_classification, enable_gateway_lookup, enable_root_netns
service_monitoring_config properties include: enabled
runtime_security_config properties include: enabled, socket

Root cause

checkKnownKey (pkg/config/nodetreemodel/config.go:524) checks c.knownKeys, which is populated
by addToKnownKeys, which is called only when BindEnvAndSetDefault / SetDefault / SetKnown
are invoked for a given key. The system-probe-specific setup functions
(pkg/config/setup/system_probe.go, system_probe_usm.go, system_probe_cws.go) are only called
during system-probe process initialisation, not during core Agent initialisation. As a result, the
core Agent's ntmConfig instance has no entries in knownKeys for any system-probe-owned key.

When the core Agent reads datadog.yaml (which may contain system_probe_config.*,
network_config.*, etc. sections, or when it reads defaults that reference those keys), every
lookup triggers a false-positive WARN.

The schema YAML (pkg/config/schema/yaml/system-probe_schema.yaml) correctly lists all these keys,
so the schema and the runtime knownKeys map are out of sync for cross-process config keys.

Reproduction Steps

1. Install Agent 7.81+ on any Linux host.
2. Do not create /etc/datadog-agent/system-probe.yaml.
3. Start the core Agent (datadog-agent).
4. grep checkKnownKey /var/log/datadog/agent.log

The warnings appear on every Agent start without any system-probe or security-agent configuration
present.

Agent configuration

No system-probe.yaml or security-agent.yaml present. Minimal datadog.yaml with only api_key
and site set. Warnings are produced purely from the Agent's own internal default values.

Operating System

AlmaLinux 9.8 (Olive Jaguar)

Other environment details

Agent installed via the Datadog Puppet module v4.1.1. The Puppet module was initially suspected but confirmed not to be the cause: the config files it would write do not exist on the affected host.