Skip to content

Commit f57e7a3

Browse files
authored
[OTAGENT-1149] Fix standalone otel-agent remote-config/configsync guards leaking from env vars (#54485)
## Summary - Standalone `otel-agent` (`DD_OTEL_STANDALONE=true`) disables `remote_configuration.enabled` at `SourceFile` priority, which is lower than `SourceEnvVar`. A deployment tool that colocates `otel-agent` with a core Datadog Agent — e.g. the Datadog Operator's DaemonSet path — injects `DD_REMOTE_CONFIGURATION_ENABLED=true` into every container including `otel-agent`, silently overriding the guard. This crashes the embedded trace agent's RC client at startup (exit 255: `could not instantiate the tracer remote config client: grpc client disabled via cmd_port: -1`), since standalone mode has already disabled the core-agent IPC (`cmd_port=-1`) the RC client would need. - Raise the guard to `SourceAgentRuntime`, which outranks `SourceEnvVar`, so it can no longer be overridden by an injected env var. - Apply the same treatment to configsync: force `agent_ipc.config_refresh_interval=0` under standalone mode, so a colocated core agent's `DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL` env var can't make "standalone" `otel-agent` keep syncing config from a core agent it isn't supposed to depend on. See [OTAGENT-1149](https://datadoghq.atlassian.net/browse/OTAGENT-1149) for the full investigation (verified end-to-end against the Datadog Operator in a kind cluster). ## Test plan - [x] Added `TestStandaloneModeIgnoresCoreAgentIPCEnvVars` reproducing the env-var-leak scenario (`cmd/otel-agent/config/agent_config_test.go`) - [x] `dda inv test --targets=./cmd/otel-agent/config/...` passes - [x] `dda inv otel-agent.build` succeeds - [x] Re-verified end-to-end against the Datadog Operator DaemonSet manifest from the ticket: built a local Linux/arm64 image with the fix, deployed via the Datadog Operator into a kind cluster with `remoteConfiguration.enabled: true` (the previously-crashing config), and confirmed `otel-agent` starts cleanly and stays up (0 restarts). Logs show `configsync disabled: agent_ipc.config_refresh_interval invalid: 0`, no RC-client crash, and the expected standalone boot sequence (`Starting dogtelextension in standalone mode`, tagger gRPC server on :15000), with OTLP receivers up and payloads posting successfully to the Datadog API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [OTAGENT-1149]: https://datadoghq.atlassian.net/browse/OTAGENT-1149?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: yang.song <yang.song@datadoghq.com>
1 parent b39d95d commit f57e7a3

4 files changed

Lines changed: 68 additions & 13 deletions

File tree

cmd/otel-agent/config/agent_config.go

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,34 @@ func NewConfigComponent(ctx context.Context, ddCfg string, uris []string) (confi
179179
fmt.Printf("setting log level to: %v\n", logLevelReverseMap[activeLogLevel])
180180
pkgconfig.Set("log_level", logLevelReverseMap[activeLogLevel], pkgconfigmodel.SourceFile)
181181

182+
// Standalone mode runs without a core Datadog Agent on the same host, so
183+
// every client that would otherwise contact it over IPC (trace-agent
184+
// hostname acquisition, remote tagger, remote workloadmeta, configsync,
185+
// ...) must be disabled. cmd_port=-1 is the conventional way to express
186+
// "no core agent IPC" and is honored by those callers; forcing it here
187+
// means users only have to set DD_OTEL_STANDALONE=true.
188+
//
189+
// All of these are set with SourceAgentRuntime, which outranks
190+
// SourceEnvVar, so they can't be silently re-enabled by a deployment tool
191+
// that colocates otel-agent with a core agent and injects that agent's
192+
// env vars (e.g. DD_REMOTE_CONFIGURATION_ENABLED=true,
193+
// DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL) into this container too.
194+
//
195+
// This must run before the getDDExporterConfig call below, since a
196+
// standalone config without a Datadog exporter (a supported shape) makes
197+
// that call return early via ErrNoDDExporter, which would otherwise skip
198+
// these guards entirely.
199+
if pkgconfig.GetBool("otel_standalone") {
200+
pkgconfig.Set("cmd_port", -1, pkgconfigmodel.SourceAgentRuntime)
201+
// There is no core agent to sync config from; disable configsync
202+
// regardless of whether it's configured over agent_ipc.port or
203+
// agent_ipc.use_socket.
204+
pkgconfig.Set("agent_ipc.config_refresh_interval", 0, pkgconfigmodel.SourceAgentRuntime)
205+
}
206+
if pkgconfig.GetInt("cmd_port") <= 0 {
207+
pkgconfig.Set("remote_configuration.enabled", false, pkgconfigmodel.SourceAgentRuntime)
208+
}
209+
182210
ddc, err := getDDExporterConfig(cfg)
183211
if err == ErrNoDDExporter {
184212
return pkgconfig, err
@@ -224,18 +252,6 @@ func NewConfigComponent(ctx context.Context, ddCfg string, uris []string) (confi
224252
if addr := ddc.Traces.Endpoint; addr != "" {
225253
pkgconfig.Set("apm_config.apm_dd_url", addr, pkgconfigmodel.SourceFile)
226254
}
227-
// Standalone mode runs without a core Datadog Agent on the same host, so
228-
// every client that would otherwise contact it over IPC (trace-agent
229-
// hostname acquisition, remote tagger, remote workloadmeta, ...) must be
230-
// disabled. cmd_port=-1 is the conventional way to express "no core agent
231-
// IPC" and is honored by those callers; forcing it here means users only
232-
// have to set DD_OTEL_STANDALONE=true.
233-
if pkgconfig.GetBool("otel_standalone") {
234-
pkgconfig.Set("cmd_port", -1, pkgconfigmodel.SourceAgentRuntime)
235-
}
236-
if pkgconfig.GetInt("cmd_port") <= 0 {
237-
pkgconfig.Set("remote_configuration.enabled", false, pkgconfigmodel.SourceFile)
238-
}
239255

240256
if !pkgconfig.IsConfigured("apm_config.features") {
241257
apmConfigFeatures := []string{}

cmd/otel-agent/config/agent_config_test.go

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,27 @@ func (suite *ConfigTestSuite) TestAgentConfigWithDatadogYamlKeysAvailable() {
207207
assert.Equal(t, 60, c.GetInt("agent_ipc.config_refresh_interval"))
208208
}
209209

210+
// TestStandaloneModeIgnoresCoreAgentIPCEnvVars reproduces OTAGENT-1149: a
211+
// deployment tool that colocates otel-agent with a core Datadog Agent (e.g.
212+
// the Datadog Operator's DaemonSet) injects that agent's env vars —
213+
// DD_REMOTE_CONFIGURATION_ENABLED=true and DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL
214+
// — into the otel-agent container too. Standalone mode must win regardless,
215+
// since there is no core agent IPC endpoint for the RC client or configsync
216+
// to talk to.
217+
func (suite *ConfigTestSuite) TestStandaloneModeIgnoresCoreAgentIPCEnvVars() {
218+
t := suite.T()
219+
t.Setenv("DD_OTEL_STANDALONE", "true")
220+
t.Setenv("DD_REMOTE_CONFIGURATION_ENABLED", "true")
221+
t.Setenv("DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL", "60")
222+
223+
c, err := NewConfigComponent(context.Background(), "", []string{"testdata/config.yaml"})
224+
require.NoError(t, err)
225+
226+
assert.Equal(t, -1, c.GetInt("cmd_port"))
227+
assert.False(t, c.GetBool("remote_configuration.enabled"))
228+
assert.Equal(t, 0, c.GetInt("agent_ipc.config_refresh_interval"))
229+
}
230+
210231
func (suite *ConfigTestSuite) TestAgentConfigSetAPMFeaturesFromDatadogYaml() {
211232
t := suite.T()
212233
fileName := "testdata/config_default.yaml"

cmd/otel-agent/subcommands/run/command.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,9 @@ func standaloneAgentFxOptions(params *cliParams) fx.Option {
313313
// Resolve hostname locally; no core agent to ask
314314
hostnameimpl.Module(),
315315
// No on-init config sync (no core agent to sync from); periodic sync is also
316-
// effectively disabled by the default agent_ipc.config_refresh_interval=0
316+
// force-disabled in agent_config.go (agent_ipc.config_refresh_interval=0 via
317+
// SourceAgentRuntime) so it can't be re-enabled by an env var meant for a
318+
// colocated core agent.
317319
configsyncfx.Module(configsync.NewParams(params.SyncTimeout, false, params.SyncOnInitTimeout)),
318320
// Local workloadmeta-backed tagger so the infraattributes processor can enrich
319321
// spans with K8s tags (pod, namespace, deployment, ...) without a core agent
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Each section from every release note are combined when the
2+
# CHANGELOG.rst is rendered. So the text needs to be worded so that
3+
# it does not depend on any information only available in another
4+
# section. This may mean repeating some details, but each section
5+
# must be readable independently of the other.
6+
#
7+
# Each section note must be formatted as reStructuredText.
8+
---
9+
fixes:
10+
- |
11+
Fixed a crash loop in standalone ``otel-agent`` (``DD_OTEL_STANDALONE=true``) when deployed
12+
alongside a core Datadog Agent that injects ``DD_REMOTE_CONFIGURATION_ENABLED=true`` or
13+
``DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL`` into its environment, such as when using the
14+
Datadog Operator. Standalone mode now reliably disables Remote Configuration and config
15+
sync regardless of these environment variables, since it has no core agent IPC endpoint
16+
to use them with.

0 commit comments

Comments
 (0)