Skip to content

Commit 66b4cca

Browse files
[Backport 7.80.x] [incident-54830] docker: tolerate invalid CIDRs in /info DefaultAddressPools (#51382)
Backport 309b25f from #51235. ___ ### What does this PR do? Adds a tolerant decoder for the Docker daemon's `/info` response in `pkg/util/docker/`, so that daemons emitting invalid CIDRs in `DefaultAddressPools[].Base` no longer break `DockerUtil` and its downstream callers. Concretely: - New helper `safeInfo(ctx, cli)` (`pkg/util/docker/safe_info.go`): tries `cli.Info(...)` first; on the SDK's JSON-decode failure prefix only, reissues `GET /info` via `cli.Dialer()` and decodes into a mirror of `system.Info` where `DefaultAddressPools` is shadowed by `json.RawMessage`, so the strict `netip.Prefix` field stays at zero value and the rest of the struct still parses. - 5 callsites of `cli.Info(ctx, client.InfoOptions{})` switched to `safeInfo(...)`: `ConnectToDocker`, `DockerUtil.GetHostname`, `DockerUtil.GetStorageStats`, `GetMetadata`, `GetTags`. - `host_tags.go` refactored: pure `buildSwarmTags(info system.Info) []string` extracted; `host_tags_test.go` tests it directly without the `client.SystemAPIClient` mock. - `BUILD.bazel` updated: new files added, unused `//pkg/util/docker/fake` test dep removed. - Release note added. ### Motivation [Incident #54830](https://dd.slack.com/archives/C0B55994JGL/p1779448036913569). Reported symptom (from agent debug logs): ``` PROCESS | DEBUG | (pkg/util/hostname/container.go:34 in callContainerProvider) | GetHostname trying provider 'docker' ... PROCESS | DEBUG | (pkg/util/docker/global.go:40 in GetDockerUtilWithRetrier) | Docker init error: temporary failure in dockerutil, will retry later: Error reading remote info: netip.ParsePrefix("invalid Prefix"): no '/' ``` Root cause: PR #48777 migrated the Docker SDK to `moby/moby` v29 to fix CVE-2026-34040 / CVE-2026-33997. The new `system.Info.DefaultAddressPools[].Base` field is typed as `netip.Prefix`, whose `UnmarshalText` is strict and rejects any string without `/`. Some Docker daemons emit the literal string `"invalid Prefix"` (exactly the output of `netip.Prefix.String()` for the zero/invalid value) for that field, which makes the entire `/info` JSON decode fail. That single decode failure cascaded into: - `DockerUtil` init failure → workloadmeta Docker collector failure → Docker core check failure → missing container/image tags on metrics, traces, logs. - Hostname provider `docker` failure → in containerized environments without an explicit `DD_HOSTNAME`, the agent could refuse to start if other providers (kubelet, EC2/GCE metadata, OS) also failed. - Host metadata (`docker_version`, `docker_swarm`) and host tags (`docker_swarm_node_role`) silently missing. This PR makes the SDK's strict decoding tolerant of the offending field while preserving the strict typing of every other field. Relates to incident #54830. Functionally supersedes #51128 (which patches only the init probe via `Ping`); the two approaches can be discussed and merged into one. ### Describe how you validated your changes - `dda inv agent.build --build-exclude=systemd` — PASS - `dda inv cluster-agent.build` — PASS - `dda inv test --targets=./pkg/util/docker/...` — PASS (30 tests, 3 new in `safe_info_test.go` covering happy path, fallback on `"invalid Prefix"`, and non-decode-error propagation) - `dda inv test --targets=./pkg/util/docker/...,./pkg/collector/corechecks/containers/docker/...,./pkg/util/containers/metrics/docker/...,./comp/core/workloadmeta/collectors/internal/docker/...` — 71/71 PASS - Bazel verification: not run locally (no `bazel` binary on the dev box); BUILD.bazel was updated manually to add `safe_info.go`/`safe_info_test.go` srcs and drop the now-unused `//pkg/util/docker/fake` test dep — CI will validate. ### Additional Notes The fallback's URL construction mirrors the SDK's `getAPIPath` (preserves the configured base path and `cli.ClientVersion()` prefix) so daemons behind reverse proxies that route on `Host` header or `/v<version>/` path receive the request the same way the SDK does. A `log.Debugf` line fires whenever the fallback engages so operators can see when the workaround is active. The match on `"Error reading remote info"` is a string prefix on the SDK's wrapped JSON-decode error. It is stable today (`moby/moby/client@v0.4.1/system_info.go:32`) and unique to JSON-decode failures of `/info`, but is the most fragile part of the patch. If moby ever exposes a typed sentinel for this case, we should switch to `errors.As`/`errors.Is`. Co-authored-by: sabrina.lu <sabrina.lu@datadoghq.com>
1 parent 8a5ef87 commit 66b4cca

8 files changed

Lines changed: 307 additions & 67 deletions

File tree

pkg/util/docker/BUILD.bazel

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ go_library(
1616
"metadata.go",
1717
"metadata_no_docker.go",
1818
"rancher.go",
19+
"safe_info.go",
1920
"storage.go",
2021
"util_common.go",
2122
"util_docker.go",
@@ -51,6 +52,7 @@ go_test(
5152
"event_stream_test.go",
5253
"host_tags_test.go",
5354
"rancher_test.go",
55+
"safe_info_test.go",
5456
"storage_test.go",
5557
"util_docker_test.go",
5658
],
@@ -59,7 +61,6 @@ go_test(
5961
deps = [
6062
"//comp/core/workloadfilter/fx-mock",
6163
"//pkg/config/mock",
62-
"//pkg/util/docker/fake",
6364
"@com_github_moby_moby_api//types/container",
6465
"@com_github_moby_moby_api//types/events",
6566
"@com_github_moby_moby_api//types/swarm",

pkg/util/docker/docker_util.go

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,11 @@ func ConnectToDocker(ctx context.Context) (*client.Client, error) {
8484
if err != nil {
8585
return nil, err
8686
}
87-
// Looks like docker is not actually doing a call to server when `NewClient` is called
88-
// Forcing it to verify server availability by calling Info()
89-
_, err = cli.Info(ctx, client.InfoOptions{})
90-
if err != nil {
87+
// client.New does not actually contact the daemon. Force a round-trip
88+
// to verify availability. safeInfo tolerates daemons that emit invalid
89+
// CIDRs in /info's DefaultAddressPools, which would otherwise fail the
90+
// strict netip.Prefix decoding introduced by the moby v29 client.
91+
if _, err := safeInfo(ctx, cli); err != nil {
9192
return nil, err
9293
}
9394

@@ -182,23 +183,23 @@ func (d *DockerUtil) RawContainerListWithFilter(ctx context.Context, options cli
182183
func (d *DockerUtil) GetHostname(ctx context.Context) (string, error) {
183184
ctx, cancel := context.WithTimeout(ctx, d.queryTimeout)
184185
defer cancel()
185-
result, err := d.cli.Info(ctx, client.InfoOptions{})
186+
info, err := safeInfo(ctx, d.cli)
186187
if err != nil {
187188
return "", fmt.Errorf("unable to get Docker info: %s", err)
188189
}
189-
return result.Info.Name, nil
190+
return info.Name, nil
190191
}
191192

192193
// GetStorageStats returns the docker global storage stats if available
193194
// or ErrStorageStatsNotAvailable
194195
func (d *DockerUtil) GetStorageStats(ctx context.Context) ([]*StorageStats, error) {
195196
ctx, cancel := context.WithTimeout(ctx, d.queryTimeout)
196197
defer cancel()
197-
result, err := d.cli.Info(ctx, client.InfoOptions{})
198+
info, err := safeInfo(ctx, d.cli)
198199
if err != nil {
199200
return []*StorageStats{}, fmt.Errorf("unable to get Docker info: %s", err)
200201
}
201-
return parseStorageStatsFromInfo(result.Info)
202+
return parseStorageStatsFromInfo(info)
202203
}
203204

204205
func isImageShaOrRepoDigest(image string) bool {

pkg/util/docker/host_tags.go

Lines changed: 15 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ import (
1313
"time"
1414

1515
"github.com/moby/moby/api/types/swarm"
16-
"github.com/moby/moby/client"
16+
"github.com/moby/moby/api/types/system"
1717
)
1818

1919
// GetTags returns tags that are automatically added to metrics and events on a
@@ -25,24 +25,22 @@ func GetTags(ctx context.Context) ([]string, error) {
2525
}
2626
ctx, cancel := context.WithTimeout(ctx, 1*time.Second)
2727
defer cancel()
28-
return getTags(ctx, du.cli)
28+
info, err := safeInfo(ctx, du.cli)
29+
if err != nil {
30+
return []string{}, err
31+
}
32+
return buildSwarmTags(info), nil
2933
}
3034

31-
func getTags(ctx context.Context, c client.SystemAPIClient) ([]string, error) {
32-
tags := []string{}
33-
result, err := c.Info(ctx, client.InfoOptions{})
34-
if err != nil {
35-
return tags, err
35+
// buildSwarmTags derives the docker swarm-related host tags from a daemon
36+
// /info response.
37+
func buildSwarmTags(info system.Info) []string {
38+
if info.Swarm.LocalNodeState != swarm.LocalNodeStateActive {
39+
return []string{}
3640
}
37-
switch result.Info.Swarm.LocalNodeState {
38-
case swarm.LocalNodeStateActive:
39-
nodeRole := swarm.NodeRoleWorker
40-
if result.Info.Swarm.ControlAvailable {
41-
nodeRole = swarm.NodeRoleManager
42-
}
43-
tags = append(tags, fmt.Sprintf("docker_swarm_node_role:%s", nodeRole))
44-
default:
45-
break
41+
nodeRole := swarm.NodeRoleWorker
42+
if info.Swarm.ControlAvailable {
43+
nodeRole = swarm.NodeRoleManager
4644
}
47-
return tags, nil
45+
return []string{fmt.Sprintf("docker_swarm_node_role:%s", nodeRole)}
4846
}

pkg/util/docker/host_tags_test.go

Lines changed: 17 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -8,73 +8,53 @@
88
package docker
99

1010
import (
11-
"context"
1211
"testing"
1312

1413
"github.com/moby/moby/api/types/swarm"
1514
"github.com/moby/moby/api/types/system"
16-
"github.com/moby/moby/client"
1715
"github.com/stretchr/testify/assert"
18-
"github.com/stretchr/testify/require"
19-
20-
"github.com/DataDog/datadog-agent/pkg/util/docker/fake"
2116
)
2217

23-
func TestGetTags(t *testing.T) {
18+
func TestBuildSwarmTags(t *testing.T) {
2419
tests := []struct {
25-
desc string
26-
client client.SystemAPIClient
27-
tags []string
20+
desc string
21+
info system.Info
22+
tags []string
2823
}{
2924
{
3025
"manager node with swarm active",
31-
&fake.SystemAPIClient{
32-
InfoFunc: func() (system.Info, error) {
33-
return system.Info{
34-
Swarm: swarm.Info{
35-
LocalNodeState: swarm.LocalNodeStateActive,
36-
ControlAvailable: true,
37-
},
38-
}, nil
26+
system.Info{
27+
Swarm: swarm.Info{
28+
LocalNodeState: swarm.LocalNodeStateActive,
29+
ControlAvailable: true,
3930
},
4031
},
4132
[]string{"docker_swarm_node_role:manager"},
4233
},
4334
{
4435
"worker node with swarm active",
45-
&fake.SystemAPIClient{
46-
InfoFunc: func() (system.Info, error) {
47-
return system.Info{
48-
Swarm: swarm.Info{
49-
LocalNodeState: swarm.LocalNodeStateActive,
50-
ControlAvailable: false,
51-
},
52-
}, nil
36+
system.Info{
37+
Swarm: swarm.Info{
38+
LocalNodeState: swarm.LocalNodeStateActive,
39+
ControlAvailable: false,
5340
},
5441
},
5542
[]string{"docker_swarm_node_role:worker"},
5643
},
5744
{
5845
"swarm inactive",
59-
&fake.SystemAPIClient{
60-
InfoFunc: func() (system.Info, error) {
61-
return system.Info{
62-
Swarm: swarm.Info{
63-
LocalNodeState: swarm.LocalNodeStatePending,
64-
ControlAvailable: true,
65-
},
66-
}, nil
46+
system.Info{
47+
Swarm: swarm.Info{
48+
LocalNodeState: swarm.LocalNodeStatePending,
49+
ControlAvailable: true,
6750
},
6851
},
6952
[]string{},
7053
},
7154
}
7255
for _, tt := range tests {
7356
t.Run(tt.desc, func(t *testing.T) {
74-
ctx := context.TODO()
75-
tags, err := getTags(ctx, tt.client)
76-
require.NoError(t, err)
77-
assert.Equal(t, tt.tags, tags)
57+
assert.Equal(t, tt.tags, buildSwarmTags(tt.info))
7858
})
7959
}
8060
}

pkg/util/docker/metadata.go

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,6 @@ import (
1414

1515
"github.com/DataDog/datadog-agent/pkg/config/env"
1616
"github.com/moby/moby/api/types/swarm"
17-
"github.com/moby/moby/client"
1817
)
1918

2019
// GetMetadata returns metadata about the docker runtime such as docker_version and if docker_swarm is enabled or not.
@@ -32,18 +31,18 @@ func GetMetadata() (map[string]string, error) {
3231
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
3332
defer cancel()
3433

35-
result, err := du.cli.Info(ctx, client.InfoOptions{})
34+
info, err := safeInfo(ctx, du.cli)
3635
if err != nil {
3736
return nil, err
3837
}
3938

4039
dockerSwarm := "inactive"
41-
if result.Info.Swarm.LocalNodeState == swarm.LocalNodeStateActive {
40+
if info.Swarm.LocalNodeState == swarm.LocalNodeStateActive {
4241
dockerSwarm = "active"
4342
}
4443

4544
return map[string]string{
46-
"docker_version": result.Info.ServerVersion,
45+
"docker_version": info.ServerVersion,
4746
"docker_swarm": dockerSwarm,
4847
}, nil
4948
}

pkg/util/docker/safe_info.go

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
// Unless explicitly stated otherwise all files in this repository are licensed
2+
// under the Apache License Version 2.0.
3+
// This product includes software developed at Datadog (https://www.datadoghq.com/).
4+
// Copyright 2016-present Datadog, Inc.
5+
6+
//go:build docker
7+
8+
package docker
9+
10+
import (
11+
"context"
12+
"encoding/json"
13+
"errors"
14+
"fmt"
15+
"net"
16+
"net/http"
17+
"path"
18+
"strings"
19+
20+
"github.com/moby/moby/api/types/system"
21+
"github.com/moby/moby/client"
22+
23+
"github.com/DataDog/datadog-agent/pkg/util/log"
24+
)
25+
26+
// safeInfo returns the Docker daemon's /info response, working around daemons
27+
// that emit invalid CIDRs in DefaultAddressPools[].Base. The moby v29 SDK
28+
// decodes Base into a netip.Prefix, whose UnmarshalText is strict and rejects
29+
// such values, which would fail the entire /info JSON decode and break every
30+
// caller (init probe, hostname provider, host tags, host metadata, storage
31+
// stats).
32+
//
33+
// Strategy: try the SDK's Info() first. If it fails with the SDK's JSON-decode
34+
// wrapper, retry with a raw HTTP request that decodes into a mirror struct
35+
// where DefaultAddressPools is captured as json.RawMessage, shadowing the
36+
// strict field and letting the rest of /info parse normally.
37+
func safeInfo(ctx context.Context, cli *client.Client) (system.Info, error) {
38+
result, err := cli.Info(ctx, client.InfoOptions{})
39+
if err == nil {
40+
return result.Info, nil
41+
}
42+
43+
// The moby client wraps JSON-decode failures of /info with this prefix
44+
// (see github.com/moby/moby/client/system_info.go). Network, HTTP-status
45+
// and other connection errors do not, and the tolerant fallback would not
46+
// help in those cases — propagate the original error.
47+
if !strings.Contains(err.Error(), "Error reading remote info") {
48+
return system.Info{}, err
49+
}
50+
51+
log.Debugf("Docker /info decode failed (%v); retrying with tolerant decoder", err)
52+
info, fallbackErr := tolerantInfo(ctx, cli)
53+
if fallbackErr != nil {
54+
return system.Info{}, errors.Join(err, fmt.Errorf("tolerant /info fallback: %w", fallbackErr))
55+
}
56+
return info, nil
57+
}
58+
59+
// tolerantInfo reissues GET /info through the moby client's dialer and decodes
60+
// the response into a struct that shadows DefaultAddressPools with a
61+
// json.RawMessage. This bypasses the strict netip.Prefix decoding of the typed
62+
// field while leaving the rest of system.Info populated.
63+
func tolerantInfo(ctx context.Context, cli *client.Client) (system.Info, error) {
64+
httpClient := &http.Client{
65+
Transport: &http.Transport{
66+
// One-shot client: no benefit from keep-alive, and DisableKeepAlives
67+
// ensures the dialed connection is closed when the response body
68+
// is, so the unreferenced transport does not retain FDs.
69+
DisableKeepAlives: true,
70+
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
71+
return cli.Dialer()(ctx)
72+
},
73+
},
74+
}
75+
76+
// The dialer takes care of reaching the daemon (unix, npipe, tcp, tcp+tls).
77+
// For TCP daemons, preserve the configured host and base path so reverse
78+
// proxies relying on Host-header or path routing reach the same backend
79+
// as the SDK does. For unix/npipe, the SDK uses DummyHost — match it.
80+
// Use the "http" scheme even for tls-fronted daemons: the dialer returns
81+
// an already-TLS-encrypted connection, and the http transport writes plain
82+
// HTTP bytes over it.
83+
reqHost := client.DummyHost
84+
basePath := ""
85+
if hostURL, err := client.ParseHostURL(cli.DaemonHost()); err == nil {
86+
if hostURL.Scheme == "tcp" {
87+
reqHost = hostURL.Host
88+
}
89+
basePath = hostURL.Path
90+
}
91+
// Match the SDK's path construction so reverse proxies routing on
92+
// /vX.Y/info don't reject the fallback (see moby client.getAPIPath).
93+
apiPath := "/info"
94+
if v := cli.ClientVersion(); v != "" {
95+
apiPath = "/v" + strings.TrimPrefix(v, "v") + apiPath
96+
}
97+
url := "http://" + reqHost + path.Join("/", basePath, apiPath)
98+
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
99+
if err != nil {
100+
return system.Info{}, err
101+
}
102+
103+
resp, err := httpClient.Do(req)
104+
if err != nil {
105+
return system.Info{}, err
106+
}
107+
defer resp.Body.Close()
108+
109+
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
110+
return system.Info{}, fmt.Errorf("unexpected status %d from /info", resp.StatusCode)
111+
}
112+
113+
// Outer DefaultAddressPools (json.RawMessage) shadows the promoted field
114+
// from the embedded system.Info: encoding/json routes the JSON value to
115+
// the less-nested field, leaving system.Info.DefaultAddressPools at its
116+
// zero value. All other fields decode through the embedded struct.
117+
var tolerant struct {
118+
system.Info
119+
DefaultAddressPools json.RawMessage `json:"DefaultAddressPools"`
120+
}
121+
if err := json.NewDecoder(resp.Body).Decode(&tolerant); err != nil {
122+
return system.Info{}, fmt.Errorf("tolerant decode of /info failed: %w", err)
123+
}
124+
return tolerant.Info, nil
125+
}

0 commit comments

Comments
 (0)