feat(opensearchtransport): add default-on client-side DNS caching (#902)

* feat(opensearchtransport): add default-on client-side DNS caching

Transient resolver outages (e.g. a node-local DNS blip producing
"dial tcp: lookup ...: i/o timeout") fail requests even to nodes whose
addresses were already resolved, because the default transport re-resolves
every dial through the system resolver with no cache to fall back on.
DNS-timeout errors are also not retried under the default config.

Install a process-local DNS cache (github.com/rs/dnscache) on the built-in
transport, enabled by default. Resolved addresses are cached and re-resolved
on an interval (default 60s, mirroring the TTL AWS publishes for managed
OpenSearch Service endpoints); when the resolver is briefly unreachable, the
last-known-good address continues to be served until it recovers, so a
transient DNS outage no longer fails requests to already-resolved hosts. A
node that genuinely moved self-corrects: the stale IP fails to dial, the
connection is marked dead, and routing skips it.

Tune or disable via the DNSCacheRefresh config field or
OPENSEARCH_GO_DNS_CACHE_REFRESH (0 = default, <0 = disable, >0 = explicit
interval). Caching is installed only when no custom Transport is supplied; a
caller-provided Transport is never modified. The refresh goroutine is bound to
the transport root context, so Close() reclaims it. Because Go's resolver does
not expose record TTLs, the interval is a re-resolution cadence, not a
per-record TTL.

Expose DNSLookups, DNSCacheMisses, and DNSLookupErrors counters via
Transport.Metrics(), mirroring the existing AddressResolver* counters.

* feat(opensearchtransport): make DNS dialer timeout and keep-alive configurable

Thread the net.Dialer dial timeout and keep-alive behind the client-side
DNS cache through Config (DNSDialTimeout, DNSKeepAlive) and matching
OPENSEARCH_GO_DNS_DIAL_TIMEOUT / OPENSEARCH_GO_DNS_KEEP_ALIVE env vars,
mirroring the existing DNSCacheRefresh pattern. Previously these were
hardcoded 30s constants with no code- or env-level override.

Bundle the three resolved durations into a dnsCacheSettings value so the
dialer constructor keeps a small signature.

---------

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>
Signed-off-by: Ryan Yuan <ryan.yuan@crowdstrike.com>
Co-authored-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

Author: ryanyuan

CHANGELOG.md

@@ -6,6 +6,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Added
 
+- Add client-side DNS caching, enabled by default on the built-in transport. Resolved addresses are cached and re-resolved on an interval (default 60s, mirroring the TTL AWS publishes for managed OpenSearch Service endpoints). When the resolver becomes briefly unreachable, the last-known-good address continues to be served until the resolver recovers, so transient resolver outages (e.g. a node-local DNS blip producing `dial tcp: lookup ...: i/o timeout`) no longer fail requests for already-resolved hosts. Tune or disable via the `DNSCacheRefresh`, `DNSDialTimeout`, `DNSKeepAlive`, and `DNSTimeout` fields on `opensearch.Config` (or `OPENSEARCH_GO_DNS_CACHE_REFRESH`, `OPENSEARCH_GO_DNS_DIAL_TIMEOUT`, `OPENSEARCH_GO_DNS_KEEP_ALIVE`, `OPENSEARCH_GO_DNS_TIMEOUT`); each follows the 0 = default, <0 = disable, >0 = explicit convention. Caching is installed only when no custom `Transport` is supplied; a caller-provided `Transport` is never modified. A host that resolves to multiple addresses races up to three of them concurrently (random start offset per connection) and takes the first to connect, spreading load and tolerating a dead address. Refresh re-resolves cached hosts sequentially, so `DNSTimeout` (default 10s) bounds each lookup to keep one hung resolution from stalling a refresh tick. The refresh goroutine is bound to the client's root context, so it is reclaimed both when `Close` is called and when `New` returns an error after the context is created. Because Go's resolver does not expose record TTLs, the refresh interval is a re-resolution cadence, not a per-record TTL. Exposes `DNSLookups`, `DNSCacheMisses`, and `DNSLookupErrors` counters via `Transport.Metrics()`
 - `cmd/osgen`: guard `json.RawMessage` in generated request/response types behind a checked-in allowlist (`cmd/osgen/rawmessage_allowlist.txt`). Because a `json.RawMessage` is the symptom of a type the generator could not resolve, a generator bug can silently widen the raw-JSON surface of the public API; generation now fails (non-zero exit) when any `json.RawMessage` use is not listed, including nested forms such as `[]json.RawMessage`, `map[string]json.RawMessage`, and `[][]json.RawMessage` (the leaf is detected at any wrapper depth). Entries are keyed `GoTypeName/jsonFieldName` (whole-response raw bodies use `<Prefix>Resp/-`, and map/array responses whose element type is unresolved use `<Prefix>Resp/[entries]` and `<Prefix>Resp/[records]`). Add `-update-raw-message-allowlist` to regenerate the allowlist from current output (sorted and grouped for minimal diffs), and `-allow-unlisted-raw-message` to downgrade the check to a warning ([#890](https://github.com/opensearch-project/opensearch-go/pull/890))
 - `cmd/osgen`: emit int-backed (const `iota`) enum types for string fields carrying an `x-enum-name` marker alongside an `enum:` constraint. Each enum generates a named int type with a zero-value `<Name>Unknown` sentinel, name<->value lookup maps, `String()`, `MarshalJSON`, and a closed-set `UnmarshalJSON` that rejects unknown wire values via a typed `*Unknown<Name>Error` (recoverable through `errors.As`). The marker is shared, so a single enum type is registered once and reused across every referencing field; a marker reused with a conflicting value set fails generation rather than silently merging. Applied to the security `status` field, which becomes a typed `RestStatus` enum ([#890](https://github.com/opensearch-project/opensearch-go/pull/890))
 - Add `OPENSEARCH_GO_POLICY_DUMP` environment variable: when set with `OPENSEARCH_GO_DEBUG=true`, dumps the router's policy tree (the dot-delimited node paths that `OPENSEARCH_GO_POLICY_*` matchers target, each labeled with its pool or role) to the debug logger at client initialization. The dump walks the structural tree so router wrappers that share an inner policy instance are each rendered in full. ([#883](https://github.com/opensearch-project/opensearch-go/issues/883))

UPGRADING_V5.md

@@ -198,3 +198,23 @@ What changes:
 - **Optional `SignerOptions`**: `signer/awsv2` additionally accepts functional `SignerOptions` to customize the underlying SigV4 signer.
 
 See [USER_GUIDE.md](USER_GUIDE.md#amazon-opensearch-service) for a full example.
+
+## Client-side DNS caching on by default
+
+When no custom `Transport` is supplied, v5 installs a process-local DNS cache on the client's HTTP transport. Resolved addresses are cached and re-resolved on an interval (default 60s), and when the resolver is briefly unreachable the last-known-good address keeps being served until it recovers. v4 performed a fresh lookup per dial via the stock `http.Transport`.
+
+This changes runtime networking for every existing user. The serve-stale behavior means a host whose IP changes -- failover, blue-green, or a scale event -- can keep receiving the previous address for up to the refresh interval before the cache re-resolves. For most deployments this is a resilience win (a transient DNS blip no longer fails requests to already-resolved hosts), but if your topology relies on immediate DNS cutover you can tune or disable it:
+
+```go
+// Tune the re-resolution interval (programmatic).
+cfg := opensearch.Config{
+    DNSCacheRefresh: 10 * time.Second,
+}
+
+// Disable caching entirely, restoring v4 per-dial resolution.
+cfg := opensearch.Config{
+    DNSCacheRefresh: -1, // <0 disables; 0 uses the 60s default
+}
+```
+
+The same knobs are available via environment variables (`OPENSEARCH_GO_DNS_CACHE_REFRESH=-1` to disable), and `DNSDialTimeout` / `DNSKeepAlive` tune the underlying dialer. A caller-supplied `Transport` is never modified, so any client that sets its own `Transport` is unaffected and opts out implicitly. Because Go's resolver does not expose record TTLs, the interval is a re-resolution cadence rather than a per-record TTL.

go.mod

@@ -6,6 +6,7 @@ require (
 	github.com/aws/aws-sdk-go-v2 v1.42.0
 	github.com/aws/aws-sdk-go-v2/config v1.32.25
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.24
+	github.com/rs/dnscache v0.0.0-20230804202142-fc85eb664529
 	github.com/stretchr/testify v1.11.1
 	github.com/wI2L/jsondiff v0.7.1
 	golang.org/x/mod v0.37.0

go.sum

@@ -30,6 +30,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/dnscache v0.0.0-20230804202142-fc85eb664529 h1:18kd+8ZUlt/ARXhljq+14TwAoKa61q6dX8jtwOf6DH8=
+github.com/rs/dnscache v0.0.0-20230804202142-fc85eb664529/go.mod h1:qe5TWALJ8/a1Lqznoc5BDHpYX/8HU60Hm2AwRmqzxqA=
 github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
 github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
@@ -46,6 +48,7 @@ github.com/wI2L/jsondiff v0.7.1 h1:Fg9+yj+1/x3UtPBJhR91TKEzRkrEEWcAcLbg9dzEaNM=
 github.com/wI2L/jsondiff v0.7.1/go.mod h1:yAt2W7U6Jd4HK0RA8DGSGk0zDtfEtOUUJVnH/xICpjo=
 golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
 golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
 golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=

guides/config-envvars.md

@@ -14,6 +14,10 @@ Every runtime variable, its default, and a one-line summary. Use this as the tab
 | ----------------------------------------------------------------------------- | ---------------------------- | ------------------------------------- |
 | [`OPENSEARCH_URL`](#connection)                                               | unset                        | Seed addresses                        |
 | [`OPENSEARCH_GO_REQUEST_TIMEOUT`](#connection)                                | `0` (none)                   | Per-attempt timeout                   |
+| [`OPENSEARCH_GO_DNS_CACHE_REFRESH`](#connection)                              | `60s`                        | Client-side DNS cache refresh         |
+| [`OPENSEARCH_GO_DNS_DIAL_TIMEOUT`](#connection)                               | `30s`                        | DNS-cache dialer dial timeout         |
+| [`OPENSEARCH_GO_DNS_KEEP_ALIVE`](#connection)                                 | `30s`                        | DNS-cache dialer keep-alive           |
+| [`OPENSEARCH_GO_DNS_TIMEOUT`](#connection)                                    | `10s`                        | DNS-cache per-lookup refresh timeout  |
 | [`OPENSEARCH_GO_ROUTER`](#routing)                                            | `true`                       | Auto-construct DefaultRouter          |
 | [`OPENSEARCH_GO_ROUTING_CONFIG`](#routing)                                    | all enabled                  | Shard-exact and adaptive MCSR toggles |
 | [`OPENSEARCH_GO_SHARD_COST`](#routing)                                        | defaults                     | Shard cost multipliers                |
@@ -49,6 +53,10 @@ Build, test, and code-generation variables (not read by the client at runtime) a
 | Variable | Accepted values | Default | Meaning | See also |
 | --- | --- | --- | --- | --- |
 | `OPENSEARCH_URL` | Comma-separated URL list (e.g. `https://a:9200,https://b:9200`) | unset | Seed addresses used by `NewClient` when no `Addresses` are set programmatically. | [opensearchapi/README.md Client Creation](../opensearchapi/README.md#client-creation); [Security: Credential Management](config-security.md#credential-management) |
+| `OPENSEARCH_GO_DNS_CACHE_REFRESH` | Duration or seconds | `60s` | Client-side DNS cache refresh interval. Resolved addresses are re-resolved on this cadence; if the resolver becomes briefly unreachable, the last-known-good address is served until it recovers, so a transient DNS outage does not fail requests to already-resolved hosts. `0` or unset = default (`60s`); negative = disable caching; positive = explicit interval. Installed only on the built-in transport; a caller-supplied `Transport` is never modified. Because Go's resolver does not expose record TTLs, this is a re-resolution cadence, not a per-record TTL. Overrides `Config.DNSCacheRefresh`. | [opensearchapi/README.md Client Creation](../opensearchapi/README.md#client-creation) |
+| `OPENSEARCH_GO_DNS_DIAL_TIMEOUT` | Duration or seconds | `30s` | Dial timeout of the `net.Dialer` behind the DNS cache. `0` or unset = default (`30s`); negative = no dial timeout; positive = explicit timeout. Only applies when the cache is installed (no custom `Transport`). Overrides `Config.DNSDialTimeout`. | [opensearchapi/README.md Client Creation](../opensearchapi/README.md#client-creation) |
+| `OPENSEARCH_GO_DNS_KEEP_ALIVE` | Duration or seconds | `30s` | Keep-alive interval of the `net.Dialer` behind the DNS cache. `0` or unset = default (`30s`); negative = disable keep-alive probes; positive = explicit interval. Only applies when the cache is installed (no custom `Transport`). Overrides `Config.DNSKeepAlive`. | [opensearchapi/README.md Client Creation](../opensearchapi/README.md#client-creation) |
+| `OPENSEARCH_GO_DNS_TIMEOUT` | Duration or seconds | `10s` | Per-lookup timeout applied to each DNS cache refresh resolution. Refresh lookups run sequentially on a single goroutine, so this bounds how long one stuck resolution can stall a refresh tick. `0` or unset = default (`10s`); negative = no per-lookup timeout; positive = explicit timeout. Only applies when the cache is installed (no custom `Transport`). Overrides `Config.DNSTimeout`. | [opensearchapi/README.md Client Creation](../opensearchapi/README.md#client-creation) |
 
 ## Routing
 

guides/transport-metrics.md

@@ -70,6 +70,16 @@ The top-level `Metrics` struct contains aggregate counters, per-connection detai
 | `HealthChecksSuccess` | `health_checks_success` | `int` | Successful health check outcomes                   |
 | `HealthChecksFailed`  | `health_checks_failed`  | `int` | Failed health check outcomes                       |
 
+### DNS Cache Counters
+
+Recorded by the client-side DNS cache installed on the built-in transport (see [`DNSCacheRefresh`](config-envvars.md#connection)). All zero when a custom `Transport` is supplied or caching is disabled.
+
+| Field             | JSON                | Type  | Description                                                         |
+| ----------------- | ------------------- | ----- | ------------------------------------------------------------------- |
+| `DNSLookups`      | `dns_lookups`       | `int` | Dials that consulted the DNS cache                                  |
+| `DNSCacheMisses`  | `dns_cache_misses`  | `int` | Lookups not served from cache (cold, or re-resolved after eviction) |
+| `DNSLookupErrors` | `dns_lookup_errors` | `int` | Lookups that returned a resolution error                            |
+
 ---
 
 ## ConnectionMetric

internal/envvars/envvars.go

@@ -47,6 +47,27 @@ const ShardRequests = "OPENSEARCH_GO_SHARD_REQUESTS"
 // RequestTimeout overrides the per-attempt HTTP round-trip timeout.
 const RequestTimeout = "OPENSEARCH_GO_REQUEST_TIMEOUT"
 
+// DNSCacheRefresh overrides the client-side DNS cache refresh interval, which
+// also bounds how long a stale (last-known-good) address is served when the
+// resolver is briefly unreachable. time.ParseDuration format, integer seconds,
+// or float seconds. 0 = default, <0 = disable caching, >0 = explicit interval.
+const DNSCacheRefresh = "OPENSEARCH_GO_DNS_CACHE_REFRESH"
+
+// DNSDialTimeout overrides the dial timeout of the net.Dialer behind the
+// client-side DNS cache. Same value format as DNSCacheRefresh.
+// 0 = default (30s), <0 = no dial timeout, >0 = explicit timeout.
+const DNSDialTimeout = "OPENSEARCH_GO_DNS_DIAL_TIMEOUT"
+
+// DNSKeepAlive overrides the keep-alive interval of the net.Dialer behind the
+// client-side DNS cache. Same value format as DNSCacheRefresh.
+// 0 = default (30s), <0 = disable keep-alive probes, >0 = explicit interval.
+const DNSKeepAlive = "OPENSEARCH_GO_DNS_KEEP_ALIVE"
+
+// DNSTimeout overrides the per-lookup timeout applied to each cache refresh
+// resolution. Same value format as DNSCacheRefresh.
+// 0 = default (10s), <0 = no per-lookup timeout, >0 = explicit timeout.
+const DNSTimeout = "OPENSEARCH_GO_DNS_TIMEOUT"
+
 // NodeStatsInterval overrides the node stats polling interval.
 const NodeStatsInterval = "OPENSEARCH_GO_NODE_STATS_INTERVAL"
 

opensearch.go

@@ -120,6 +120,28 @@ type Config struct {
 	// 0 = no per-attempt timeout (default), >0 = explicit timeout.
 	RequestTimeout time.Duration
 
+	// DNSCacheRefresh controls how often the client-side DNS cache re-resolves
+	// cached hostnames. The cache is installed only when no custom Transport is
+	// provided; a caller-supplied Transport is never modified.
+	// 0 = default (60s), <0 = disable caching, >0 = explicit interval.
+	DNSCacheRefresh time.Duration
+
+	// DNSDialTimeout sets the dial timeout of the net.Dialer behind the
+	// client-side DNS cache; only applies when the cache is installed.
+	// 0 = default (30s), <0 = no dial timeout, >0 = explicit timeout.
+	DNSDialTimeout time.Duration
+
+	// DNSKeepAlive sets the keep-alive interval of the net.Dialer behind the
+	// client-side DNS cache; only applies when the cache is installed.
+	// 0 = default (30s), <0 = disable keep-alive probes, >0 = explicit interval.
+	DNSKeepAlive time.Duration
+
+	// DNSTimeout bounds each cache refresh lookup behind the client-side DNS
+	// cache; only applies when the cache is installed. Prevents one hung
+	// resolution from stalling a (sequential) refresh tick.
+	// 0 = default (10s), <0 = no per-lookup timeout, >0 = explicit timeout.
+	DNSTimeout time.Duration
+
 	CompressRequestBody bool // Default: false.
 
 	// DiscoverNodesOnStart triggers an asynchronous discovery cycle as soon
@@ -318,6 +340,11 @@ func NewClient(cfg Config) (*Client, error) {
 		RetryBackoff:         cfg.RetryBackoff,
 		RequestTimeout:       cfg.RequestTimeout,
 
+		DNSCacheRefresh: cfg.DNSCacheRefresh,
+		DNSDialTimeout:  cfg.DNSDialTimeout,
+		DNSKeepAlive:    cfg.DNSKeepAlive,
+		DNSTimeout:      cfg.DNSTimeout,
+
 		CompressRequestBody: cfg.CompressRequestBody,
 
 		EnableMetrics:     cfg.EnableMetrics,