[BUG] SigV4 signer not applied to background health-check/discovery/node-stats requests (continuous 403s on Amazon OpenSearch Service)

[mifisignal]

What is the bug?

opensearchtransport applies the configured Signer (e.g. signer/awsv2 SigV4) only on the Perform path. The background health-check, discovery, and node-stats pollers build their own request and dispatch it with a raw c.transport.RoundTrip(req), calling setReqAuth but never signRequest.

setReqAuth only ever sets HTTP basic auth:

func (c *Client) setReqAuth(u *url.URL, req *http.Request) {
	if _, ok := req.Header["Authorization"]; !ok {
		if u.User != nil { ... req.SetBasicAuth(...); return }
		if c.username != "" && c.password != "" { req.SetBasicAuth(c.username, c.password); return }
	}
}

When the client is configured for SigV4, there is no username/password, so these background requests leave the process with no credentials at all. Against Amazon OpenSearch Service (IAM-principal-based access policy) every one is rejected with HTTP 403.

The failures are swallowed by the pollers, so there is no error surfaced to the application.

Affected call sites

signRequest is called from only two places, both on the Perform path (opensearchtransport.go, currently ~L1542 and ~L1878). Seven other sites dispatch via raw RoundTrip after setReqAuth only:

File	Approx. lines (main)	Request
opensearchtransport/cluster_health.go	202 / 209	GET /_nodes/_local/stats/jvm,breaker,thread_pool (fetchAndEvaluateNodeStats)
opensearchtransport/discovery.go	1125 / 1128	discovery probe
opensearchtransport/discovery.go	1914 / 1917	discovery probe
opensearchtransport/discovery.go	2167 / 2170	discovery probe
opensearchtransport/opensearchtransport.go	2214 / 2221	health probe
opensearchtransport/opensearchtransport.go	2293 / 2300	health probe
opensearchtransport/opensearchtransport.go	2425 / 2432	health probe

Shape of the defect, from cluster_health.go:

req, err := http.NewRequestWithContext(ctx, http.MethodGet, "/_nodes/_local/stats/jvm,breaker,thread_pool", nil)
...
c.setReqURL(conn.URL, req)
c.setReqAuth(conn.URL, req)      // basic auth only -- signer never invoked
c.setReqUserAgent(req)
...
res, err := c.transport.RoundTrip(req)   // bypasses Perform, so bypasses signRequest

How can one reproduce the bug?

1. Point a client at an Amazon OpenSearch Service domain (managed, VPC, fine-grained access control disabled, IAM-based access policy).
2. Configure SigV4 via signer/awsv2, with no Username/Password set.
3. Issue normal indexing/search traffic so the transport establishes ready connections and starts its background pollers.
4. Observe requests to GET /_nodes/_local/stats/jvm,breaker,thread_pool returning 403, at the NodeStatsInterval cadence (default auto-derived, 5–30s), per ready connection.

What is the expected behavior?

Background health-check, discovery, and node-stats requests should be signed with the same Signer as application requests, so they authenticate identically. Concretely, the raw-dispatch paths should call signRequest(req) before c.transport.RoundTrip(req).

Impact

• Per-pool AIMD congestion control and overload detection (NodeStatsInterval, OverloadedHeapThreshold, OverloadedBreakerRatio) never receive a single sample on Amazon OpenSearch Service with SigV4, so no congestion window is ever adjusted and no overloaded node is ever demoted. The feature is inert while appearing configured.
• Continuous authorization failures against the domain for the lifetime of every client, invisible to the application because the poller discards the error.
• No configuration or IAM change can work around it: the identity and domain policies can already permit the path, but an unsigned request presents no principal to authorize.

The only available mitigation is to disable the feature entirely, e.g. OPENSEARCH_GO_DISCOVERY_CONFIG=-node_stats.

What is your host/environment?

• github.com/opensearch-project/opensearch-go/v4 v4.7.1
• Go client against Amazon OpenSearch Service OpenSearch_3.7, VPC domain, FGAC disabled, IAM/SigV4 data-plane access.