[delegatedauth] Support all AWS credential sources in every Agent flavor | WIF-75 (#54318)

### What does this PR do?

Removes the `ec2` build tag from Agent Cloud Auth's AWS credential resolution, so every Agent flavor can use EKS IRSA, ECS task roles, EKS Pod Identity and EC2 IMDS. Newly covered: trace-agent, standalone Dogstatsd, private action runner, IoT, Heroku. `ec2` still gates DBM and EC2 host tagging, untouched.

### Why

`ec2` also gates DBM and EC2 host tagging, so flavors that opted out of those silently lost Cloud Auth with them. Most importantly APM has never supported it, and `comp/trace/config/fx` sets `FailIfAPIKeyMissing: true`, so a customer using Cloud Auth for APM can't drop the static `api_key`.

### What changed

**Build-tag removal.** The three `!ec2` stubs are deleted and the tag dropped from credential resolution and detection. `pkg/util/aws/creds` had no other consumer. No replacement tag: gating cost +56 KiB per excluded flavor, and gating is what created the silent gap to begin with.

**Two SDK providers replaced.** IRSA and IMDS were already resolved by our own code. The remaining two SDK dependencies are gone: `credentials/endpointcreds` for the ECS/EKS container leg (replaced by a GET + JSON decode) and `credentials.NewStaticCredentialsProvider` (a 2-line struct). `aws-sdk-go-v2/aws` and `signer/v4` stay, both already linked for the SigV4 proof. No change to the STS or SigV4 paths.

The container replacement keeps the retry behavior `endpointcreds` provided: connection errors, truncated bodies, 5xx and 429 are retried, bounded by an attempt cap and an elapsed budget, while a 403 or a malformed document is not. IRSA and IMDS have no inner retries, unchanged, so the three legs are intentionally not uniform.

**Diagnosability.** Detection returns why it failed rather than a bool, failures name the one mechanism that was tried and what to check for it, and blank-but-error-free credentials are now a failure rather than a false "resolved". A half-configured credential pair (ex: `AWS_ACCESS_KEY_ID` without `AWS_SECRET_ACCESS_KEY`) logs a warning naming it, and is still skipped rather than erroring, matching the AWS SDK. `agent status` reports disable reason, per-key credential source, last and next refresh, and last error.

**Config precedence fix.** Setting only `delegated_auth.aws.region` built a non-nil `ProviderConfig`, which downstream reads as "explicitly configured" and skipped provider detection entirely. The region is now applied to the auto-detected provider.

**`cloud_provider_metadata` opt-out.** Three IMDS entry points reached `DoHTTPRequest` directly rather than through `GetMetadataItem`, so an operator who excluded `aws` was probed anyway. All three now check first. This predates the PR, but removing the build tag is what exposes it to flavors that otherwise never touch IMDS.

### Size

| Build | vs `main` |
|---|---|
| trace-agent binary | +73.2 KiB |
| `iot_agent_deb_amd64` package | +4.03 KiB |
| `agent_rpm_arm64` package | +55.97 KiB |

Static quality gates moved +0.06 MiB on `agent_rpm_arm64` and `agent_suse_arm64`, and nowhere else.

Keeping `endpointcreds` would have cost +404.8 KiB instead of +73.2 KiB on the trace-agent, almost all of it smithy-go middleware and its reflection metadata, and would have put IoT over its hard limit.

The main Agent package grows even though its binary already had this code: `omnibus/config/software/datadog-agent.rb` ships `trace-agent` and `privateactionrunner` inside it, and those two are what newly compile the credential code.

### Deliberately not in this PR

- `otel-agent` (DDOT) still uses the noop delegated-auth module.
- Adding inner retries to the IRSA and IMDS legs.

### Describe how you validated your changes

**Staging end to end** on `charcadet.us1.staging.dog`, two deploys differing only in the agent image. All four flavors on the cluster (`agent`, `trace-agent`, `process-agent`, `private-action-runner`) go from `missing AWS credentials` to a delegated key, verified across 16 pods and 48 containers. Per-phase log links and reasoning in [this comment](#54318 (comment)). IRSA and static-env credentials on real AWS are not covered end to end and remain for `qa/rc-required`.

**Local** (`trace-agent`, default flavor tags, no `ec2`): container-credentials path reached end to end against a stub endpoint. A pre-PR non-`ec2` build stops in the `_noec2` stub and cannot get this far.

**Tests.** `bzl test //comp/core/delegatedauth/...:all //pkg/util/aws/creds:all` passes, including flavor targets that never compiled this code before. New coverage for the container provider (each retry class asserts exact request counts, so the non-retry cases cannot silently start retrying), region precedence, detection-failure recording, status fields, and the metadata opt-out.


Co-authored-by: stephen.rosenthal <stephen.rosenthal@datadoghq.com>

Author: srosenthal-dd

comp/core/delegatedauth/api/cloudauth/aws/BUILD.bazel

@@ -5,8 +5,7 @@ go_library(
     name = "aws",
     srcs = [
         "aws.go",
-        "resolve_credentials_ec2.go",
-        "resolve_credentials_noec2.go",
+        "resolve_credentials.go",
     ],
     importpath = "github.com/DataDog/datadog-agent/comp/core/delegatedauth/api/cloudauth/aws",
     visibility = ["//visibility:public"],
@@ -20,26 +19,20 @@ go_library(
         "//pkg/version",
         "@com_github_aws_aws_sdk_go_v2//aws",
         "@com_github_aws_aws_sdk_go_v2//aws/signer/v4:signer",
-        "@com_github_aws_aws_sdk_go_v2_credentials//:credentials",
-        "@com_github_aws_aws_sdk_go_v2_credentials//endpointcreds",
     ],
 )
 
 dd_agent_go_test(
     name = "aws_test",
     srcs = [
         "aws_test.go",
-        "resolve_credentials_ec2_test.go",
-        "resolve_credentials_noec2_test.go",
+        "resolve_credentials_providers_test.go",
         "resolve_credentials_test.go",
     ],
     embed = [":aws"],
-    gotags_sets = [["ec2"]],
     deps = [
         "//pkg/config/mock",
         "//pkg/util/aws/creds",
-        "@com_github_aws_aws_sdk_go_v2_credentials//:credentials",
-        "@com_github_aws_aws_sdk_go_v2_credentials//endpointcreds",
         "@com_github_stretchr_testify//assert",
         "@com_github_stretchr_testify//require",
     ],

comp/core/delegatedauth/api/cloudauth/aws/aws.go

@@ -16,6 +16,7 @@ import (
 	"errors"
 	"fmt"
 	"net/http"
+	"sync/atomic"
 	"time"
 
 	cloudauthconfig "github.com/DataDog/datadog-agent/comp/core/delegatedauth/api/cloudauth/config"
@@ -25,7 +26,6 @@ import (
 
 	pkgconfigmodel "github.com/DataDog/datadog-agent/pkg/config/model"
 	"github.com/DataDog/datadog-agent/pkg/util/aws/creds"
-	"github.com/DataDog/datadog-agent/pkg/util/log"
 	"github.com/DataDog/datadog-agent/pkg/version"
 )
 
@@ -53,6 +53,13 @@ const (
 // AWSAuth contains the implementation for the AWS cloud auth
 type AWSAuth struct {
 	region string
+
+	// lastSource names the credential mechanism selected by the most recent resolveCredentials
+	// attempt (ex: "DelegatedAuthIMDS"). It is written before the credentials are fetched, so it
+	// reflects what was attempted rather than what succeeded; see CredentialSourceReporter. Read by
+	// the status page via common.CredentialSourceReporter. Credentials are resolved on the
+	// delegated-auth refresh goroutine and read by whoever renders status, so it is atomic.
+	lastSource atomic.Pointer[string]
 }
 
 // NewAWSAuth creates a new AWSAuth from an AWSProviderConfig.
@@ -66,6 +73,20 @@ func NewAWSAuth(config *cloudauthconfig.AWSProviderConfig) *AWSAuth {
 	}
 }
 
+// Compile-time check that the status page's optional interface stays satisfied. Without it a rename
+// would silently drop the credential source from `agent status` rather than fail the build.
+var _ common.CredentialSourceReporter = (*AWSAuth)(nil)
+
+// LastCredentialSource implements common.CredentialSourceReporter. It names the mechanism selected
+// for the most recent resolution attempt, whether or not that attempt succeeded, so it is populated
+// exactly when an operator needs it most.
+func (a *AWSAuth) LastCredentialSource() string {
+	if s := a.lastSource.Load(); s != nil {
+		return *s
+	}
+	return ""
+}
+
 // GenerateAuthProof generates an AWS-specific authentication proof using SigV4 signing.
 // This proof includes a signed AWS STS GetCallerIdentity request that proves access to AWS credentials.
 // The context parameter allows for cancellation of the proof generation.
@@ -75,14 +96,20 @@ func (a *AWSAuth) GenerateAuthProof(ctx context.Context, cfg pkgconfigmodel.Read
 		return "", ctx.Err()
 	}
 
-	// Get local AWS Credentials. cfg is threaded through so the IRSA web-identity STS call can use
-	// the Agent's configured HTTP transport (proxy / custom CA / TLS settings).
-	credentials := a.getCredentials(ctx, cfg)
-
 	if config == nil || config.OrgUUID == "" {
 		return "", errors.New("missing org UUID in config")
 	}
 
+	// Get local AWS Credentials. cfg is threaded through so the IRSA web-identity STS call can use
+	// the Agent's configured HTTP transport (proxy / custom CA / TLS settings). The error names the
+	// credential mechanism that was tried and why it failed, and is returned rather than only logged
+	// so it reaches the caller's error log and the status page instead of a generic
+	// "missing AWS credentials".
+	credentials, err := a.getCredentials(ctx, cfg)
+	if err != nil {
+		return "", err
+	}
+
 	// Use the credentials to generate the signing data
 	data, err := a.generateAwsAuthData(ctx, config.OrgUUID, credentials)
 	if err != nil {
@@ -99,18 +126,56 @@ func (a *AWSAuth) GenerateAuthProof(ctx context.Context, cfg pkgconfigmodel.Read
 	return authProof, nil
 }
 
-// getCredentials retrieves AWS credentials using the build-tag-selected chain:
-//   - ec2 build: AWS SDK chain limited to env -> web identity -> container -> IMDS (shared config/SSO/profiles disabled)
-//   - non-ec2 build: static env vars only (AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY)
-func (a *AWSAuth) getCredentials(ctx context.Context, cfg pkgconfigmodel.Reader) *creds.SecurityCredentials {
-	resolved := a.resolveCredentials(ctx, cfg)
-	if resolved == nil {
-		return &creds.SecurityCredentials{}
+// getCredentials retrieves AWS credentials from the environment the Agent is running in, using the
+// one mechanism that matches it: env, IRSA web identity, ECS/EKS container credentials, or IMDS
+// (shared config/SSO/profiles are deliberately unsupported, see resolveCredentials).
+//
+// A failure here means delegated auth cannot fetch an API key at all, so the error is annotated
+// with what the operator should look at for the mechanism that was actually tried, then returned.
+// Nothing is logged here: the caller already reports the failure, and logging as well would emit
+// the same remediation text three times per attempt.
+func (a *AWSAuth) getCredentials(ctx context.Context, cfg pkgconfigmodel.Reader) (*creds.SecurityCredentials, error) {
+	resolved, err := a.resolveCredentials(ctx, cfg)
+	if err != nil {
+		// A canceled context means the Agent is shutting down or this instance was replaced, not that
+		// the environment is misconfigured. Return it unadorned so callers can recognize it and skip
+		// the remediation hint.
+		if ctx.Err() != nil {
+			return nil, err
+		}
+		return nil, fmt.Errorf("%w. %s", err, credentialRemediation(a.LastCredentialSource(), cfg))
 	}
-	if resolved.AccessKeyID == "" || resolved.SecretAccessKey == "" {
-		log.Debugf("AWS credential resolution returned empty credentials")
+	return resolved, nil
+}
+
+// credentialRemediation returns the check to suggest for the credential mechanism that was tried.
+// Naming only that mechanism matters because the chain is first-match: telling an IRSA pod to
+// verify IMDS reachability sends the operator down a path the Agent never took.
+func credentialRemediation(source string, cfg pkgconfigmodel.Reader) string {
+	switch source {
+	case creds.SourceEnvironment:
+		return "AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set, so no other credential source was tried; check that they are valid and not expired"
+	case creds.SourceWebIdentity:
+		return "AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are set, so IRSA was used; check that the token file is mounted and readable and that the role's trust policy allows this service account"
+	case creds.SourceContainer:
+		return "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI is set, so ECS/EKS container credentials were used; check that the task role or Pod Identity association exists and that the credential endpoint is reachable"
+	case creds.SourceIMDS:
+		// Report the IMDS versions actually attempted rather than ec2_prefer_imdsv2 alone. The IMDS
+		// helper allows v2 when either ec2_prefer_imdsv2 or ec2_imdsv2_transition_payload_enabled is
+		// set (UseIMDSv2 in pkg/util/aws/creds/internal), and the latter defaults to true, so naming
+		// only the first key tells a default-configured operator that v2 was not tried when it was.
+		imdsVersions := "v1 only"
+		if cfg.GetBool("ec2_prefer_imdsv2") || cfg.GetBool("ec2_imdsv2_transition_payload_enabled") {
+			imdsVersions = "v2 then v1"
+		}
+		return fmt.Sprintf("no credential environment variables were set, so EC2 IMDS was used; check that an instance profile is attached and that IMDS is reachable (ec2_metadata_timeout=%dms, IMDS versions attempted=%s)",
+			cfg.GetInt("ec2_metadata_timeout"), imdsVersions)
+	default:
+		// The source is unknown only if selection itself failed before recording one. Say nothing
+		// specific rather than defaulting to IMDS advice, which would misdirect exactly the way the
+		// old catch-all message did.
+		return "the credential mechanism could not be determined"
 	}
-	return resolved
 }
 
 func (a *AWSAuth) getConnectionParameters() (string, string, string) {

comp/core/delegatedauth/api/cloudauth/aws/aws_test.go

@@ -9,11 +9,13 @@ import (
 	"context"
 	"encoding/base64"
 	"encoding/json"
+	"fmt"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
+	configmock "github.com/DataDog/datadog-agent/pkg/config/mock"
 	"github.com/DataDog/datadog-agent/pkg/util/aws/creds"
 )
 
@@ -217,3 +219,51 @@ func TestGenerateAwsAuthDataWithoutToken(t *testing.T) {
 	// X-Amz-Security-Token should NOT be present for permanent credentials
 	assert.NotContains(t, headers, "X-Amz-Security-Token")
 }
+
+func TestCredentialRemediationIMDSVersions(t *testing.T) {
+	// The IMDS leg allows v2 when either ec2_prefer_imdsv2 or
+	// ec2_imdsv2_transition_payload_enabled is set (UseIMDSv2 in pkg/util/aws/creds/internal), and
+	// the latter defaults to true. Reporting ec2_prefer_imdsv2 alone told an operator running the
+	// default configuration that v2 was not attempted when it was.
+	tests := []struct {
+		name       string
+		preferV2   bool
+		transition bool
+		want       string
+	}{
+		{name: "defaults attempt v2", preferV2: false, transition: true, want: "v2 then v1"},
+		{name: "prefer_imdsv2 alone", preferV2: true, transition: false, want: "v2 then v1"},
+		{name: "both set", preferV2: true, transition: true, want: "v2 then v1"},
+		{name: "neither set is v1 only", preferV2: false, transition: false, want: "v1 only"},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			cfg := configmock.NewFromYAML(t, fmt.Sprintf(
+				"ec2_prefer_imdsv2: %t\nec2_imdsv2_transition_payload_enabled: %t\n",
+				tc.preferV2, tc.transition))
+
+			got := credentialRemediation(creds.SourceIMDS, cfg)
+
+			assert.Contains(t, got, "IMDS versions attempted="+tc.want)
+			// The old key must not reappear: it is the config value, not the effective behavior.
+			assert.NotContains(t, got, "ec2_prefer_imdsv2=")
+		})
+	}
+}
+
+func TestCredentialRemediationNamesOnlyTheAttemptedSource(t *testing.T) {
+	// The chain is first-match, so remediation must describe one mechanism. Advising an IRSA pod to
+	// check IMDS reachability sends the operator down a path the Agent never took.
+	cfg := configmock.New(t)
+
+	assert.Contains(t, credentialRemediation(creds.SourceWebIdentity, cfg), "IRSA was used")
+	assert.NotContains(t, credentialRemediation(creds.SourceWebIdentity, cfg), "IMDS")
+
+	assert.Contains(t, credentialRemediation(creds.SourceContainer, cfg), "container credentials were used")
+	assert.NotContains(t, credentialRemediation(creds.SourceContainer, cfg), "IMDS")
+
+	assert.Contains(t, credentialRemediation(creds.SourceEnvironment, cfg), "no other credential source was tried")
+
+	// An unrecognized source must not fall back to IMDS advice.
+	assert.Equal(t, "the credential mechanism could not be determined", credentialRemediation("", cfg))
+}

comp/core/delegatedauth/api/cloudauth/aws/go.mod

@@ -11,7 +11,6 @@ require (
 	github.com/DataDog/datadog-agent/pkg/util/log v0.73.0-rc.5
 	github.com/DataDog/datadog-agent/pkg/version v0.72.2
 	github.com/aws/aws-sdk-go-v2 v1.43.3
-	github.com/aws/aws-sdk-go-v2/credentials v1.19.33
 	github.com/stretchr/testify v1.11.1
 )