Java SDK cannot read EKS Pod Identity credentials on IPv6 clusters (jobs hang in Running)

[mikluko]

Bug description

On an IPv6 Amazon EKS cluster using EKS Pod Identity for S3 access, Terrakube's Java components (executor, api, registry) fail to load AWS credentials. Terraform's own (Go) provider works, so terraform plan runs and completes, but the executor then throws while uploading state/output to S3, and the job hangs indefinitely in "Running" — logs stream fine, but the status never transitions.

Root cause

On an IPv6 cluster the Pod Identity webhook injects the IPv6 credentials endpoint:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://[fd00:ec2::23]/v1/credentials

The bundled AWS SDK for Java v2 (sdk-core 2.46.17) rejects it:

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain ...
ContainerCredentialsProvider(): The full URI (http://[fd00:ec2::23]/v1/credentials) contained within
environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host.
Host should resolve to a loopback address or have the full URI be HTTPS.

ContainerCredentialsProvider only allows the IPv6 Pod Identity host when AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE=IPv6. Other AWS SDKs (C++, botocore, Go v2, JS v3) allow it unconditionally, which is why Terraform (Go) succeeds on the same pod while the Java layer fails.

Environment

• Terrakube 2.32.0 (Helm chart 4.7.4)
• Amazon EKS, IPv6 cluster, eks-pod-identity-agent addon
• S3 storage with AwsEnableRoleAuth=true, executor AwsIncludeBackendKeys=false
• Executor image AWS SDK: sdk-core 2.46.17

Workaround (confirmed working)

Set on api/registry/executor:

env:
  - name: AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE
    value: IPv6

Side effect (redirecting IMDS to the IPv6 endpoint) is irrelevant because Terrakube uses Pod Identity, not IMDS.

References

• fix: allow IPv6 EKS Pod Identity host regardless of IMDS endpoint mode aws/aws-sdk-java-v2#7086 — "allow IPv6 EKS Pod Identity host regardless of IMDS endpoint mode" (open, unreleased at time of writing)
• [EKS] [eks-pod-identity][request]: Explicitly set AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE environment variable aws/containers-roadmap#2683