Kingdomless Halo Panel Exchange fails on AWS EKS worker node with IMDSv2 enabled.

[shuhelamin]

Description of the bug:
The Kingdomless Halo Panel Exchange application fails during execution when running on AWS EKS with IMDSv2(Insatnce MetaData Service version-2) enforced. Currently, the application is running with IMDSv1 enabled. However, IMDSv1 usage has been identified as a security vulnerability by our internal security team. Additionally, for AWS EKS versions 1.33 and higher, IMDSv2 is the recommended and default configuration. After enforcing IMDSv2, the application encounters failures while attempting to retrieve credentials from the AWS EC2 Instance Metadata endpoint.
Platform: AWS EKS
Deployment Type: Kubernetes CronJob
Node(EC2) Configuration: IMDSv2 enforced.

Steps to reproduce:

1. Deploy the Kingdomless Halo Panel Exchange application as a Deployment/CronJob on AWS EKS
2. Enforce IMDSv2 on the EKS worker nodes(EC2).
3. Run the Deployment/CronJob

Component(s) affected:
Kingdomless Halo Panel Exchange application

Version:
The release version is: 0.5.27

Environment: Prod (Origin)

Additional context:
Application logs show the following error:
Caused by: java.io.IOException: Failed to retrieve AWS IAM role.
at com.google.auth.oauth2.InternalAwsSecurityCredentialsSupplier.retrieveResource(InternalAwsSecurityCredentialsSupplier.java:216)
at com.google.auth.oauth2.InternalAwsSecurityCredentialsSupplier.retrieveResource(InternalAwsSecurityCredentialsSupplier.java:193)
at com.google.auth.oauth2.InternalAwsSecurityCredentialsSupplier.getCredentials(InternalAwsSecurityCredentialsSupplier.java:112)
at com.google.auth.oauth2.AwsCredentials.retrieveSubjectToken(AwsCredentials.java:141)
at com.google.auth.oauth2.AwsCredentials.refreshAccessToken(AwsCredentials.java:121)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:270)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:267)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.auth.oauth2.OAuth2Credentials$RefreshTask.run(OAuth2Credentials.java:635)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:582)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:233)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:183)
at com.google.auth.oauth2.ExternalAccountCredentials.getRequestMetadata(ExternalAccountCredentials.java:343)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:199)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.storage.spi.v1.HttpStorageRpc$InvocationIdInitializer.initialize(HttpStorageRpc.java:169)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:565)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:555)
... 18 more
Caused by: com.google.api.client.http.HttpResponseException: 401 Unauthorized
GET http://169.254.169.254/latest/meta-data/iam/security-credentials

[TNATALI]

Thank you. Acknowleding this issue. We will work on this once we have our resources available.

[TNATALI]

@shuhelamin What is the impact of this issue? Can you give us more context and information?

[shuhelamin]

@shuhelamin What is the impact of this issue? Can you give us more context and information?

Impact:
Currently, there is no immediate impact, as we have reverted the changes and are using IMDSv1 on the worker nodes. However, IMDSv1 cannot be used long term since it is considered less secure. IMDSv2 is the recommended and more secure option and continued use of IMDSv1 is not acceptable from a security standpoint.

Context:
Our Kingdomless Halo Panel Exchange is running on an AWS EKS cluster, where the worked nodes are currently configured with IMDSv1. This configuration has been flagged as a security vulnerability by our security team, and we have been advised to migrate the worker nodes to IMDSv2.
The Halo application needs to access AWS service such as S3 buckets and relies on AWS credentials obtained via the IAM role attached to the worker nodes. These credentials are retrieved using AWS Instance Metadata Service (IMDS).
IMDS enables applications to securely obtain temporary credentials associated with the IAM role. IMDSv2 introduces a session-based mechanism with additional security protections, making it more secure than IMDSv1. After enforcing IMDSv2, the application fails to retrieve the IAM role credentials, which results in runtime failures.