Troubleshooting guide for AWS credential errors

[unexge]

It is a common pain point to set up AWS credentials in Kubernetes/CSI Driver as there are various ways to configure credential source (i.e., driver-level and pod-level) and credential mechanism (i.e., IMDS, long-term AWS credentials, EKS Pod Identity, IRSA). Ideally, there should be a troubleshooting guide that explains common misconfigurations and how to access credential related logs.

Just sharing some random stuff that could be useful to add into a troubleshooting guide:

Getting credential related logs

CRT handles obtaining AWS credentials for Mountpoint. But Mountpoint does not emit CRT logs by default currently. To get credential related logs, one should be adding debug-crt to their mountOptions to enable (debug) logs for CRT. Since debug logs could be verbose, filtering to credential related logs would be very useful. Ideally running something like:

$ kubectl logs -n mount-s3 mp-... | grep -e AuthCredentialsProvider -e AWSProfile

Should return credential related CRT logs for a given Mountpoint Pod for troubleshooting.

Validating IRSA configuration

There are various ways for a IRSA configuration to go wrong as one needs to configure multiple resources (e.g., service accounts, IAM roles, OIDC provider etc.). There is this script to check various resources to ensure they're configured correctly: #276 (comment). We can put some form of this script to a troubleshooting guide for IRSA errors.

More helpful logs if there are no credentials configured

The CSI Driver emits logs for each credential mechanism it can source credentials from except node-level credentials from IMDS. The CSI Driver also knows whether IMDS is accessible, it would be nice for CSI Driver to emit logs when:

• There are no credentials configured (i.e., env in this line is empty), and only way for Mountpoint to access to AWS is node-level credentials from IMDS.
  • The CSI Driver can emit a log line saying No explicit credentials are configured, ensure containers can access to IMDS, and node has a sufficient permissions to call S3
  • The CSI Driver can also emit a log line if it knows IMDS is not accessible No explicit credentials are configured and looks like containers cannot access to IMDS. Mountpoint might not be able to access S3, please follow https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/CONFIGURATION.md#aws-credentials to configure correct credentials