Skip to content

Improve ADOT and IRSA docs: automate RBAC, dedicated ServiceAccounts #10414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
snarkychef wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Improve ADOT and IRSA docs: automate RBAC, dedicated ServiceAccounts #10414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 98 additions & 102 deletions docs/content/en/docs/getting-started/optional/irsa.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ IAM Roles for Service Account (IRSA) enables applications running in clusters to

The steps below are based on the [guide for configuring IRSA for DIY Kubernetes,](https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md) with modifications specific to EKS Anywhere's cluster provisioning workflow. The main modification is the process of generating the keys.json document. As per the original guide, the user has to create the service account signing keys, and then use that to create the keys.json document prior to cluster creation. This order is reversed for EKS Anywhere clusters, so you will create the cluster first, and then retrieve the service account signing key generated by the cluster, and use it to create the keys.json document. The sections below show how to do this in detail.

### Create an OIDC provider and make its discovery document publicly accessible
#### Create an OIDC provider and make its discovery document publicly accessible

You must use a single OIDC provider per EKS Anywhere cluster, which is the best practice to prevent a token from one cluster being used with another cluster. These steps describe the process of using a S3 bucket to host the OIDC `discovery.json` and `keys.json` documents.

Expand Down Expand Up @@ -54,53 +54,6 @@ You must use a single OIDC provider per EKS Anywhere cluster, which is the best

1. Make a note of the `Provider` field of OIDC provider after it is created.

1. Assign an IAM role to the OIDC provider.

1. Navigate to the AWS IAM Console.

1. Click on the OIDC provider.

1. Click _Assign role_.

1. Select _Create a new role_.

1. Select _Web identity_ as the trusted entity.

1. In the _Web identity_ section:

* If your _Identity provider_ is not auto selected, select it.
* Select `sts.amazonaws.com` as the _Audience_.

1. Click _Next_.

1. Configure your desired _Permissions poilicies_.

1. Below is a sample trust policy of IAM role for your pods. Replace `ACCOUNT_ID`, `ISSUER_HOSTPATH`, `NAMESPACE` and `SERVICE_ACCOUNT`.
_Example: Scoped to a service account_
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/ISSUER_HOSTPATH"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"ISSUER_HOSTPATH:sub": "system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT"
},
}
}
]
}
```

1. Create the IAM Role and make a note of the _Role name_.

1. After the cluster is created you can grant service accounts access to the role by modifying the trust relationship. See the [How to use trust policies with IAM Roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/) for more information on trust policies. Refer to [Configure the trust relationship for the OIDC provider's IAM Role](#configure-the-trust-relationship-for-the-oidc-providers-iam-role) for a working example.

### Create (or upgrade) the EKS Anywhere cluster

When creating (or upgrading) the EKS Anywhere cluster, you need to configure the kube-apiserver's `service-account-issuer` flag so it can issue and mount projected service account tokens in pods. For this, use the value obtained in the first section for `$ISSUER_HOSTPATH` as the `service-account-issuer`. Configure the kube-apiserver by setting this value through the EKS Anywhere cluster spec:
Expand Down Expand Up @@ -164,83 +117,126 @@ The [Amazon Pod Identity Webhook](https://github.com/aws/amazon-eks-pod-identity
make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest
```

1. Create a service account with an `eks.amazonaws.com/role-arn` annotation set to the IAM Role created for the OIDC provider.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-serviceaccount
namespace: default
annotations:
# set this with value of OIDC_IAM_ROLE
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/s3-reader"

# optional: Defaults to "sts.amazonaws.com" if not set
eks.amazonaws.com/audience: "sts.amazonaws.com"

# optional: When set to "true", adds AWS_STS_REGIONAL_ENDPOINTS env var
# to containers
eks.amazonaws.com/sts-regional-endpoints: "true"

# optional: Defaults to 86400 for expirationSeconds if not set
# Note: This value can be overwritten if specified in the pod
# annotation as shown in the next step.
eks.amazonaws.com/token-expiration: "86400"
```
### Create IAM role for your workload

1. Finally, apply the `my-service-account.yaml` file to create your service account.
Each workload (such as ADOT, fluentbit, cert-manager, or custom applications) needs an IAM role with the permissions it requires. Repeat this section for each workload that requires AWS access.

```bash
kubectl apply -f my-service-account.yaml
```
1. Navigate to the AWS IAM Console.

1. You can validate IRSA by following [IRSA setup and test]({{< relref "../../packages/adot/adot_amp_amg/#irsa-set-up-test" >}}). Ensure the awscli pod is deployed in the same namespace of ServiceAccount `pod-identity-webhook`.
1. Click on the OIDC provider created earlier.

1. Click _Assign role_.

### Configure the trust relationship for the OIDC provider's IAM Role
1. Select _Create a new role_.

In order to grant certain service accounts access to the desired AWS resources, edit the trust relationship for the OIDC provider's IAM Role (`OIDC_IAM_ROLE`) created in the first section, and add in the desired service accounts.
1. Select _Web identity_ as the trusted entity.

1. Choose the role in the console to open it for editing.
1. In the _Web identity_ section:
* If your _Identity provider_ is not auto selected, select it.
* Select `sts.amazonaws.com` as the _Audience_.

1. Choose the **Trust relationships** tab, and then choose **Edit trust relationship**.
1. Click _Next_.

1. Find the line that looks similar to the following:
```
"$ISSUER_HOSTPATH:aud": "sts.amazonaws.com"
```
1. Configure your desired _Permissions poilicies_.

1. Add another condition after that line which looks like the following line. Replace `KUBERNETES_SERVICE_ACCOUNT_NAMESPACE` and `KUBERNETES_SERVICE_ACCOUNT_NAME` with the name of your Kubernetes service account and the Kubernetes namespace that the account exists in.
```
"$ISSUER_HOSTPATH:sub": "system:serviceaccount:KUBERNETES_SERVICE_ACCOUNT_NAMESPACE:KUBERNETES_SERVICE_ACCOUNT_NAME"
```
The allow list example below applies `my-serviceaccount` service account to the `default` namespace and all service accounts to the `observability` namespace for the `us-west-2` region. Remember to replace `Account_ID` and `S3_BUCKET` with the required values.
1. Below is a sample trust policy of IAM role for your pods. Replace `ACCOUNT_ID`, `ISSUER_HOSTPATH`, `NAMESPACE` and `SERVICE_ACCOUNT`.
_Example: Scoped to a service account_
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$Account_ID:oidc-provider/s3.us-west-2.amazonaws.com/$S3_BUCKET"
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/ISSUER_HOSTPATH"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"s3.us-west-2.amazonaws.com/$S3_BUCKET:aud": "sts.amazonaws.com",
"s3.us-west-2.amazonaws.com/$S3_BUCKET:sub": [
"system:serviceaccount:default:my-serviceaccount",
"system:serviceaccount:amazon-cloudwatch:*"
]
}
}
"StringEquals": {
"ISSUER_HOSTPATH:sub": "system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT"
},
}
]
}
}
]
}
```

1. Create the IAM Role and make a note of the _Role ARN_.

1. Annotate your workload's ServiceAccount with the IAM role ARN:

Example ServiceAccount:
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-serviceaccount
namespace: default
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"

# optional: Defaults to "sts.amazonaws.com" if not set
eks.amazonaws.com/audience: "sts.amazonaws.com"

# optional: When set to "true", adds AWS_STS_REGIONAL_ENDPOINTS env var
# to containers
eks.amazonaws.com/sts-regional-endpoints: "true"

# optional: Defaults to 86400 for expirationSeconds if not set
# Note: This value can be overwritten if specified in the pod
# annotation as shown in the next step.
eks.amazonaws.com/token-expiration: "86400"
```

1. See the [How to use trust policies with IAM Roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/) for more information on trust policies.

### Advanced: Sharing an IAM role across multiple clusters

1. Refer [this](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) doc for different ways of configuring one or multiple service accounts through the condition operators in the trust relationship.
When multiple EKS Anywhere clusters need access to the same AWS resources (e.g., multiple ADOT deployments writing to a shared AMP workspace), you can configure a single IAM role with multiple statements—one per cluster's OIDC provider.

1. Choose Update Trust Policy to finish.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/s3.us-west-2.amazonaws.com/dev-bucket"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"s3.us-west-2.amazonaws.com/dev-bucket:aud": "sts.amazonaws.com",
"s3.us-west-2.amazonaws.com/dev-bucket:sub": "system:serviceaccount:observability:adot-collector"
}
}
},
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/s3.us-west-2.amazonaws.com/staging-bucket"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"s3.us-west-2.amazonaws.com/staging-bucket:aud": "sts.amazonaws.com",
"s3.us-west-2.amazonaws.com/staging-bucket:sub": "system:serviceaccount:observability:adot-collector"
}
}
},
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/s3.us-west-2.amazonaws.com/prod-bucket"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"s3.us-west-2.amazonaws.com/prod-bucket:aud": "sts.amazonaws.com",
"s3.us-west-2.amazonaws.com/prod-bucket:sub": "system:serviceaccount:observability:adot-collector"
}
}
}
]
}
```
Loading
Loading