diff --git a/docs/admin-manual/auth/integrations/aws-authentication-and-authorization.md b/docs/admin-manual/auth/integrations/aws-authentication-and-authorization.md index d453714735292..250f89f0e69a7 100644 --- a/docs/admin-manual/auth/integrations/aws-authentication-and-authorization.md +++ b/docs/admin-manual/auth/integrations/aws-authentication-and-authorization.md @@ -489,6 +489,89 @@ PROPERTIES ); ``` +### AWS EKS Cluster IAM Role Authentication and Authorization + +For applications (such as Apache Doris) running in an Amazon EKS cluster that need to be granted AWS Identity and Access Management (IAM) permissions, Amazon EKS provides the following two primary methods: + +**1.IAM Roles for Service Accounts (IRSA)​** + +**2. EKS Pod Identity​** + +Both methods require correct configuration of the IAM Role, corresponding trust policy, and IAM policy in the EKS cluster. For specific configuration methods, please refer to the AWS official documentation: + +[Granting AWS Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters](https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-accounts-iam) + +Doris FE/BE supports automatically detecting and obtaining credentials via the `WSCredentialsProviderChain` method. + +### Bucket Policy Authentication and Authorization + +For Doris machines deployed using IAM Roles, import, export, and TVF scenarios also support using Amazon S3 bucket policies to control access to objects in AWS S3 buckets. This allows restricting access to the object bucket only to users associated with the EC2 machine. The specific steps are as follows: + +1、Set the Bucket Policy for the target bucket. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectVersion", + "s3:DeleteObject", + "s3:DeleteObjectVersion", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3::://*" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::", + } + ] +} +``` + +After configuring the Bucket Policy, Doris FE/BE supports automatically detecting and obtaining credentials via the AWSCredentialsProviderChainmethod. Please replace arn:aws:iam::111122223333:rootwith the ARN of the account or Role bound to the EC2 machine. + +2、Use the corresponding SQL syntax for data access. No AK/SK, ARN, or other information is required. + +```sql + SELECT * FROM S3 ( + "uri" = "s3://your_bucket/path/to/tvf_test/test.parquet", + "format" = "parquet", + "s3.endpoint" = "s3.us-east-1.amazonaws.com", + "s3.region" = "us-east-1" + ) +``` + +Doris FE/BE supports automatically detecting and obtaining credentials via the `AWSCredentialsProviderChain` method. + +Reference documentation: [Bucket Policy](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html) + +### Best Practices for Authentication Methods +| Authentication Method | Applicable Scenarios | Advantages | Disadvantages | +| :-------------------------------------------- | :----------------------------------------- | ----------------------- | -------- | +| AK/SK Authentication | Import/Export/StorageVault scenarios with privately deployed, security-controlled storage or non-AWS S3 object storage. | Simple configuration, supports object storage compatible with AWS S3. | Risk of secret key leakage; manual key rotation required. | +| IAM ROLE Authentication | Import/Export/StorageVault scenarios on AWS S3 public cloud with high-security requirements. | High security, automatic AWS credential rotation, centralized permission configuration. | Complex Bucket Policy/Trust configuration process. | +| Bucket Policy Authentication | Import/Export/StorageVault scenarios on AWS S3 public cloud with a small number of buckets | Moderate configuration complexity, adheres to the principle of least privilege, automatically detects AWS credentials. | Permission configuration is scattered across various bucket policies. | + ### FAQ #### 1. How to set AWS SDK DEBUG level logs for BE and Recycler? diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/integrations/aws-authentication-and-authorization.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/integrations/aws-authentication-and-authorization.md index b5781b886323d..dcb6e546a1d98 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/integrations/aws-authentication-and-authorization.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/auth/integrations/aws-authentication-and-authorization.md @@ -508,6 +508,91 @@ PROPERTIES ); ``` +### AWS EKS集群中Iam Role认证鉴权 + +对于在 Amazon EKS 集群中运行的应用（例如 Apache Doris），要授予其 AWS Identity and Access Management（IAM）权限，Amazon EKS 提供了以下两种主要方式： + +**1. 服务账户的 IAM 角色 （IRSA）** + +**2. EKS 容器组身份 (Pod Identify)** + +这两种方式均需在 EKS 集群中正确配置IAM Role和对应的信任策略、IAM策略, 具体配置方法请参阅AWS官方文档： + +[Granting AWS Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters](https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-accounts-iam) + +Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证 + +### Bucket Policy 认证鉴权 + +对于IAM Role部署的Doris机器，导入、导出、TVF的场景也支持使用 Amazon S3 存储桶策略来保护对AWS S3存储桶中的对象进行访问，这样， +这样可以限制只有EC2机器所属用户才能访问对象存储桶，具体步骤如下： + +1、设置目标存储桶的Bucket Policy + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectVersion", + "s3:DeleteObject", + "s3:DeleteObjectVersion", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3::://*" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::", + } + ] +} +``` + +配置好Bucket Policy 之后Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证, +请将`arn:aws:iam::111122223333:root` 替换为ec2机器所绑定的账户或者Role的ARN + +2、使用对应功能的SQL语法进行数据访问，不需要ak/sk，arn等信息 + +```sql + SELECT * FROM S3 ( + "uri" = "s3://your_bucket/path/to/tvf_test/test.parquet", + "format" = "parquet", + "s3.endpoint" = "s3.us-east-1.amazonaws.com", + "s3.region" = "us-east-1" + ) +``` + +Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证 + +参考文档：[Bucket Policy](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html) + +### 鉴权方式最佳实践 +| 鉴权方式 | 适用场景 | 优 点 | 缺 点 | +| :-------------------------------------------- | :----------------------------------------- | ----------------------- | -------- | +| AK/SK 鉴权方式 | 私有化部署安全性可控或非AWS S3的对象存储的导入/导出/StorageVault场景 | 配置简单，支持兼容AWS S3的对象存储 | 存在密钥泄漏风险，需要手动进行密钥轮换 | +| IAM ROLE 鉴权方式 | AWS S3公有云安全性要求较高的导入/导出/StorageVault场景 | 安全性高,自动轮换AWS凭证, 权限配置集中| 配置Bucket Policy/Trust流程复杂 | +| Bucket Policy 鉴权方式 | AWS S3公有云，bucket数量较少的导入/导出/StorageVault场景 | 配置流程复杂度适中，遵循最小权限原则，自动探测AWS凭证 | 权限配置分散在各个bucket policy中 | + ### FAQ #### 1. 如何设置`BE`和`Recycler`的Aws Sdk DEBUG级别日志? diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md index b5781b886323d..dcb6e546a1d98 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md @@ -508,6 +508,91 @@ PROPERTIES ); ``` +### AWS EKS集群中Iam Role认证鉴权 + +对于在 Amazon EKS 集群中运行的应用（例如 Apache Doris），要授予其 AWS Identity and Access Management（IAM）权限，Amazon EKS 提供了以下两种主要方式： + +**1. 服务账户的 IAM 角色 （IRSA）** + +**2. EKS 容器组身份 (Pod Identify)** + +这两种方式均需在 EKS 集群中正确配置IAM Role和对应的信任策略、IAM策略, 具体配置方法请参阅AWS官方文档： + +[Granting AWS Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters](https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-accounts-iam) + +Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证 + +### Bucket Policy 认证鉴权 + +对于IAM Role部署的Doris机器，导入、导出、TVF的场景也支持使用 Amazon S3 存储桶策略来保护对AWS S3存储桶中的对象进行访问，这样， +这样可以限制只有EC2机器所属用户才能访问对象存储桶，具体步骤如下： + +1、设置目标存储桶的Bucket Policy + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectVersion", + "s3:DeleteObject", + "s3:DeleteObjectVersion", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3::://*" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::", + } + ] +} +``` + +配置好Bucket Policy 之后Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证, +请将`arn:aws:iam::111122223333:root` 替换为ec2机器所绑定的账户或者Role的ARN + +2、使用对应功能的SQL语法进行数据访问，不需要ak/sk，arn等信息 + +```sql + SELECT * FROM S3 ( + "uri" = "s3://your_bucket/path/to/tvf_test/test.parquet", + "format" = "parquet", + "s3.endpoint" = "s3.us-east-1.amazonaws.com", + "s3.region" = "us-east-1" + ) +``` + +Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证 + +参考文档：[Bucket Policy](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html) + +### 鉴权方式最佳实践 +| 鉴权方式 | 适用场景 | 优 点 | 缺 点 | +| :-------------------------------------------- | :----------------------------------------- | ----------------------- | -------- | +| AK/SK 鉴权方式 | 私有化部署安全性可控或非AWS S3的对象存储的导入/导出/StorageVault场景 | 配置简单，支持兼容AWS S3的对象存储 | 存在密钥泄漏风险，需要手动进行密钥轮换 | +| IAM ROLE 鉴权方式 | AWS S3公有云安全性要求较高的导入/导出/StorageVault场景 | 安全性高,自动轮换AWS凭证, 权限配置集中| 配置Bucket Policy/Trust流程复杂 | +| Bucket Policy 鉴权方式 | AWS S3公有云，bucket数量较少的导入/导出/StorageVault场景 | 配置流程复杂度适中，遵循最小权限原则，自动探测AWS凭证 | 权限配置分散在各个bucket policy中 | + ### FAQ #### 1. 如何设置`BE`和`Recycler`的Aws Sdk DEBUG级别日志? diff --git a/versioned_docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md b/versioned_docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md index e567469a1439b..eeddb76669763 100644 --- a/versioned_docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md +++ b/versioned_docs/version-3.0/admin-manual/auth/integrations/aws-authentication-and-authorization.md @@ -489,6 +489,89 @@ PROPERTIES ); ``` +### AWS EKS Cluster IAM Role Authentication and Authorization + +For applications (such as Apache Doris) running in an Amazon EKS cluster that need to be granted AWS Identity and Access Management (IAM) permissions, Amazon EKS provides the following two primary methods: + +**1.IAM Roles for Service Accounts (IRSA)​** + +**2. EKS Pod Identity​** + +Both methods require correct configuration of the IAM Role, corresponding trust policy, and IAM policy in the EKS cluster. For specific configuration methods, please refer to the AWS official documentation: + +[Granting AWS Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters](https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-accounts-iam) + +Doris FE/BE supports automatically detecting and obtaining credentials via the `WSCredentialsProviderChain` method. + +### Bucket Policy Authentication and Authorization + +For Doris machines deployed using IAM Roles, import, export, and TVF scenarios also support using Amazon S3 bucket policies to control access to objects in AWS S3 buckets. This allows restricting access to the object bucket only to users associated with the EC2 machine. The specific steps are as follows: + +1、Set the Bucket Policy for the target bucket. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:GetObjectVersion", + "s3:DeleteObject", + "s3:DeleteObjectVersion", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource": "arn:aws:s3::://*" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111122223333:root" + ] + }, + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::", + } + ] +} +``` + +After configuring the Bucket Policy, Doris FE/BE supports automatically detecting and obtaining credentials via the AWSCredentialsProviderChainmethod. Please replace arn:aws:iam::111122223333:rootwith the ARN of the account or Role bound to the EC2 machine. + +2、Use the corresponding SQL syntax for data access. No AK/SK, ARN, or other information is required. + +```sql + SELECT * FROM S3 ( + "uri" = "s3://your_bucket/path/to/tvf_test/test.parquet", + "format" = "parquet", + "s3.endpoint" = "s3.us-east-1.amazonaws.com", + "s3.region" = "us-east-1" + ) +``` + +Doris FE/BE supports automatically detecting and obtaining credentials via the `AWSCredentialsProviderChain` method. + +Reference documentation: [Bucket Policy](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html) + +### Best Practices for Authentication Methods +| Authentication Method | Applicable Scenarios | Advantages | Disadvantages | +| :-------------------------------------------- | :----------------------------------------- | ----------------------- | -------- | +| AK/SK Authentication | Import/Export/StorageVault scenarios with privately deployed, security-controlled storage or non-AWS S3 object storage. | Simple configuration, supports object storage compatible with AWS S3. | Risk of secret key leakage; manual key rotation required. | +| IAM ROLE Authentication | Import/Export/StorageVault scenarios on AWS S3 public cloud with high-security requirements. | High security, automatic AWS credential rotation, centralized permission configuration. | Complex Bucket Policy/Trust configuration process. | +| Bucket Policy Authentication | Import/Export/StorageVault scenarios on AWS S3 public cloud with a small number of buckets | Moderate configuration complexity, adheres to the principle of least privilege, automatically detects AWS credentials. | Permission configuration is scattered across various bucket policies. | + ### FAQ #### 1. How to set AWS SDK DEBUG level logs for BE and Recycler?