Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,92 @@ PROPERTIES
);
```

### AWS EKS集群中Iam Role认证鉴权

对于在 Amazon EKS 集群中运行的应用(例如 Apache Doris),要授予其 AWS Identity and Access Management(IAM)权限,Amazon EKS 提供了以下两种主要方式:

**1. 服务账户的 IAM 角色 (IRSA)**

**2. EKS 容器组身份 (Pod Identify)**

这两种方式均需在 EKS 集群中正确配置IAM Role和对应的信任策略、IAM策略, 具体配置方法请参阅AWS官方文档:

[Granting AWS Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters](https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#service-accounts-iam)

Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证

### Bucket Policy 认证鉴权

对于IAM Role部署的Doris机器,导入、导出、TVF的场景也支持使用 Amazon S3 存储桶策略来保护对AWS S3存储桶中的对象进行访问,这样,
这样可以限制只有EC2机器所属用户才能访问对象存储桶,具体步骤如下:

1、设置目标存储桶的Bucket Policy

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:root"
]
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:root"
]
},
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::<bucket>",
}
]
}
```

配置好Bucket Policy 之后Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证,
请将`arn:aws:iam::111122223333:root` 替换为ec2机器所绑定的账户或者Role的ARN

2、使用对应功能的SQL语法进行数据访问,不需要ak/sk,arn等信息

```sql
SELECT * FROM S3 (
"uri" = "s3://your_bucket/path/to/tvf_test/test.parquet",
"format" = "parquet",
"s3.endpoint" = "s3.us-east-1.amazonaws.com",
"s3.region" = "us-east-1"
)
```

Doris FE/BE支持以`AWSCredentialsProviderChain`的方式自动探测获取凭证

参考文档:[Bucket Policy](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html)

### 鉴权方式最佳实践
| 鉴权方式 | 适用场景 | 优 点 | 缺 点 |
| :-------------------------------------------- | :----------------------------------------- | ----------------------- | -------- |
| [AK/SK](./import-way/stream-load-manual) | 私有化部署安全性可控或非AWS S3的对象存储的导入/导出/StorageVault场景 | 配置简单,支持兼容AWS S3的对象存储 | 存在密钥泄漏风险,需要手动进行密钥轮换 |
| [IAM ROLE](./import-way/broker-load-manual.md) | AWS S3公有云安全性要求较高的导入/导出/StorageVault场景 | 安全性高,自动轮换AWS凭证, 权限配置集中| 配置Bucket Policy/Trust Policy流程复杂 |
| [Bucket Policy](./import-way/insert-into-manual.md) | AWS S3公有云,bucket数量较少的导入/导出/StorageVault场景 | 配置流程复杂度适中,遵循最小权限原则,自动探测AWS凭证 | 权限配置分散在各个bucket policy中 |


### FAQ

#### 1. 如何设置`BE`和`Recycler`的Aws Sdk DEBUG级别日志?
Expand Down