Skip to content

Dev: 新增Amazon S3的AWS SDK for Java 2.x版本 #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 29 additions & 27 deletions docs/存储平台.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
| SFTP | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ❌ | ❌ | ✔️ | ❌ | ❌ | |
| WebDAV | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ❌ | ✔️ | ✔️ | ❌ | ❌ | |
| Amazon S3 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ✔️ | ✔️ | |
| Amazon S3 V2 SDK | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| MinIO | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ❌ | ✔️ | |
| 阿里云 OSS | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ✔️ | ✔️ | |
| 华为云 OBS | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ✔️ | ✔️ | |
Expand All @@ -31,33 +32,34 @@

对于兼容 Amazon S3 的存储平台,直接将配置写在 Amazon S3 中即可,具体兼容性见下图。

| 存储平台 | 官方 SDK | Amazon S3 SDK | S3 兼容说明 |
|---------------|--------|---------------|----------------------------------------------------------------------------------------------------|
| Amazon S3 | ✔️ | ✔️ | |
| MinIO | ✔️ | ✔️ | [查看](http://docs.minio.org.cn/docs/master/java-client-quickstart-guide) |
| 阿里云 OSS | ✔️ | ✔️ | [查看](https://help.aliyun.com/document_detail/64919.html#title-cds-fai-yxp) |
| 华为云 OBS | ✔️ | ✔️ | [查看](https://support.huaweicloud.com/topic/74416-1-O-obsduixiangcunchufuwus3xieyi) |
| 七牛云 Kodo | ✔️ | ✔️ | [查看](https://developer.qiniu.com/kodo/4086/amazon-s3-compatible) |
| 腾讯云 COS | ✔️ | ✔️ | [查看](https://cloud.tencent.com/document/product/436/37421) |
| 百度云 BOS | ✔️ | ✔️ | [查看](https://cloud.baidu.com/doc/BOS/s/Fjwvyq9xo) |
| 金山云 KS3 | ❌ | ✔️ | [查看](https://docs.ksyun.com/documents/959) |
| 美团云 MSS | ❌ | ✔️ | [查看](https://www.mtyun.com/doc/products/storage/mss/zhu-yao-gong-neng#兼容%20AWS%20S3%20协议) |
| 京东云 OSS | ❌ | ✔️ | [查看](https://docs.jdcloud.com/cn/object-storage-service/compatibility-api-overview) |
| 天翼云 OOS | ❌ | ✔️ | [查看](https://www.ctyun.cn/h5/help2/10000101/10001711) |
| 移动云 EOS | ❌ | ✔️ | [查看](https://ecloud.10086.cn/op-help-center/doc/article/24569) |
| 沃云 OSS | ❌ | ✔️ | [查看](https://support.woyun.cn/document.html?id=133&arcid=127) |
| 网易数帆 NOS | ❌ | ✔️ | [查看](https://www.163yun.com/help/documents/89796157866430464) |
| Ucloud US3 | ❌ | ✔️ | [查看](https://docs.ucloud.cn/ufile/s3/s3_introduction) |
| 青云 QingStor | ❌ | ✔️ | [查看](https://docs.qingcloud.com/qingstor/s3/) |
| 平安云 OBS | ❌ | ✔️ | [查看](https://yun.pingan.com/ssr/help/storage/obs/OBS_SDK_.Java_SDK_) |
| 首云 OSS | ❌ | ✔️ | [查看](http://www.capitalonline.net.cn/zh-cn/service/distribution/oss-new/#product-adv) |
| IBM COS | ❌ | ✔️ | [查看](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-compatibility-api) |
| Cloudflare R2 | ❌ | ✔️ | [查看](https://developers.cloudflare.com/r2/api/s3/api/) |
| 快快云 OSS | ❌ | ✔️ | [查看](https://www.kuaikuaicloud.com/help_center/document_detail/id/239.html) |
| 火山云 TOS | ❌ 后续支持 | ✔️ | [查看](https://www.volcengine.com/docs/6349/147050) |
| JuiceFS | ❌ | ✔️ | [查看](https://juicefs.com/docs/zh/community/s3_gateway) |
| Ceph | ❌ | ✔️ | [查看](https://docs.ceph.com/en/latest/radosgw/s3/java/) |
| 其它兼容 S3 协议的平台 | ❌ | ✔️ | |
| 存储平台 | 官方 SDK | Amazon S3 SDK | S3 兼容说明 |
|------------------|--------|---------------|----------------------------------------------------------------------------------------------------|
| Amazon S3 | ✔️ | ✔️ | |
| Amazon S3 V2 SDK | ✔️ | ✔️ | |
| MinIO | ✔️ | ✔️ | [查看](http://docs.minio.org.cn/docs/master/java-client-quickstart-guide) |
| 阿里云 OSS | ✔️ | ✔️ | [查看](https://help.aliyun.com/document_detail/64919.html#title-cds-fai-yxp) |
| 华为云 OBS | ✔️ | ✔️ | [查看](https://support.huaweicloud.com/topic/74416-1-O-obsduixiangcunchufuwus3xieyi) |
| 七牛云 Kodo | ✔️ | ✔️ | [查看](https://developer.qiniu.com/kodo/4086/amazon-s3-compatible) |
| 腾讯云 COS | ✔️ | ✔️ | [查看](https://cloud.tencent.com/document/product/436/37421) |
| 百度云 BOS | ✔️ | ✔️ | [查看](https://cloud.baidu.com/doc/BOS/s/Fjwvyq9xo) |
| 金山云 KS3 | ❌ | ✔️ | [查看](https://docs.ksyun.com/documents/959) |
| 美团云 MSS | ❌ | ✔️ | [查看](https://www.mtyun.com/doc/products/storage/mss/zhu-yao-gong-neng#兼容%20AWS%20S3%20协议) |
| 京东云 OSS | ❌ | ✔️ | [查看](https://docs.jdcloud.com/cn/object-storage-service/compatibility-api-overview) |
| 天翼云 OOS | ❌ | ✔️ | [查看](https://www.ctyun.cn/h5/help2/10000101/10001711) |
| 移动云 EOS | ❌ | ✔️ | [查看](https://ecloud.10086.cn/op-help-center/doc/article/24569) |
| 沃云 OSS | ❌ | ✔️ | [查看](https://support.woyun.cn/document.html?id=133&arcid=127) |
| 网易数帆 NOS | ❌ | ✔️ | [查看](https://www.163yun.com/help/documents/89796157866430464) |
| Ucloud US3 | ❌ | ✔️ | [查看](https://docs.ucloud.cn/ufile/s3/s3_introduction) |
| 青云 QingStor | ❌ | ✔️ | [查看](https://docs.qingcloud.com/qingstor/s3/) |
| 平安云 OBS | ❌ | ✔️ | [查看](https://yun.pingan.com/ssr/help/storage/obs/OBS_SDK_.Java_SDK_) |
| 首云 OSS | ❌ | ✔️ | [查看](http://www.capitalonline.net.cn/zh-cn/service/distribution/oss-new/#product-adv) |
| IBM COS | ❌ | ✔️ | [查看](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-compatibility-api) |
| Cloudflare R2 | ❌ | ✔️ | [查看](https://developers.cloudflare.com/r2/api/s3/api/) |
| 快快云 OSS | ❌ | ✔️ | [查看](https://www.kuaikuaicloud.com/help_center/document_detail/id/239.html) |
| 火山云 TOS | ❌ 后续支持 | ✔️ | [查看](https://www.volcengine.com/docs/6349/147050) |
| JuiceFS | ❌ | ✔️ | [查看](https://juicefs.com/docs/zh/community/s3_gateway) |
| Ceph | ❌ | ✔️ | [查看](https://docs.ceph.com/en/latest/radosgw/s3/java/) |
| 其它兼容 S3 协议的平台 | ❌ | ✔️ | |

### Alist

Expand Down
28 changes: 28 additions & 0 deletions docs/快速入门.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,17 @@
</dependency>
```


#### **Amazon S3 V2 SDK**

```xml
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.29.29</version>
</dependency>
```

#### **FTP**

```xml
Expand Down Expand Up @@ -396,6 +407,23 @@ amazon-s3: # 0.0.7 及以前的版本,配置名称是:aws-s3

更多参数请参考 `org.dromara.x.file.storage.spring.SpringFileStorageProperties.SpringAmazonS3Config`

#### **Amazon S3 V2 SDK**

```yaml
amazon-s3-v2: # Amazon S3 SDK V2
- platform: amazon-s3-v2-1 # 存储平台标识
enable-storage: true # 启用存储
access-key: ??
secret-key: ??
region: ?? # 可选
end-point: ?? # 必传
bucket-name: ??
domain: ?? # 访问域名,注意“/”结尾,例如:https://abc.hn-bkt.clouddn.com/
base-path: s3/ # 基础路径
```

更多参数请参考 `org.dromara.x.file.storage.spring.SpringFileStorageProperties.SpringAmazonS3Config`

#### **FTP**

```yaml
Expand Down
9 changes: 8 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@
<!--region 3rd SDK-->
<aliyun-sdk-oss.version>3.16.1</aliyun-sdk-oss.version>
<aws-java-sdk-s3.version>1.12.429</aws-java-sdk-s3.version>
<aws-java-sdk-s3-v2.version>2.29.29</aws-java-sdk-s3-v2.version>
<bce-java-sdk.version>0.10.251</bce-java-sdk.version>
<commons-net.version>3.9.0</commons-net.version>
<commons-pool2.version>2.11.1</commons-pool2.version>
Expand Down Expand Up @@ -216,7 +217,13 @@
<artifactId>esdk-obs-java</artifactId>
<version>${esdk-obs-java.version}</version>
</dependency>
<!-- Amazon S3 -->
<!-- Amazon S3 v2 -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>${aws-java-sdk-s3-v2.version}</version>
</dependency>
<!-- Amazon S3 v1 -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
Expand Down
22 changes: 21 additions & 1 deletion x-file-storage-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,14 @@
<scope>provided</scope>
<optional>true</optional>
</dependency>
<!-- Amazon S3 -->
<!-- Amazon S3 V2 -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<!-- Amazon S3 V1 -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
Expand Down Expand Up @@ -197,5 +204,18 @@
<optional>true</optional>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>21</source>
<target>21</target>
<compilerArgs>--enable-preview</compilerArgs>
</configuration>
</plugin>
</plugins>
</build>

</project>
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,11 @@ public class FileStorageProperties {
*/
private List<? extends AmazonS3Config> amazonS3 = new ArrayList<>();

/**
* Amazon S3 V2
*/
private List<? extends AmazonS3V2Config> amazonS3V2 = new ArrayList<>();

/**
* FTP
*/
Expand Down Expand Up @@ -557,6 +562,55 @@ public static class AmazonS3Config extends BaseConfig {
private Map<String, Object> attr = new LinkedHashMap<>();
}

/**
* Amazon S3 V2
*/
@Data
@Accessors(chain = true)
@EqualsAndHashCode(callSuper = true)
public static class AmazonS3V2Config extends BaseConfig {

private String accessKey;

private String secretKey;

private String region;

private String endPoint;

private String bucketName;

/**
* 访问域名
*/
private String domain = "";

/**
* 基础路径
*/
private String basePath = "";

/**
* 默认的 ACL,详情 {@link Constant.AwsS3ACL}
*/
private String defaultAcl;

/**
* 自动分片上传阈值,达到此大小则使用分片上传,默认 128MB
*/
private int multipartThreshold = 128 * 1024 * 1024;

/**
* 自动分片上传时每个分片大小,默认 32MB
*/
private int multipartPartSize = 32 * 1024 * 1024;

/**
* 其它自定义配置
*/
private Map<String, Object> attr = new LinkedHashMap<>();
}

/**
* FTP
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.dromara.x.file.storage.core.tika.TikaContentTypeDetect;
import org.dromara.x.file.storage.core.tika.TikaFactory;
import org.dromara.x.file.storage.core.util.Tools;
import software.amazon.awssdk.services.s3.S3Client;

@Slf4j
@Getter
Expand Down Expand Up @@ -241,6 +242,7 @@ public FileStorageService build() {
fileStorageList.addAll(buildUpyunUssFileStorage(properties.getUpyunUss(), clientFactoryList));
fileStorageList.addAll(buildMinioFileStorage(properties.getMinio(), clientFactoryList));
fileStorageList.addAll(buildAmazonS3FileStorage(properties.getAmazonS3(), clientFactoryList));
fileStorageList.addAll(buildAmazonS3V2FileStorage(properties.getAmazonS3V2(), clientFactoryList));
fileStorageList.addAll(buildFtpFileStorage(properties.getFtp(), clientFactoryList));
fileStorageList.addAll(buildSftpFileStorage(properties.getSftp(), clientFactoryList));
fileStorageList.addAll(buildWebDavFileStorage(properties.getWebdav(), clientFactoryList));
Expand Down Expand Up @@ -447,6 +449,25 @@ public static List<AmazonS3FileStorage> buildAmazonS3FileStorage(
.collect(Collectors.toList());
}

/**
* 根据配置文件创建 Amazon S3 存储平台,使用v2SDK
*/
public static List<AmazonS3V2FileStorage> buildAmazonS3V2FileStorage(
List<? extends AmazonS3V2Config> list, List<List<FileStorageClientFactory<?>>> clientFactoryList) {
if (CollUtil.isEmpty(list)) return Collections.emptyList();
buildFileStorageDetect(list, "Amazon S3 v2", "software.amazon.awssdk.services.s3.S3Client");
return list.stream()
.map(config -> {
log.info("加载 Amazon S3 v2 存储平台:{}", config.getPlatform());
FileStorageClientFactory<S3Client> clientFactory = getFactory(
config.getPlatform(),
clientFactoryList,
() -> new AmazonS3V2FileStorageClientFactory(config));
return new AmazonS3V2FileStorage(config, clientFactory);
})
.collect(Collectors.toList());
}

/**
* 根据配置文件创建 FTP 存储平台
*/
Expand Down
Loading