Skip to content

S3 getInputStream is not closed leading to lack of connection pool resource #2365

@cklee1967

Description

@cklee1967

Describe the bug
I am creating an adaptor to interface a CMIS service to store content in an S3 store (with folder data in a relational database).
The problem I encountered is when I am reading content from the S3, i.e. Resource.getInputStream.
In the AWS S3 Client implementation, there is a HttpClient connection pool (default is 50 connections).

When I start to download multiple content from the S3 store via the CMIS interface, it will cause an Exception: timeout in getting connection from pool.

Searching the web I found that it could be caused by the InputStream not closed by the CMIS interface.
Please refer to: aws/aws-sdk-java#1405

To Reproduce
Need to to link the CMIS interface to the S3 store.
Create a SpringBootApplication with the following and import the spring-content-cmis, spring-content-s3, spring-content-jpa packages.

@EnableCmis
@import(JpaLockingAndVersioningConfig.class)
@EnableJpaRepositories
@EnableS3Stores

Define a S3 client bean:
@bean
public S3Client client() throws URISyntaxException {
AwsCredentialsProvider provider = new AwsCredentialsProvider() {
@OverRide
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(s3accessKey, s3secretKey);
}
};

    return S3Client.builder()
            .endpointOverride(new URI(s3Url))
            .region(Region.AP_SOUTH_1)
            .credentialsProvider(provider)
            .build();
}

Using the CMIS interface download some CMIS file multiple times (at least 50 which is the default max connections).

Expected behavior
No limit on the number of file or content downloaded.

Additional context
Please refer to: aws/aws-sdk-java#1405

If I change the getInputStream() method in the SimpleStorageResource.java, to load the entire content from input stream this.amazonS3.getObject(getObjectRequestBuilder.build()) into a Byte Array, then close the InputStream and finally, return an InputSteam based on the byte array, it will not encounter the timeout on connection pool exception.

What is the proper way to handle the contentStream for CMIS or S3 object inputStream which needs to be closed quickly, to avoid a connection pool issue ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions