Skip to content

Remove multipart upload code #98

@bbockelm

Description

@bbockelm

We've been getting reported issues from users uploading large files to an origin with a S3 backend. It appears the CompleteMultipartUpload request takes long enough to complete when we invoke Close() that the client talking to the origin times out.

Indeed, from AWS's documentation:

The processing of a CompleteMultipartUpload request could take several minutes to finalize. After Amazon S3 begins processing the request, it sends an HTTP response header that specifies a 200 OK response. While processing is in progress, Amazon S3 periodically sends white space characters to keep the connection from timing out. A request could fail after the initial 200 OK response has been sent.

The Pelican client has a 10s timeout by default which is fairly incompatible with a "several minute" delay.

Luckily, this is mostly not necessary. XRootD now informs the OSS layer of the size of the object that will be uploaded; in that case, we could just use pure streaming with a single S3 interaction. We don't do this currently to avoid having too many variant code paths; however, it now seems like a necessity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions