Skip to content

S3Store#put uses unbounded memory when Path.size is None #34

@Daenyth

Description

@Daenyth

S3Store invokes TransferManager#upload, which says

     * When uploading options from a stream, callers <b>must</b> supply the size of
     * options in the stream through the content length field in the
     * <code>ObjectMetadata</code> parameter.
     * If no content length is specified for the input
     * stream, then TransferManager will attempt to buffer all the stream
     * contents in memory and upload the options as a traditional, single part
     * upload. Because the entire stream contents must be buffered in memory,
     * this can be very expensive, and should be avoided whenever possible.

When you use too much, you get a java.lang.OutOfMemoryError: Java heap space

It is possible to do multipart uploads to use fixed memory; for reference, this is what Alpakka does for akka-streams:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions