Handling large file uploads - stream from file instead of binary

[bytesofpie]

I need to allows users to upload roughly ~150mb files to S3, not very often, but since waffle_ecto is using the binary upload, it loads the entire file into memory. I'd rather not pay for more memory when Waffle has support for streaming a file to S3 if the file is from disk and not a binary.

Is there a way I can configure waffle_ecto to stream to local disk, then do a stream upload to S3? That way I don't have 150mb loaded into memory for every concurrent upload. Using gigalixir I'd essentially have to pay for $10 per concurrent upload I want to support without crashing due to running out of memory.

[achempion]

Hi, https://github.com/elixir-waffle/waffle/blob/master/lib/waffle/storage/s3.ex#L184 Waffle already supports streaming for S3 compatible storage.

waffle_ecto is using the binary upload

You can't really escape memory file loading for binary type as it expects you to pass the whole file as an argument but we do support different type of uploads, such as `Plug.Upload`.

Is there a way I can configure waffle_ecto to stream to local disk, then do a stream upload to S3

What do you mean by streaming to local disk? Do you have some downloading stream and you would like to pass it directly to S3? Could you provide more information for your use-case. Thank you.

[bytesofpie]

Am I correct in thinking there are 2 upload portions? The user first uploads the file when they submit the multipart form to my phoenix app. Then waffle uploads it to S3 and then once it has a path, saves it into Ecto. My app is running out of memory near the end of the first user upload portion. I guess I just have to increase the memory of the server, no way to avoid receiving all the bytes at once.

[achempion]

If you're using standard multi-part uploads then you will get your file as a temporary path and it would be streamed into the S3 storage. I shouldn't be read into the memory at any stage.

Then waffle uploads it to S3 and then once it has a path, saves it into Ecto

Not quite, we knew the path in advance (you define it via uploader configuration) and then upload a file to the known path.