Fix aws_s3_stream EntityTooSmall error for small files - #893
Conversation
The lines archive format was not adding a trailing newline after the last message in a batch. When multiple batches are written to the same output stream (e.g., with aws_s3_stream and batching enabled), the batches would concatenate without proper separation, causing the last line of one batch to merge with the first line of the next batch. This resulted in "message contains multiple valid documents" errors when parsing JSONL files, as two JSON objects would appear on a single line. The fix adds a trailing newline to ensure each batch is properly terminated and can be safely concatenated with subsequent batches.
Prevents EntityTooSmall errors by ensuring timer-based flushes never upload parts < 5MB. S3 requires all non-final parts to be >= 5MB; only the final part (uploaded during Close) can be smaller.
This reverts commit 9c3cd69.
|
@triddell this still doesn't actually guarantee that messages get delivered, since it's not actually acking part uploads, doesn't support resuming in-progress uploads, and aborts in-progress uploads for messages that have already been acked. please take a look at my PR #895, which solves your <5MB problem and also the other aforementioned issues, among other things. |
|
@maxtheaxe hey, I'll take a look and do some testing in my environment with it. those are great additions. have you been using this output? |
|
@triddell am trying to—I'm thinking in addition to the things I added here, I probably also need an |
|
@maxtheaxe I spent some time testing #895 against a real S3 bucket as promised — One thing did turn up though: against real S3, #895 deadlocks on shutdown for So the small-file path still needs a fix before #895 fully supersedes this one. |
|
Flagging a real-world dependency on this one: we run bento embedded in AWS Lambda with bounded inputs |
|
@triddell does usage of read_until in the way I described in the last comment on my PR solve your use case? if so, here's a docker build of the latest there (with the <5MB fix) |
Why this fix is needed — the
|
|
@maxtheaxe Just made the comment above here and then another on your PR. It seems we can't do both of these approaches, what's needed for a CLI execution and what you are looking at for a streaming version, at the same time without a flag changing the behavior. Would like to hear your feedback for sure. I run Bento in lambdas as a CLI app, actually parallel versions of Bento within a lambda often times. I definitely need the CLI execution to not have to wait for a timeout and I think that would be everyone's expectation. When the input is done, write the file(s), stop the Bento execution. |
|
@triddell I do think we can do both of these, but I can't spend anymore time on it today. will get back to you soon. |
c6327b1 to
92769be
Compare
|
Follow-on change on this branch: an opt-in Doing gzip via a
|
2 participants
Description
Fixes the
EntityTooSmallerror that occurs whenaws_s3_streamtries to upload files smaller than 5 MiB using multipart upload.S3 requires multipart upload parts (except the final part) to be >= 5 MiB. This fix automatically detects when files are too small for multipart and falls back to S3's simple
PutObjectAPI instead.Changes
Close()logic: Check if any parts have been uploaded before completing multipart uploadPutObjectfor files < 5 MiBPutObjectTest Coverage
Added comprehensive unit tests:
All existing tests continue to pass.
Breaking Changes
None. The change is transparent to users - the API and configuration remain the same. Files are uploaded successfully regardless of size.
Checklist