Add an option to buffer based on S3 minimum chunk size

[mdedetrich]

What is currently missing?

When gzip compression was added to Guardian (see #196) due to how gzip compression works the implementation was not ideal. Specifically we compress each Kafka record individually where as ideally we would like to compress larger chunks of data since that will give much better space savings via compression.

How could this be improved?

This feature could be improved by buffering the data in memory up until the S3 minimum chunk size (5 megs) and then compressing that entire memory chunk at once rather than doing it per message.

Is this a feature you would work on yourself?

• I plan to open a pull request for this feature