refactor: async writer + multi-part

[ion-elgreco]

Description

Changed to use arrows async writer and use multi-part uploads.

Related Issue(s)

• closes Use async writer + multipart + explore Datafusion sink #1984

[codecov]

Codecov Report

Attention: Patch coverage is 64.89362% with 33 lines in your changes missing coverage. Please review.

Project coverage is 72.13%. Comparing base (19fb4a1) to head (e096e70).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/operations/write/writer.rs	62.00%	8 Missing and 11 partials ⚠️
crates/core/src/operations/write/async_utils.rs	68.18%	13 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3255      +/-   ##
==========================================
- Coverage   72.14%   72.13%   -0.02%     
==========================================
  Files         143      144       +1     
  Lines       45668    45748      +80     
  Branches    45668    45748      +80     
==========================================
+ Hits        32949    33002      +53     
- Misses      10634    10656      +22     
- Partials     2085     2090       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rtyler]

This is really such a major behavior change. I am not terribly familiar with the maturity level of multipart uploads in object_store I don't think this is necessarily a bad change, but I am doubtful of this addressing the originally linked issue.

As best as I can tell the buffers are still going to fill up memory until the flush, and then the flush is going to fan out to have parallel uploads

flowchart LR
    write --> buffer_batch;
    write --> buffer_batch;
    write --> buffer_batch;
    buffer_batch --> flush;
    flush --> p1;
    flush --> p2;
    flush --> p3;
    flush --> p4;
    p1 --> close;
    p2 --> close;
    p3 --> close;
    p4 --> close;

Loading

[ion-elgreco]

Good point, in its current form we indeed still buffer longer until the flush, but the buffering and writing has more parallelism now, so it should be faster.

We could do two things here btw:

• release this as is for people to experiment with (maybe after 1.0)
• iterate on this to also flush after the min - part size is available, also create some benchmark tests

[cjolowicz]

@ion-elgreco In reply to your comment on #3157 , yes I think this would likely speed up the upload phase of z-ordering. Additionally, we've been watching this with interest because it would unblock compaction to target sizes > 5GB, which is the upper limit for single part uploads in R2.

[ion-elgreco]

@cjolowicz maybe give it a spin, I don't have any environments or datasets to actually test the performance myself

[rtyler]

I remain skeptical we will see substantive performance improvements here, but willing to try in the next release 😈