Concurrent write support

[adriangb]

I'm trying to understand what the story is with DeltaLake and concurrent writes. Based on #2279 it sounds like there's some active work in the area for delta-rs specifically and it's known that I'd encounter version mismatch conflicts as of the current version of delta-rs.

But more generally, am I right in assuming that since there's a transaction log that needs to be kept in serialized order there will overhead with concurrent writers so performance will never be able to match that of just writing out files to blob storage? My data is append-only.

Thanks!

[ion-elgreco]

@adriangb on ADLS, GCS and S3 with locking client there is concurrent write support. When two writers try to write the same commit it will try to resolve the conflict.

Currently, there is indeed an issue where it tries to advance the state with missing log segments, this causes a version mismatch but the commit is still successful. It just means, that the state was not able to get updated. For now you could catch this errors and just continue.

[adriangb]

Thanks for clarifying. If I have say 100 concurrent writers, can't there be dozens of failures before it can commit? And if each one of those failures has a 20-30ms latency (object stores) that would add up.

[ion-elgreco]

It might be possible

[adriangb]

Ok thank you for confirming 👍🏻

[adriangb]

It might be possible

I've thought about this a bit more, isn't it more than possible? I'd think it's almost guaranteed that if you have many concurrent writers you're going to have contention.

it's only retrying the commit so it's not really costly

In a streaming scenario this is actually quite costly: I've measured latency in GCS for a copy if not exists to be ~150ms (not network latency, e.g. a PUT is ~30ms from my region). So if that gets retried say 5 times before it succeeds we're talking 1s of latency from retries.

Is there anything I can do to mitigate this if this is an append-only table?

[ion-elgreco]

@adriangb delta relies on optimistic concurrency..

You could wait until we have refactored our api calls to use a Conditional put, #2296. This should reduce the latency since there are less requests.

Also you can always add an outer_retry in case you have a lot of contention

[SiarheiMatsevich]

@ion-elgreco could you please explain what values are expected for backoff in storage_options?
I added such values into storage options but have not noticed any change. It's not clear from the documentation page what values should be used here, I use strings, should I use integers or floats instead?

az_options["backoff_config.init_backoff"] = "500ms"
az_options["backoff_config.max_backoff"] = "1m"
az_options["backoff_config.base"] = "2.0"

[ion-elgreco]

Those are correctly configured