feat: add support for blind appends

[abhi-airspace-intelligence]

Description

Consider the following case:

1. I kick off a compaction of a delta table
2. Simultaneously, I start a write
3. The compaction finishes, logically removing/adding some files
4. I go to commit my write

This will fail, since by default, the conflict checker assumes that the write depended on all the files in the table at a given state, this with ConcurrentDeleteRead. However, if it is a true blind append, the writer never depended on the state of the table.

Instead, this PR marks a write commit as a blind append if it is in mode Append and there are no remove actions.

Marked as draft bc I'd like to write tests and confirm this is the correct behavior.

Related Issue(s)

Likely closes #2700

Documentation

Reference here: https://books.japila.pl/delta-lake-internals/OptimisticTransactionImpl/?h=blind+app#isBlindAppend
Planning on diving into the delta-spark code when I have the time to ensure this is the correct heuristic.

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[rtyler]

👋 Thanks for starting the PR in draft to get the discussion going!

Blind appends are ... tricky 😄

For your specific situation I am curious if setting the appendOnly table property would suffice:
append
only
If that table property doesn't solve the issue you describe, then I would say
we have a bug in our commit code that should be rectified, because on an
appendOnly table there should be no problem writing.

There's not really anything in the protocol that states what the rules should
be for a blind append. In the scenario that you describe it is probably safe
enough to issue an append but i'm not sure I would call that a blind append.

When I think of a blind append, I think about something that doesn't
care/consider whether the schema has been changed underneath either, i.e. isn't
considering if a metadata action happened between its current state and
whatever $latest_version is.

[abhiaagarwal]

@rtyler the offending code is here:

https://github.com/delta-io/delta-rs/blob/main/crates/core/src/kernel/transaction/conflict_checker.rs#L209-L212

Yes, my tables are already marked as append only, but I still run into the conflict bug. Where if there is no read predicate specified, then all the files that were read in the snapshot are returned. However, in the case of blind appends, while the table state was read, it doesn't have any actual impact on what was written. Additionally, even if a table is not AppendOnly by property, I'd still say that marking commits as blind appends if they are is a decent practice.

isBlindAppend isn't specified in the protocol, but delta-spark has special handling for it based on my look at the codebase.

[abhiaagarwal]

For what it's worth, this PR has solved my problem entirely of being able to run concurrent appends + compactions as a background tokio task without any sort of concurrency issues.

[abhi-airspace-intelligence]

I opened an issue in the delta-spark repository