[RFC]: Consolidate DeltaTable and DeltaOps APIs

[roeap]

While migrating our internals to using delta-kernel-rs, we saw that there might be some room for improvement in how we structure our main structs/apis. Certain feature implementations we struggle with IMO steam from the fact, that there is some conceptual misalignment.

Leaving aside the two separate snapshots for now, I believe DeltaTable and DeltaOps could or even should be consolidated into DeltaTable. The main reason being that we have gotten into a place where we practically are using sessions without acknowledging it (at least not always). In the abstract form the session is responsible for resources required to execute a query. For datafusion based operations we create df sessions ad-hoc, while in other cases LogStore is effectively the session.

DeltaOps just wraps a pub DeltaTable to create the builders. When consolidated this would reduce the nimber of exposed stats and hopefully improve discoverability for operations. We can then also more flexibly handle session management - i.e. allowing users to provide a datafusion session we can use to execute operations.

We may even one day want to expose dataframe-like APIs on the delta table.

SO in short in believe this would simplify our APIs as well as allow for easier future optimizations.

[rtyler]

I discussed this idea at some length with @roeap in our weekly Delta Lake Live stream. I think this is a good idea, and would make our operations a lot more visible and consistent.

@roeap compared the DeltaTable to a DataFrame like API which I quite like the more I think about it. For example, if we make DeltaTable never perform internal mutations, but instead the operations are always returning a new DeltaTable with everything moved over, except the new dt would have a new inner Snapshot representing the new table state. For example:

let mut dt = open_table(&some_url).await?;

dt = dt.optimize().with_superpowers().etc().await?;
dt = dt.vacuum().await?;

I think the more we want to push with Snapshot being the source of truth, DeltaTable should become more functional as a wrapper for operations that carries an internal snapshot with it. For user clarity, I think it's important to receive a "new" DeltaTable whenever state is modified (e.g. a new commit has been made, followed by a load()), that way we're not blurring the lines between what each of these things is

[ion-elgreco]

I like this a lot being able to chain ops!