v1.0 Release tracking issue

[ion-elgreco]

Description

We are close to 1.0 releases, however we need to have some minor bugs still addressed:

Binary dtype handling

• Binary col in delta.dataSkippingStatsColumns not working #2854

Z-order

• Zorder after schema evolution error - "Cannot cast file schema" #3185
• Hello, I am conducting tests with Z-order optimization and the min_commit_interval. It seems that while optimize respects the min_commit_interval, the Z-order operation does not use this setting as expected. #2979

Stats interaction

• nested fields count towards limit of stats calculation of first 32 columns #3172

Snapshot

• Snapshot references old logs still after cleanup_expired_logs_for #3057
• cleanup_expired_logs_for doesn't update logSegment after removal #3062

More async (nice-to-have)

• Use async writer + multipart + explore Datafusion sink #1984

Handle deprecations

• Remove deprecations for v1.0 Python release [breaking] #1919

Improve IO runtime

• Use object_store spawnService #3427
  Folks at arrow-rs have a new http client abstraction available in 0.12 in this PR [ObjectStore] Add SpawnService for running requests on different tokio runtime/Handle apache/arrow-rs#7253 a spawn service will be added, so all IO tasks can be properly spawned the provided runtime handle. This should improve when doing list stream which in our current DeltaStorageBackend still end up on the main runtime.

[rtyler]

I recommend using Milestones for keeping track of this 😄

Also, are you intending to make a Python 1.0 before we make the Rust version 1.0? 🤔

[ion-elgreco]

I recommend using Milestones for keeping track of this 😄

Issue is a bit easier for others to track, sind I pinned it

Also, are you intending to make a Python 1.0 before we make the Rust version 1.0? 🤔

No :) these issues also apply to the rust version actually. In the discussion of road to 1.0 we only tracked features. Which I think all of them are kind of complete except DV but I'm willing to skip that for 1.0

[roeap]

issues support sub-issues now as well :)

One major thing that I really would like to get done before 1.0 is kernel adoption.

There is much potential to break API s there, albeit not necessarily in a huge manner, but break nonetheless.

I'll prepare an issue to collect some of the work and dependencies.

Also, I am actively working on this, but its large ...

[ion-elgreco]

@roeap are there any benefits for python users? Api-wise there shouldn't be any change for python