Kernel error when vacuuming table

[adamreeve]

Environment

Delta-rs version:

Built from current main branch (commit d148c32b57a5a9f5623cb18bdef16a413ae4b02e) by running make develop in the python directory.

I cannot reproduce this issue with the previous release version (1.1.4)

Binding:

Environment:

• Cloud provider: NA
• OS: Fedora Linux 42
• Other:

---

Bug

What happened:

Error when calling Table.vacuum:

Traceback (most recent call last):
  File "/home/adam/dev/delta-lake/repro_crash.py", line 24, in <module>
    table.vacuum()
  File "/home/adam/dev/delta-lake/delta-rs/python/deltalake/table.py", line 559, in vacuum
    return self._table.vacuum(
           ^^^^^^^^^^^^^^^^^^^
_internal.DeltaError: Generic error: Kernel error: Generic delta kernel error: Requested field not found in parquet schema, and field is not nullable: type

What you expected to happen:

No crash.

How to reproduce it:

from deltalake import write_deltalake, DeltaTable
import pyarrow as pa


delta_dir = "./table"

data = pa.Table.from_pydict({
    'x': pa.array(list(range(100)), type=pa.int32()),
})
write_deltalake(
        table_or_uri=delta_dir,
        data=data,
        mode='append')

table = DeltaTable(delta_dir)
table.create_checkpoint()

write_deltalake(
        table_or_uri=delta_dir,
        data=data,
        mode='append')

table = DeltaTable(delta_dir)  # No crash if this line is removed
table.vacuum()

More details:

[adamreeve]

Sorry I realised I've made a mistake in how I was reproducing this. I think there's still an issue but the repro code is wrong. I'll close this for now.

[adamreeve]

Repro fixed now, I can still reproduce this on main but not with 1.1.4.

[adamreeve]

Looks like this was caused by #3660. I can reproduce this with b34cff1 but not 2920177.

[rtyler]

Thanks for reporting! I'm guessing you bisected this that big commit because that's when a lot of changes happened underneath the hood 😄

I'll follow up with the @roeap on this when we're hacking today.

[abhi-airspace-intelligence]

Also seeing this error in Rust. Doing a vacuum right after a compaction. Pointed to d30b11f673b0111dbb0f904bf89d5b917ea652ed

[abhi-airspace-intelligence]

@roeap are you still working on this? This is blocking some work for me, happy to take over it if you have other priorities!

[roeap]

@abhi-airspace-intelligence - more indirectly, as we care cleaning up some tech debt left over from the kernel migration. I added it to a tracking issue for further kernel integration.

That said, we are always grateful for contributions :). It's kind of hard to gauge from the error message where this goes wrong, but I suspect that this will likely lie on similar code paths as the current refactoring to update our scans.

Since VACUUM should never read data files, we we should see this during some log replay, which means via kernel. the parquet reading in there should actually account for a schema mismatch between requested data and the data in the parquet file, so maybe we are reading log files somewhere without this.

In short - happy to support if you want to look into this!

[abhi-airspace-intelligence]

I'll attach LLDB and figure it out! I have a repo working based on @adamreeve's ported into rust. thanks!