Keep arrow metadata in Delta Table metadata

[j-bennet]

Environment

Delta-rs version:

0.10.0

Binding:

Python

Environment:

• Cloud provider:
• OS: macOS
• Other:

---

Bug

delta-rs loses metadata for parquet written with pandas (example data is attached).

test.parquet.zip

from deltalake import DeltaTable
import pyarrow.parquet as pq

if __name__ == "__main__":
    # read it back with delta-rs
    dt = DeltaTable("test.parquet")
    print("\nDeltaTable schema:")
    print(dt.schema().to_pyarrow().to_string())

    # read it back with pyarrow
    table = pq.read_table("test.parquet")
    print("\nPyarrow schema:")
    print(table.schema.to_string())

This outputs:

DeltaTable schema:
col2: string
col1: int32

Pyarrow schema:
col2: dictionary<values=string, indices=int32, ordered=0>
col1: dictionary<values=int32, indices=int32, ordered=0>
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 509

The schema metadata part in pyarrow.table is nowhere to be found in DeltaTable. Is it present, but not public? How can it be accessed?

[wjones127]

We don't save this metadata in Delta Tables. We could perhaps create a custom key in the configuration field of the table metadata, and preserve that in the Arrow schemas returned (which get passed onto Pandas).

[kylebarron]

Is this a limitation of Delta Tables or of the client library? I'm specifically wondering in the context of GeoParquet, which uses schema metadata to declare per-column information like geometry type. With pyarrow we'd set schema metadata on the table and then write to Parquet, but it doesn't look like that works here.

[ion-elgreco]

@kylebarron if you would like to do this you could hijack the configuration key in the table metadata for it. But it still requires an implementation on the client side to use it.

And it will only work for that specific cliënt

[ion-elgreco]

@kylebarron This is actually nowadays supported after all the changes to schema evolution. See an example below:

import pyarrow as pa
from deltalake import write_deltalake
from deltalake import DeltaTable


id_col = pa.field("id", pa.int32())
gc = pa.field("gc", pa.int32()).with_metadata({"delta.generationExpression": "10"})
data = pa.Table.from_pydict(
    {"id": [1, 2], "gc": [10, 10]}, schema=pa.schema([id_col, gc])
)

write_deltalake(
    "test_table",
    data
)

dt = DeltaTable("test_table")

print(dt.schema().to_pyarrow())
print(dt.to_pyarrow_table().schema)

id: int32
gc: int32
  -- field metadata --
  delta.generationExpression: '10'

id: int32
gc: int32
  -- field metadata --
  delta.generationExpression: '10'