Error decoding field 'stats' when creating checkpoint

[Nekit2217]

Environment

Delta-rs version: 0.18.1

Binding: rust

---

Bug

What happened:
While running kafka-delta-ingest the service crashes with an error when trying to create a checkpoint:
DeltaTable { source: Arrow { source: JsonError("whilst decoding field 'add': while decoding field 'stats': offset overflow decoding Utf8") } }

The error pops up here: https://github.com/apache/arrow-rs/blob/49840ec0f110da5e9a21ce97affd32313d0b720f/arrow-json/src/reader/string_array.rs#L81

What you expected to happen:
The checkpoint will be created successfully

How to reproduce it:
Not sure how to reproduce this, presumably it would require a table with a large delta log and data volume

More details:
Not sure what additional information to provide:
Partitioning by year, month, day, and hour;
Current transaction number: 00000000000002020757;
Approximately 4TB are written to the table per day;
Each parquet file is on average 800MB

[Nekit2217]

I fixed the error by changing the type of the stats field, but it feels like a workaround

        static ref ADD_FIELDS: Vec<ArrowField> = arrow_defs![
            path:Utf8,
            size:Int64,
            modificationTime:Int64,
            dataChange:Boolean,
            stats:Utf8,

        static ref ADD_FIELDS: Vec<ArrowField> = arrow_defs![
            path:Utf8,
            size:Int64,
            modificationTime:Int64,
            dataChange:Boolean,
            stats:LargeUtf8,

[Nekit2217]

After the change, spark stopped reading data

[sherlockbeard]

Utf8 is like to 2gb of data @_@

can you share your table schema and spark error?
also please check if there are any huge json in _delta_log
if you have stats on string or binary columns and they might create this problem.

i can kind of reproduce it . (plz don't do it if you love your laptop)

import pandas as pd
from deltalake import DeltaTable, write_deltalake
import pyarrow as pa

py_list_ran = {}
for i in range(1, 600):
    py_list_ran[chr(97) * 4000 * i] = 10000 * i



py_table = pa.Table.from_pylist([py_list_ran])



write_deltalake("temp5", py_table, engine="rust", configuration={"delta.dataSkippingNumIndexedCols": "-1"})

table = DeltaTable("temp5")

table.create_checkpoint()

print(table.schema())
# table.optimize.compact()

Exception: Json error: whilst decoding field 'add': whilst decoding field 'stats': offset overflow decoding Utf8

EDIT1

you can also recreate it using this it will stop when combined log is more than 2 gb

from deltalake import DeltaTable, write_deltalake, WriterProperties
from deltalake._internal import TableNotFoundError
import pyarrow as pa

for j in range(1, 50):
    py_list_ran = {}
    for i in range(1, 100):
        py_list_ran[chr(97) * 4000 * i] = 10000 * i



    py_table = pa.Table.from_pylist([py_list_ran])



    write_deltalake("temp5", py_table, engine="rust", configuration={"delta.dataSkippingNumIndexedCols": "-1"}, mode="append")

table = DeltaTable("temp5")

table.create_checkpoint()

print(table.schema())

    self._table = RawDeltaTable(
                  ^^^^^^^^^^^^^^
Exception: Json error: whilst decoding field 'add': whilst decoding field 'stats': offset overflow decoding Utf8

[Nekit2217]

Hi, sorry for the delay

Here is the schema:

schema

root
|-- field: timestamp (nullable = true)
|-- field: struct (nullable = true)
|    |-- field: long (nullable = true)
|    |-- field: long (nullable = true)
|    |-- field: long (nullable = true)
|-- field: struct (nullable = true)
|    |-- field: long (nullable = true)
|    |-- field: long (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: long (nullable = true)
|    |-- field: long (nullable = true)
|-- field: struct (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: decimal(30,15) (nullable = true)
|    |-- field: boolean (nullable = true)
|    |-- field: integer (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: integer (containsNull = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: integer (containsNull = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: array (nullable = true)
|    |    |    |    |    |-- field: string (containsNull = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: array (nullable = true)
|    |    |    |    |    |-- field: string (containsNull = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: long (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |    |-- field: boolean (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: integer (containsNull = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: long (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: string (containsNull = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: timestamp (nullable = true)
|    |    |-- field: long (nullable = true)
|    |    |-- field: long (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: long (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: string (containsNull = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: map (nullable = true)
|    |    |    |    |-- field: string
|    |    |    |    |-- field: string (valueContainsNull = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|-- field: struct (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: string (nullable = true)
|    |-- field: decimal(30,15) (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: string (containsNull = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: string (containsNull = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: integer (containsNull = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: boolean (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: array (nullable = true)
|    |    |    |-- field: string (containsNull = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: decimal(30,15) (nullable = true)
|    |    |    |-- field: integer (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: long (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: string (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: decimal(30,15) (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |    |-- field: long (nullable = true)
|    |    |-- field: struct (nullable = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|    |    |    |-- field: array (nullable = true)
|    |    |    |    |-- field: string (containsNull = true)
|-- field: integer (nullable = true)
|-- field: integer (nullable = true)
|-- field: integer (nullable = true)
|-- field: integer (nullable = true)
|-- field: struct (nullable = true)
|    |-- field: struct (nullable = true)
|    |    |-- field: double (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |    |-- field: integer (nullable = true)
|    |-- field: array (nullable = true)
|    |    |-- field: struct (containsNull = true)
|    |    |    |-- field: string (nullable = true)
|    |-- field: array (nullable = true)
|    |    |-- field: string (containsNull = true)
|    |-- field: array (nullable = true)
|    |    |-- field: struct (containsNull = true)
|    |    |    |-- field: string (nullable = true)
|    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: struct (nullable = true)
|    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: struct (nullable = true)
|    |    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |    |-- field: double (nullable = true)
|    |    |    |    |-- field: array (nullable = true)
|    |    |    |    |    |-- field: struct (containsNull = true)
|    |    |    |    |    |    |-- field: string (nullable = true)
|    |    |    |    |    |    |-- field: double (nullable = true)
|    |    |    |-- field: string (nullable = true)

Spark does not return any errors, it simply ignores the checkpoints that were recorded with stats: LargeUtf8

No huge json logs, 30 KB on average

We currently work around this by using a multi-part checkpoint that is created when running the optimization on the databricks,but this is very slow

[ion-elgreco]

@roeap in your new log replay, I think this should be addressed as well. We are currently always parsing stats columns in to Utf8, while we should actually do LargeUtf8.

I tried doing some surgery on the existing logReplay but it's quite fragile so I stopped eventually, however it did fix reading these tables, it broke other tests since I might have missed places