Extension deserializer should tolerate equivalent string/binary layouts (strict storage_type check breaks Delta compaction)

[brian-arnold]

Summary

The extension-type deserializer requires an exact physical Arrow storage type (storage_type != _storage → raise). Because string, large_string, and string_view are physically-different-but-logically-identical UTF-8 layouts, any Arrow/Delta operation that re-canonicalizes the physical layout — notably delta-rs optimize/compaction of a DeltaTableDatabase — makes previously-readable cached records fail to load with:

Arrow extension type 'orcapod.file': expected storage_type DataType(large_string) but got DataType(string).

This affects every string-backed extension (orcapod.file, orcapod.directory, spikeinterface.recording, …). The data is intact and unchanged — only its physical string layout was normalized — but the strict check rejects it at read time.

Where it happens (commit 966d759a)

src/orcapod/extension_types/registry.py (~L78–94), in __arrow_ext_deserialize__:

if storage_type != _storage:      # _storage = pa.large_string() for file/directory/etc.
    raise ValueError(
        f"Arrow extension type '{_name}': expected storage_type "
        f"{_storage!r} but got {storage_type!r}."
    )

__arrow_ext_deserialize__ runs at parquet/IPC scan time, whenever a field carries ARROW:extension:name, and is handed the file's physical storage type — before DeltaTableDatabase's own normalizing .cast(...) (which already casts string_view → large_string to dodge missing string_view kernels, see databases/delta_lake_databases.py L398/L874, ITL-22) can run.

Concrete trigger (observed)

A DeltaTableDatabase result store was compacted with delta-rs optimize.compact() + create_checkpoint(). Comparing a table's data files before/after:

• Before (orcapod-written): column physical type string_view, no ARROW:extension:name embedded in the parquet field → Arrow reads a plain string_view column, never calls __arrow_ext_deserialize__, and orcapod's read-path cast normalizes it. Reads fine.
• After (delta-rs rewrite): column physical type string, and ARROW:extension:name now embedded in the parquet field (regenerated from the Delta log schema) → Arrow reconstructs the extension at scan time → __arrow_ext_deserialize__(storage_type=string) → string != large_string → raises, before any cast.

So it's the combination: delta-rs both downgrades the physical layout and embeds the extension metadata into the parquet field, triggering the strict check that the originals bypassed.

Net effect downstream: the entire result cache became unreadable after a routine maintenance compaction, forcing full recompute (recovered here by Delta restore, since VACUUM had not run).

Proposed fix

Relax the storage-type check in _deserialize to accept any member of the same logical family and cast to the canonical wide type, instead of requiring exact physical equality:

• UTF-8 string layouts (string / large_string / string_view) → canonical large_string
• binary layouts (binary / large_binary / binary_view) → canonical large_binary

Keep the rest strict:

• Still reject genuine cross-category mismatches (e.g. string where binary/large_binary was registered) — those are real semantic errors.
• Keep the serialized != _metadata (extension-parameter) check strict — that's where type identity/parameters actually live.

Notes / rationale:

• The cast direction is always widening (string/string_view → large_string), which is lossless. The only unsafe direction (narrowing a >2 GB value to 32-bit offsets) never occurs when normalizing toward large_*.
• This does not weaken reproducibility: these are physical encodings of identical values, and the reader already normalizes string_view → large_string today. Logical identity is preserved by the extension name + metadata, both still validated.
• Please confirm identity/content hashing is computed on the normalized (canonical) form, so physical layout can never affect a record/dataset hash (i.e. normalize-then-hash). Given string_view is already canonicalized before use, this is likely already the case, but worth verifying as part of this change so the relaxation can't introduce spurious cache misses.

Impact

• Makes DeltaTableDatabase result stores survive routine Delta maintenance (compaction/OPTIMIZE, and any other operation that re-canonicalizes physical layout).
• Removes a sharp edge where standard small-files compaction silently invalidates the cache.

Reproduced with orcapod 966d759a760cf9759172435028c9e1252a785d2c, pyarrow 25.0.0, arro3 0.8.1, deltalake (delta-rs) as used by DeltaTableDatabase.