Skip to content

Allow choosing Large variants of string, binary, list types to avoid overflow panics #1402

@chiragjn

Description

@chiragjn

Describe the bug

The issue started from delta-io/delta-rs#3790 where we are trying to z-order over a large subset of data. While sorting and merging we run into an overflow panic because few of the string columns don't end up fitting in a string array and would need a large string array

User ion-elgreco made an attempt here #300 which we ended up using in our fork to solve this issue specifically for z-order truefoundry/delta-rs@42c892a

We saw #1294 that made a change to allow large types on write paths, it would be good to have a choice on read paths too which would help during z-order, aggregations during querying, and other cases

To Reproduce

We have re-production script for z-order panic here delta-io/delta-rs#3790 (comment)

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions