no support for OR in partition predicates

[echai58]

Environment

Delta-rs version: 0.13.0

Binding: python

Environment:

• Cloud provider: n/a
• OS: ubuntu 22.04
• Other: testing locally via jupyter notebook

---

Bug

What happened:
There are inconsistencies between the python type-hints, docstring comments, and rust implementation for partition predicates.

In table.py, we see partitions: Optional[List[Tuple[str, str, Any]]] (https://github.com/delta-io/delta-rs/blob/main/python/deltalake/table.py#L704). There is also a docstring comment here referencing a DeltaTable.files_by_partitions, which seems to be a deprecated method, so I'm going to assume that is an out of date comment.

In _internal.pyi, we see Optional[FilterType] (https://github.com/delta-io/delta-rs/blob/main/python/deltalake/_internal.pyi#L86), which is defined as a DNF or Conjuction (https://github.com/delta-io/delta-rs/blob/main/python/deltalake/_internal.pyi#L717-L720).

In lib.rs, we see PartitionFilterValue (https://github.com/delta-io/delta-rs/blob/main/python/src/lib.rs#L655), which is defined as a string, or a vector of strings (https://github.com/delta-io/delta-rs/blob/main/python/src/lib.rs#L60-L63).

To my understanding, these three, which should all be defining the same partition filter type, all have different typing. The one in table.py and lib.rs are relatively congruent with each other, if we treat Any as Union[str,List[str]], which sort of makes sense.

The bigger difference is with _internal.pyi, which presents a whole different form of filter. This DNF style would allow for OR in the predicate filter, but this seems to be impossible in the python or rust typing. Would it be difficult to add support for OR in partition predicates via this DNF style or another?

What you expected to happen:
At the minimum, in the above code's current standing, I am not sure what the correct typing for predicate filters are. If the python one is incorrect, that should be fixed to provide accurate type hints and allow for correct static type checking when using python bindings.

Perhaps this should be raised in a Feature Request, but I'm also wondering why there isn't support for OR in predicate filters?

[ion-elgreco]

I think the type consistency is resolved now

[FrankPortman]

Wanted to bump this. My understanding is OR is still not supported via the DNF form and we have to rewrite clauses ourselves by first querying all partitions and then applying some sort of logic to turn into the appropriate IN clause (which can be quite large, depending on what you would have ORd together.)

Is that right or has there been an update?

[roeap]

We are currently looking to the least intrusive way for adopting kernel more broadly. One this is done we can access a more rigorous way of doing partition pruning.

While I cannot quite yet promise a specific delivery date, I am actively working on this and landing commits in kernel and here that pays into this.

Thanks for your patience and bear with us just a little bit longer 😄.

[FrankPortman]

No worries any and all OSS is appreciated. Just out of curiosity what does "kernel" mean here?

[roeap]

delta-kernel-rs is a new implementation of the innermost delta log handling logic.

it incorporates a lot of what we learned over the years of reading and writing delta 😄.

[FrankPortman]

@roeap I've seen a lot of work land related to kernel lately. Are things in a state where more flexible partition filters are easy to contribute by non-experts (such as myself). I am open to adding a PR for true DNF filters that include or/< and so on, if you can give some pointers.

[roeap]

@FrankPortman - awesome! I think they are, with a "but".. so lets get that out of the way :). We need @rtyler to stamp #3669, which includes using kernel Predicates derives from PartitionFilters.

So the flow as it is in the mentioned PR is DNF -> PartitionFilter (delta-rs type) -> Predicate (kernel type).

The idea would be to skip PartitionFilter and immediately translate DNF to Predicate. The good news is we can already build a function that does DNF -> Predicate (maybe looking at the current translation can give inspiration?).

Once the above PR lands, we "just" need to translate DNF first chance we get and get rid of PartitionFIlter along the way.

One. nice side-effect of this is that we are no longer constrained to just filtering on partitions, but can provide a predicate that references any columns in the table. One difference being though that for partition filters, we can make stronger guarantees as for generic file skipping.

[FrankPortman]

@roeap thanks for the detailed response.

One concern I had is that right now partition_filters claims that it takes DNF inputs but as far as I can tell it's not truly DNF as it is missing one level of nesting that would allow it to represent arbitrary ANDs and ORs (right now it can just do AND).

I do think proper DNF would be very valuable, but this also feels like a nontrivial API and behavior change. Any suggestions or votes of confidence here to proceed? There's also a chance I grossly misunderstood how the current thing works, so would appreciate any correction that the current format can take actual DNF.

TLDR; I want to allow it to accept "actual" DNF and also expand the clauses it covers (<, >, etc).

[FrankPortman]

Re providing filters on any column vs partition columns, maybe this is a guard we can hide behind a flag? At least during the initial rollout?