Skip to content

bug: incorrect typing for Table.filter #11667

@serl

Description

@serl

What happened?

I have this example script which is totally valid and does what it should, but the type-checker insists that's wrong.

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "ibis-framework[duckdb,polars]==10.8.0",
#     "duckdb==1.3.2",
# ]
# ///

import ibis

table = ibis.memtable(
    {
        "a": [1, 2, 3, 4, 5],
        "b": [10, 0, 30, -4, 30],
        "c": [True, True, True, False, False],
        "d": [None, None, 1, 2, 3],
    }
)

print("Original", table.to_polars())

# 1. Argument of type "bool" cannot be assigned to parameter "predicates" of type ...
filtered = table.filter(True).to_polars()
print("Filtered with True", filtered)

# 2. Operator ">" not supported for types "Column" and "Literal[0]"
filtered = table.filter(table.b > 0).to_polars()
print("Filtered with > condition", filtered)

# 3. Argument of type "bool" cannot be assigned to parameter "predicates" of type ...
filtered = table.filter(table.b == 30).to_polars()
print("Filtered with == condition", filtered)

# 4. Argument of type "Column" cannot be assigned to parameter "predicates" of type ...
filtered = table.filter(table.c).to_polars()
print("Filtered with bool column", filtered)

# 5. this one is all right. Note: notnull() returns a BooleanValue
filtered = table.filter(table.d.notnull()).to_polars()
print("Filtered with not null column", filtered)

I think there are some low-hanging fruits for which I would do a PR if you agree, and some other fruits which might be poisonous or something:

  1. This use-case seems useless, but we use it for some feature flags (and BTW it works at runtime). I'd add both bool and Sequence[bool] as valid predicates.
  2. This is very happy if instead of 0 I put ibis.literal(0), but the result doesn't change. Maybe updating the type hints for the Value.__gt__ method (and its friends) to include more types for the other argument could help? Note that ibis.literal("some string") would pass the typecheck and happily explode at runtime (IbisTypeError).
  3. Same as 2 actually. (But 1 should help as well, even if that makes no sense).
  4. Easy fix: add Column as a valid predicate (or maybe Value?). But since it works only if the column is actually boolean, is it really a fix?

I'd prepare a PR to add bool as it seems safe if you agree. What do you think about the rest?

What version of ibis are you using?

10.8.0

What backend(s) are you using, if any?

No response

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIncorrect behavior inside of ibis

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions