feat(RFC): A richer Expr IR
#2572
Draft
+7,337
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Thinking it might go: `Op` -> `Node` -> `Plan` But who knows really
Just trying to get the names & hierarchies done
- Will probably want to set up the `object.__setattr__` part somewhere - Not decided on how initialization should work
- Gonna need to revisit `Function` vs `FunctionExpr` - `functions.py` is confusing me now
Starting understand `Function` vs `FunctionExpr` a bit better now
- Probably going to loop back and explcitly define more input fields - At least until we accept `Expr` everywhere, these should more closely reflect `narwhals` than `polars`
Once `result: ResultIRs` is made immutable (or mutability stays within function boundaries) - most of `expr_expansion` will be safe to cache
dangotbanned
commented
Jun 11, 2025
Resolves - Parts of (#2572 (comment)) - All of (#2572 (comment))
10 tasks
We don't use it in `col`, and selectors are resolved in an entirely different way
Added after this PR started in (#2549)
The parsing will be reused by `when`, `BaseFrame.filter`, etc
I'm putting utils here to simplify keeping this branch in sync
Just added in #2654
I imagine the `rust` impl benefits from individual functions, but without a compiler we won't
Keeping things in sync with (#2556)
dangotbanned
commented
Jun 12, 2025
Comment on lines
+434
to
+446
| def expand_columns( | ||
| origin: ExprIR, /, columns: Columns, *, exclude: Excluded | ||
| ) -> Seq[ExprIR]: | ||
| if not _all_columns_match(origin, columns): | ||
| msg = "expanding more than one `col` is not allowed" | ||
| raise ComputeError(msg) | ||
| result: deque[ExprIR] = deque() | ||
| for name in columns.names: | ||
| if name not in exclude: | ||
| expanded = replace_with_column(origin, Columns, name) | ||
| expanded = rewrite_special_aliases(expanded) | ||
| result.append(expanded) | ||
| return tuple(result) |
There was a problem hiding this comment.
TODO: Plan how to handle exceptions & caching
All of these can raise currently:
expand_columnsexpand_indicesreplace_wildcardrewrite_special_aliases
IIRC, the issues are:
- Raising an exception within a cached function
I think makes the next (matching) call returnNoneinstead of raising again- No idea where I got that ^ from, seems to be working fine
- Returning the exception
- I think there's an issue of raising a previously raised exception
- Possible solution is introducing a result type that always reconstructs the exception before raising
- Maybe I've been reading too much
rust😅
- Maybe I've been reading too much
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Will close #2571
What type of PR is this? (check all applicable)
Related issues
Exprinternal representation #2571Checklist
If you have comments or can explain your changes, please do so below
Important
See (#2571) for detail!!!!!!!
What's here so far is mostly mocking up the solution - not actually implementing it yet.
Very open to feedback
Tasks
pl.Expr.metapl.Expr.meta)ExprIRmetamethods_typing_compatmodule #2578Merge another PR with (perf: Avoid module-levelimportlib.util.find_spec#2391 (comment)) firstTypeVardefaults moreTypeVar("T", bound=Thing, default=Thing), instead of an opaqueExprIRSelector(s)narwhals/narwhals/_plan/expr.py
Lines 336 to 337 in 0bada48
BinaryExprthat describes the restricted set of operators that are allowedpolars, since they wrappl.colinternallyIntoExprin more places (including and beyond whatnarwhalsallows now)demo.py*_horizontalconcat_str)dummy.pyover,sort_by)FunctionOptions+ friends see commentWhere does the{flags: ...}->{collect_groups: ..., flags: ...}expansion happen?polars>=1.3.0fixed the issue (see comment)Ternarywhen-then-otherwise🥳)Metais_*,has_*,output_namemeta methodsroot_namesundo_aliases,popNamename.py(Expr::KeepName,Expr::RenameAlias)polarswill help with themetamethodsCat,Struct,List(a3e29d1)String(72c33ce)DateTime(aee0a7e)_expression_parsing.pyrulesrustversion worksExpansionFlagsexpand_function_inputsrewrite_projectionsreplace_selectorexpand_selectorreplace_selector_innerreplace_and_add_to_resultsreplace_nthprepare_excludedexpand_columnsexpand_dtypesreplace_dtype_or_index_with_columndtypes_match(probably can solve w/ existingnarwhals)expand_indicesreplace_index_with_columnreplace_wildcardrewrite_special_aliasesreplace_wildcard_with_columnreplace_regexexpand_regexExprIR.map_irExprIR #2572 (comment))ExprIR.map_irfor most nodesWindowExpr.map_irFunctionExpr.map_irRollingExpr,AnonymousExprinheritselectors