Define `filter (where ...)` clause by hadley · Pull Request #203 · tidyverse/data-dict

hadley · 2026-08-10T18:03:03Z

Came up as part of #134

This is the equivalent of sum(order_total[status_cd == 90])

Came up as part of #134

gaborcsardi

Mostly straightforward, at least at the spec level.

I think we don't actually need to make FILTER and WHERE a reserved word, but I guess we might as well.

The interaction with COLUMNS() is not explicitly defined, we should probably say something about that.

We'll need to add some check spec to validation.md for FILTER.

gaborcsardi · 2026-08-11T14:05:02Z

+  expr: SUM(order_total) FILTER (WHERE status_cd = 90)
+```
+
+The condition is a `row`-shape boolean expression over the same table, and only rows where it is true are folded; following [CHECK semantics](#truth-and-null), a row whose condition is null is excluded, exactly as `false` is. A filter that admits no rows leaves the aggregate with [empty input](#empty-input), so `SUM(x) FILTER (WHERE false)` is null and `COUNT(x) FILTER (WHERE false)` is 0.


Small contradiction: WHERE false is actually not row-shape.

gaborcsardi · 2026-08-11T14:09:00Z


+#### Filtering: `FILTER (WHERE ...)` {#filter}
+
+Any aggregate may be narrowed to a subset of rows with a `FILTER (WHERE ...)` clause, written after the closing parenthesis:


I guess FILTER and WHERE are case insensitive. Also, we need to add them as reserved words?

gaborcsardi · 2026-08-11T14:23:25Z

+  expr: SUM(order_total) FILTER (WHERE status_cd = 90)
+```
+
+The condition is a `row`-shape boolean expression over the same table, and only rows where it is true are folded; following [CHECK semantics](#truth-and-null), a row whose condition is null is excluded, exactly as `false` is. A filter that admits no rows leaves the aggregate with [empty input](#empty-input), so `SUM(x) FILTER (WHERE false)` is null and `COUNT(x) FILTER (WHERE false)` is 0.


This is not CHECK semantics, I think. it is WHERE and HAVING semantics in SQL. CHECK semantics allows nulls.

hadley · 2026-08-11T19:26:10Z

You're imagining that you could use COLUMNS() in WHERE()? Like ROW_COUNT() FILTER (WHERE COLUMNS(*) IS NOT NULL)?

Although that does not do what I expect in duckdb (from claude)

CREATE TABLE t AS SELECT * FROM (VALUES (1,10,100),(2,NULL,200),(3,30,NULL)) v(x,a,b);

SELECT sum(x) FILTER (WHERE COLUMNS(*) IS NOT NULL) FROM t;
-- ┌───┬───┬───┐
-- │ x │ a │ b │
-- ├───┼───┼───┤
-- │ 6 │ 4 │ 3 │

gaborcsardi · 2026-08-11T20:27:49Z

I think we can make ROW_COUNT() FILTER (WHERE COLUMNS(*) IS NOT NULL) do what you expect by simply expanding COLUMNS() within the condition and AND-ing them, as usual. I guess this just works. We still have the at most one COLUMNS() per expressions of course.

So technically nothing is needed, although an example may be still nice.

The clause parses on any call, as the grammar admits; the checker then requires an aggregate (S21), a boolean condition (S21), and no aggregate inside the condition (S30) — it is evaluated per row, before any fold has finished. The IR carries the lowered filter, eval folds only rows the condition keeps (null excludes, as false does), and both emitters refuse it as Unsupported until translation is implemented.

#Conflicts: # crates/data-dict/src/emit/mod.rs # crates/data-dict/src/emit/r_tidyverse.rs

hadley · 2026-08-12T17:29:13Z

@gaborcsardi now with implementation

gaborcsardi · 2026-08-13T08:55:14Z

+Any aggregate may be narrowed to a subset of rows with a `FILTER (WHERE ...)` clause, written after the closing parenthesis:
+
+```yaml
+- name: net_revenue


This needs #202 to be a valid example. Just noting.

gaborcsardi · 2026-08-13T09:02:05Z

+  expr: SUM(order_total) FILTER (WHERE status_cd = 90)
+```
+
+You can use `COLUMNS()` in the filter clause, with its usual interpretation of `AND`ing together the results from all selected columns:


I think the example is wrong here, it does not do what you want if you AND together the results from all selected columns. We could be more explicit and say that the aggregate is folded once per selected column, and use a different example. E.g:

`COLUMNS()` may appear in the filter clause, and [the usual rule](#selecting-multiple-columns) applies: the whole expression — the fold included — is evaluated once per selected column, so the `AND` sits outside the aggregate, not inside the condition. - assert: AVG(qty) FILTER (WHERE COLUMNS('flag_.*')) > 0 description: Mean quantity is positive among the rows each flag marks.

gaborcsardi · 2026-08-13T09:08:45Z

+            Some(filter) => {
+                cx.push("sum(");
+                cx.free(filter)?;
+                cx.push(", na.rm = TRUE)");


This is going to break on constant conditions, e.g. ROW_COUNT() FILTER (WHERE TRUE) is translated to sum(TRUE, na.rm = TRUE), which is always 1.

I guess we'd need to add a rep_len() with length n() / .N here, and it would be unsupported in base R.

Op::RowCount => match filter { Some(f) if f.shape == Shape::Const => { // A constant condition recycles, so it needs the row count to // count against. cx.push("sum(rep_len("); cx.free(f)?; match d { Dialect::Tidyverse => cx.push(", n())"), Dialect::DataTable => cx.push(", .N)"), Dialect::Base => return Err(Unsupported { .. }), } cx.push(", na.rm = TRUE)"); } Some(f) => { /* current code */ } None => { /* current code */ } }

gaborcsardi · 2026-08-13T09:21:02Z

 | S19 | Malformed assertion | E | An `assert` expression fails to parse (a syntax error in the [expression language](expressions.md)). |
 | S20 | Unknown assertion column | E | An `assert` expression, or a `COLUMNS([...])` list, references a column not present on the table, or a field access names a field not declared on its `struct`. |
-| S21 | Ill-typed assertion | E | An `assert` expression is syntactically valid but semantically wrong: an operator or function applied to the wrong operand type (including a column a `COLUMNS(...)` selects, and an argument outside a signature's [type class](expressions.md#type-classes), such as `SUM` of a string), a wrong function arity, a non-boolean top-level expression, more than one `COLUMNS(...)`, a malformed `SIMILAR TO` / `COLUMNS('...')` regex, a field access on anything but a `struct` (including through a `list`), or a bare `struct` or `list` column used where a value is needed (anywhere but `IS [NOT] NULL` and `COUNT`, which ask only whether a value is null). |
+| S21 | Ill-typed assertion | E | An `assert` expression is syntactically valid but semantically wrong: an operator or function applied to the wrong operand type (including a column a `COLUMNS(...)` selects, and an argument outside a signature's [type class](expressions.md#type-classes), such as `SUM` of a string), a wrong function arity, a non-boolean top-level expression, a `FILTER (WHERE ...)` clause on a non-aggregate function or with a non-boolean condition, more than one `COLUMNS(...)`, a malformed `SIMILAR TO` / `COLUMNS('...')` regex, a field access on anything but a `struct` (including through a `list`), or a bare `struct` or `list` column used where a value is needed (anywhere but `IS [NOT] NULL` and `COUNT`, which ask only whether a value is null). |


At least for NOW() FILTER (...) we do not report S21, but the NOW() is parsed and then S19 is reported for the malformed expression. (I am not saying that this is super important, though...)

gaborcsardi · 2026-08-13T09:24:38Z

 | S28 | Invalid type | E | A column's `type` is not a recognised type string. Valid types are the fixed scalars (`string`, `number`, `number(id)`, `number(ordinal)`, `number(quantity)`, `boolean`, `date`, `datetime`), `enum`, `struct`, and `list(element_type)` where the element type is any of the above — including another `list(...)`, nested to any depth. |
 | S29 | Invalid constraint on list or struct | E | A `primary_key`, `foreign_key`, or `unique` constraint appears on a `list` or `struct` column. (There is no such check for fields: a field can't carry `constraints` at all, which the schema enforces structurally.) |
-| S30 | Nested aggregate | E | An aggregate function's argument contains another aggregate call, as in `AVG(MIN(x))`: an aggregate folds one value per row, and another aggregate gives it a single value. This is the only [shape](expressions.md#shapes) an expression can get wrong — every other combination is legal, including mixing a row-level subexpression with an aggregate one (`value <= 2 * MIN(value)`). |
+| S30 | Nested aggregate | E | An aggregate function's argument contains another aggregate call, as in `AVG(MIN(x))`: an aggregate folds one value per row, and another aggregate gives it a single value. The same goes for a `FILTER (WHERE ...)` condition. These are the only ways an expression's [shape](expressions.md#shapes) can be wrong — every other combination is legal, including mixing a row-level subexpression with an aggregate one (`value <= 2 * MIN(value)`). |


We need to change the message in validate_spec.rs to reflect this change. Currently it is

"S30" => "An aggregate can't be nested inside another aggregate.",

which does not cover the new failure mode.

Define filter (where ...) clause

445d779

Came up as part of #134

hadley requested a review from gaborcsardi August 10, 2026 18:03

gaborcsardi approved these changes Aug 11, 2026

View reviewed changes

gaborcsardi reviewed Aug 11, 2026

View reviewed changes

hadley added 6 commits August 11, 2026 17:05

Merge commit '13451b0ab7c7d56df208a24332a1be5e12ed6e16'

ff59cbc

Add translations

e75099f

Add a couple more tests

333c20d

Merge commit 'd768f19dc417ba2cb235689c608c28af6f905fe9'

6246e88

#Conflicts: # crates/data-dict/src/emit/mod.rs # crates/data-dict/src/emit/r_tidyverse.rs

Simplify tests

f78963c

hadley requested a review from gaborcsardi August 12, 2026 17:25

Mention COLUMNS

4e8d192

gaborcsardi reviewed Aug 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

Define `filter (where ...)` clause - #203

Define `filter (where ...)` clause#203
hadley wants to merge 8 commits into
mainfrom
filter-where

hadley commented Aug 10, 2026

Uh oh!

gaborcsardi left a comment

Uh oh!

gaborcsardi Aug 11, 2026

Uh oh!

gaborcsardi Aug 11, 2026 •

edited

Loading

Uh oh!

gaborcsardi Aug 11, 2026

Uh oh!

hadley commented Aug 11, 2026 •

edited

Loading

Uh oh!

gaborcsardi commented Aug 11, 2026

Uh oh!

hadley commented Aug 12, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		#### Filtering: `FILTER (WHERE ...)` {#filter}

		Any aggregate may be narrowed to a subset of rows with a `FILTER (WHERE ...)` clause, written after the closing parenthesis:

Uh oh!

Conversation

hadley commented Aug 10, 2026

Uh oh!

gaborcsardi left a comment

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

hadley commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaborcsardi commented Aug 11, 2026

Uh oh!

hadley commented Aug 12, 2026

Uh oh!

gaborcsardi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

gaborcsardi Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gaborcsardi Aug 11, 2026 •

edited

Loading

hadley commented Aug 11, 2026 •

edited

Loading