fix: unify `ColumnNotFound` for `duckdb` and `pyspark` #2493

EdAbati · 2025-05-04T20:46:31Z

What type of PR is this? (check all applicable)

Related issues

Closes error reporting: unify "column not found" error message for DuckDB / spark-like #2472

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

EdAbati · 2025-05-04T21:13:43Z

I think I can make some other clean-up of repetitive code. I'll try tomorrow morning

EdAbati · 2025-05-05T07:04:48Z

I made a followup PR #2495 with the cleanup :)

MarcoGorelli

thanks for working on this! just got a comment on the .columns usage

MarcoGorelli · 2025-05-05T07:29:55Z

narwhals/_duckdb/expr.py

@@ -186,7 +187,14 @@ def from_column_names(
        context: _FullContext,
    ) -> Self:
        def func(df: DuckDBLazyFrame) -> list[duckdb.Expression]:
-            return [col(name) for name in evaluate_column_names(df)]
+            col_names = evaluate_column_names(df)
+            missing_columns = [c for c in col_names if c not in df.columns]


df.columns comes with overhead unfortunately, I think we should avoid calling it where possible. How much overhead depends on the operation

I was hoping we could do something like we do for Polars. That is to say, when we do select / with_columns, we wrap them in try/except, and in the except block we intercept the error message to give a more useful / unified one

Ah interesting, I was not aware 😕

What is happening in the background in duckdb that causes this overhead ? Do you have a link to the docs? (Just want to learn more)

Also, is it a specific caveat of duckdb? I don't think we should worry about that in spark-like but I might be wrong

I will update the code tonight anyway (but of course feel free to add commits to this branch if you need it for today's release)

df.columns comes with overhead unfortunately, I think we should avoid calling it where possible. How much overhead depends on the operation

@MarcoGorelli could we add that to (#805) and put more of a focus towards it? 🙏

I don't think it's documented, but evaluating .columns may sometimes require doing a full scan. Example:

In [48]: df = pl.DataFrame({'a': rng.integers(0, 10_000, 100_000_000), 'b': rng.integers(0, 10_000, 100_000_000)}) In [49]: rel = duckdb.table('df') 100% ▕████████████████████████████████████████████████████████████▏ In [50]: rel1 = duckdb.sql("""pivot rel on a""") In [51]: %timeit rel.columns 385 ns ± 7.62 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) In [52]: %timeit rel1.columns 585 μs ± 3.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Granted, we don't have pivot in the Narwhals lazy API, but a pivot may appear in the history of the relation which someone passes to nw.from_native, and the output schema of pivot is value-dependent (😩 )

The same consideration should apply to spark-like

How do those timings compare to other operations/metadata lookups on the same tables?

.alias for example is completely non-value-dependent, so that stays fast

In [60]: %timeit rel.alias 342 ns ± 2.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) In [61]: %timeit rel1.alias 393 ns ± 2.6 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

narwhals/_spark_like/dataframe.py

EdAbati · 2025-05-06T06:12:47Z

narwhals/_spark_like/dataframe.py

+            try:
+                return self._with_native(self.native.select(*new_columns_list))
+            except AnalysisException as e:
+                msg = f"Selected columns not found in the DataFrame.\n\nHint: Did you mean one of these columns: {self.columns}?"


Not 100% sure about this error message. I don't we can access the missing column names at this level, am I missing something?

I think what you're written is great - even though we can't access them, we can still try to be helpful

…und-error

EdAbati · 2025-05-09T12:04:41Z

tests/frame/select_test.py

I split the test into lazy and eager to simplify a bit the if-else statements. I hope it is a bit more readable ?

EdAbati · 2025-05-09T12:05:28Z

tests/frame/select_test.py

+        return df
+
+    if constructor_id == "polars[lazy]":
+        msg = r"^e|\"(e|f)\""


Before it was msg = "e|f". Now it is a bit stricter

EdAbati · 2025-05-09T12:05:58Z

tests/frame/select_test.py

+    with pytest.raises(ColumnNotFoundError, match=msg):
+        df.select(nw.col("fdfa"))


before this was not tested for polars

EdAbati · 2025-05-09T12:08:06Z

tests/frame/select_test.py

+    constructor_lazy: ConstructorLazy, request: pytest.FixtureRequest
+) -> None:
+    constructor_id = str(request.node.callspec.id)
+    if any(id_ == constructor_id for id_ in ("sqlframe", "pyspark[connect]")):


sqlframe and pystpark.connect raise errors at collect. 😕

~~I need to double check pystpark.connect. Currently cannot set it up locally... Working on it ⏳~~

Do you have an idea on how to deal with these?

MarcoGorelli · 2025-05-11T10:54:37Z

narwhals/_duckdb/dataframe.py

+        try:
+            return self._with_native(self.native.select(*selection))
+        except duckdb.BinderException as e:
+            raise ColumnNotFoundError.from_available_column_names(self.columns) from e


do we risk catching other errors with this? BinderException might be a bit broad, shall we also match on str(e) before raising ColumnNotFoundError?

we should probably also do this for:

with_columns

simple_select

filter

?

@MarcoGorelli this has reminded me about an issue with SparkLikeLazyFrame

narwhals/narwhals/_spark_like/dataframe.py

Lines 152 to 164 in b7001e4

def _to_arrow_schema(self) -> pa.Schema: # pragma: no cover

import pyarrow as pa # ignore-banned-import

from narwhals._arrow.utils import narwhals_to_native_dtype

schema: list[tuple[str, pa.DataType]] = []

nw_schema = self.collect_schema()

native_schema = self.native.schema

for key, value in nw_schema.items():

try:

native_dtype = narwhals_to_native_dtype(value, self._version)

except Exception as exc: # noqa: BLE001,PERF203

native_spark_dtype = native_schema[key].dataType # type: ignore[index]

This one is a bigger problem because it captures CTRL+C, so you can't easily stop the test suite while it's running

Good point @MarcoGorelli !

Regarding, simple_select we are already catching a blind Exception:

narwhals/narwhals/dataframe.py

Lines 168 to 180 in 2a16835

if flat_exprs and all(isinstance(x, str) for x in flat_exprs) and not named_exprs:

# fast path!

try:

return self._with_compliant(

self._compliant_frame.simple_select(*flat_exprs)

)

except Exception as e:

# Column not found is the only thing that can realistically be raised here.

available_columns = self.columns

missing_columns = [x for x in flat_exprs if x not in available_columns]

raise ColumnNotFoundError.from_missing_and_available_column_names(

missing_columns, available_columns

) from e

And should already be caught by the first test:

narwhals/tests/frame/select_test.py

Lines 104 to 109 in 2a16835

if isinstance(df, nw.LazyFrame):

with pytest.raises(ColumnNotFoundError, match=msg):

df.select(selected_columns).collect()

else:

with pytest.raises(ColumnNotFoundError, match=msg):

df.select(selected_columns)

Maybe we should have a _missing_column_exception property in BaseFrame that should be specified for each backend. So here we can catch the exact exception here.
What do you think? I could do that in a separate PR to see how it looks like

Regarding filter and with_column, I think we are not testing what happens if we use not existing columns in these methods yet, or am I missing something? I can make another small PR for just this

EdAbati added 3 commits May 4, 2025 22:45

unify ColumnNotFound

6f7a574

revert

8fe45e6

Merge branch 'main' into unify-column-not-found-error

45f09e0

EdAbati mentioned this pull request May 5, 2025

chore: always use check_columns_exist where possible #2495

Open

10 tasks

EdAbati marked this pull request as ready for review May 5, 2025 07:04

EdAbati requested review from MarcoGorelli and dangotbanned May 5, 2025 07:09

MarcoGorelli requested changes May 5, 2025

View reviewed changes

EdAbati added 2 commits May 6, 2025 08:03

try except during select

6d58bc2

catch correct exception

be834b3

EdAbati added pyspark Issue is related to pyspark backend pyspark-connect error reporting labels May 6, 2025

EdAbati commented May 6, 2025

View reviewed changes

narwhals/_spark_like/dataframe.py Outdated Show resolved Hide resolved

EdAbati commented May 6, 2025

View reviewed changes

EdAbati added 7 commits May 6, 2025 08:13

coverage

19d6e24

Merge branch 'main' into unify-column-not-found-error

f0a9821

Merge remote-tracking branch 'upstream/main' into unify-column-not-fo…

c1cabd1

…und-error

separate lazy and eager tests

917d073

coverage

5d2972d

cleanup exception

4080101

use constructor_id

8442596

EdAbati commented May 9, 2025

View reviewed changes

EdAbati changed the title ~~fix: unify ColumnNotFound for duckdb and pyspark/sqlframe~~ fix: unify ColumnNotFound for duckdb and pyspark May 9, 2025

what is going on in pyspark connect?

937a123

ignore pyspark connect

0544f30

MarcoGorelli reviewed May 11, 2025

View reviewed changes

EdAbati added 3 commits May 11, 2025 12:58

Merge branch 'main' into unify-column-not-found-error

f41c037

Merge branch 'main' into unify-column-not-found-error

bb17703

Merge branch 'main' into unify-column-not-found-error

df18f34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: unify `ColumnNotFound` for `duckdb` and `pyspark` #2493

fix: unify `ColumnNotFound` for `duckdb` and `pyspark` #2493

EdAbati commented May 4, 2025

EdAbati commented May 4, 2025

EdAbati commented May 5, 2025 •

edited

Loading

MarcoGorelli left a comment

MarcoGorelli May 5, 2025

EdAbati May 5, 2025 •

edited

Loading

dangotbanned May 5, 2025

MarcoGorelli May 5, 2025

dangotbanned May 5, 2025

MarcoGorelli May 5, 2025

EdAbati May 6, 2025

MarcoGorelli May 9, 2025

EdAbati May 9, 2025

EdAbati May 9, 2025 •

edited

Loading

EdAbati May 9, 2025

EdAbati May 9, 2025 •

edited

Loading

MarcoGorelli May 11, 2025

dangotbanned May 11, 2025

EdAbati May 12, 2025

		with pytest.raises(ColumnNotFoundError, match=msg):
		df.select(nw.col("fdfa"))

	def _to_arrow_schema(self) -> pa.Schema: # pragma: no cover
	import pyarrow as pa # ignore-banned-import

	from narwhals._arrow.utils import narwhals_to_native_dtype

	schema: list[tuple[str, pa.DataType]] = []
	nw_schema = self.collect_schema()
	native_schema = self.native.schema
	for key, value in nw_schema.items():
	try:
	native_dtype = narwhals_to_native_dtype(value, self._version)
	except Exception as exc: # noqa: BLE001,PERF203
	native_spark_dtype = native_schema[key].dataType # type: ignore[index]

	if flat_exprs and all(isinstance(x, str) for x in flat_exprs) and not named_exprs:
	# fast path!
	try:
	return self._with_compliant(
	self._compliant_frame.simple_select(*flat_exprs)
	)
	except Exception as e:
	# Column not found is the only thing that can realistically be raised here.
	available_columns = self.columns
	missing_columns = [x for x in flat_exprs if x not in available_columns]
	raise ColumnNotFoundError.from_missing_and_available_column_names(
	missing_columns, available_columns
	) from e

	if isinstance(df, nw.LazyFrame):
	with pytest.raises(ColumnNotFoundError, match=msg):
	df.select(selected_columns).collect()
	else:
	with pytest.raises(ColumnNotFoundError, match=msg):
	df.select(selected_columns)

fix: unify ColumnNotFound for duckdb and pyspark #2493

Are you sure you want to change the base?

fix: unify ColumnNotFound for duckdb and pyspark #2493

Conversation

EdAbati commented May 4, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

EdAbati commented May 4, 2025

EdAbati commented May 5, 2025 • edited Loading

MarcoGorelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdAbati May 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdAbati May 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdAbati May 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fix: unify `ColumnNotFound` for `duckdb` and `pyspark` #2493

fix: unify `ColumnNotFound` for `duckdb` and `pyspark` #2493

EdAbati commented May 5, 2025 •

edited

Loading

EdAbati May 5, 2025 •

edited

Loading

EdAbati May 9, 2025 •

edited

Loading

EdAbati May 9, 2025 •

edited

Loading