Apply python_scan_all_frames to externally supplied duckdb connections#2790
Merged
Merged
Conversation
PR #2527 upgraded duckdb to >=1.1.0 and restored the pre-1.1.0 `python_scan_all_frames` behavior by passing it via connect-time `config`. That config only reaches connections Soda creates itself (the file-read and `:memory:`/path branches); it never reaches the `duckdb_connection` branch used by `Scan.add_duckdb_connection(...)` -- the canonical pandas/polars workflow. Because duckdb >=1.1.0 defaults `python_scan_all_frames` to False, a user-supplied connection can no longer resolve a DataFrame defined in a caller frame, and the check fails with "Table ... does not exist". The repo's existing `test_pandas_df` did not catch this: the shared fixture pre-registers its own connection (built through the config path, which already sets the flag), so a fixture-based scan never exercises the externally-supplied-connection branch. Fix: - Restore `python_scan_all_frames` at runtime (via SET) on externally supplied connections, driven by `self.configuration` so overrides still win. (custom_user_agent can only be set at connect time, so it cannot be applied to a pre-existing connection.) - Let callers override the applied settings through a new optional `configuration` argument on `add_duckdb_connection`, reusing the existing data-source `configuration` plumbing. Default restores the old behavior; pass `configuration={"python_scan_all_frames": False}` to opt out. - Add regression tests that exercise the real user-connection branch (bare Scan, not the fixture): one guards the restored default, one verifies the override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Niels-b
force-pushed
the
fix/duckdb-scan-all-frames-external-connection
branch
from
July 14, 2026 07:55
1f2ded8 to
e453794
Compare
|
m1n0
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2527. That PR restored the pre-1.1.0
python_scan_all_framesbehavior by passing it via connect-timeconfig, but the config only reaches connections Soda creates itself (the file-read and:memory:/path branches). It never reaches theduckdb_connectionbranch used byScan.add_duckdb_connection(...)— the canonical pandas/polars DataFrame workflow.Since duckdb ≥1.1.0 defaults
python_scan_all_framestoFalse, a user-supplied connection can no longer resolve a DataFrame defined in a caller frame, and the check fails withTable ... does not exist.Why the existing test didn't catch it
test_pandas_dfuses the shared fixture, which pre-registers its own duckdb connection (built through the config path, which already sets the flag).DataSourceManager.get_data_sourcereturns that cached instance, soadd_duckdb_connection(con)writes to a dict that's never read — the branch is never exercised, and the bug is masked.Changes
duckdb_data_source.py— restorepython_scan_all_framesat runtime (viaSET) on externally supplied connections, driven byself.configurationso overrides still win. (custom_user_agentcan only be set at connect time, so it can't be applied to a pre-existing connection.)scan.py/configuration.py— new optionalconfigurationargument onadd_duckdb_connection, reusing the existing data-sourceconfigurationplumbing. Default restores the old behavior; passconfiguration={"python_scan_all_frames": False}to opt out.test_duckdb.py— two regression tests that exercise the real user-connection branch (bareScan, not the fixture): one guards the restored default (fails without this fix), one verifies the override is honored.Behavior
add_duckdb_connection(con)(default)Trueadd_duckdb_connection(con, configuration={"python_scan_all_frames": False})Falseadd_duckdb_connection(con, configuration={"python_scan_all_frames": True})TrueTesting
Verified against a dedicated venv with
duckdb==1.5.4(resolved from the>=1.1.0,<2.0range). Full duckdb suite: 6 passed. The restored-default test fails without the source fix and passes with it.🤖 Generated with Claude Code