Skip to content

test(core-py): document and pin the physical-table visibility contract#2576

Open
ttw225 wants to merge 2 commits into
Canner:mainfrom
ttw225:test/core-py-register-visibility
Open

test(core-py): document and pin the physical-table visibility contract#2576
ttw225 wants to merge 2 commits into
Canner:mainfrom
ttw225:test/core-py-register-visibility

Conversation

@ttw225

@ttw225 ttw225 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Why

#2495 made apply_wren_on_ctx snapshot top-level catalog membership per
derivation. register_parquet/register_csv write into the pre-existing
default catalog (datafusion.public), whose internals are live-shared
with derived contexts — so tables registered after a session context was
created are still visible through query/dry_run/list_tables. The
DataFusionConnector consumer relies on exactly this, but until now the
behavior was implicit: no documentation stated it and no test guarded it.
#2504 asks for both.

What

  • Document the visibility contract on register_parquet/register_csv
    (where tables land, why late registration is visible, and that brand-new
    top-level catalogs must exist before MDL construction, load_mdl, or a
    transform), plus load_mdl's concurrency contract (its exclusive borrow
    rejects overlapping calls with RuntimeError).
  • Add a README section for registering local files, including the
    tableReference ↔ registered-name mapping requirement, and fix the Quick
    Start sample (it passed a manifest object and a non-existent
    remote_functions kwarg to the constructor).
  • Add tests/test_physical_tables.py: register-before-load_mdl,
    register-after-derivation-without-reload (Python twin of the wren-core
    catalog_race visibility test), and register-then-reload. pyarrow
    joins the dev group to write Parquet fixtures and decode the Arrow IPC
    bytes query() returns, enabling row-level assertions.

No production code changes; no release bump expected.

Note: fixtures declare/store c_custkey as int32 on purpose — writing the
IPC stream with the logical plan's schema corrupts results when MDL types
differ from physical file types. That serialization issue was found by
these tests and is fixed separately in a follow-up fix(core-py) PR.

Test Plan

  • just test-py: 38 passed (35 existing + 3 new), on this branch without
    any production change.
  • just test-rs: 37 passed.
  • ruff check / ruff format --check: clean on the new test file.

Part of #2504.

Summary by CodeRabbit

  • New Features

    • Registered Parquet and CSV tables are now consistently visible to queries, previews, table listings, and semantic model resolution.
    • Tables registered before or after creating a derived context can be used without reloading the session.
  • Documentation

    • Added guidance on registration order, catalog visibility, model loading, and concurrent model-loading calls.
    • Expanded quick-start examples and linked runnable physical-table examples.
  • Tests

    • Added coverage for physical table registration and visibility scenarios.

@ttw225
ttw225 marked this pull request as ready for review July 24, 2026 06:59
@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file python Pull requests that update Python code core labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

SessionContext documentation and examples now define physical table registration and MDL loading behavior. New PyArrow-based tests verify catalog visibility, direct querying, dry runs, and semantic resolution for Parquet and CSV tables.

Changes

Physical table visibility

Layer / File(s) Summary
Registration and MDL contract
core/wren-core-py/README.md, core/wren-core-py/src/context.rs
Examples and API documentation describe default catalog placement, registration timing, MDL loading order, context visibility, and concurrent-load restrictions.
Physical table test setup
core/wren-core-py/pyproject.toml, core/wren-core-py/tests/test_physical_tables.py
Adds the PyArrow development dependency and shared Arrow, manifest, IPC, and Parquet fixtures.
Visibility and semantic resolution tests
core/wren-core-py/tests/test_physical_tables.py
Tests registration before and after context creation, direct query visibility, and semantic resolution after load_mdl.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: goldmedal

Poem

A rabbit hops through tables bright,
Parquet and CSV come into sight.
MDL loads, models align,
Queries bloom in rows divine.
“Binky!” says the hare—“The contract’s right!”

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: documenting and testing the physical-table visibility contract in core-py.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant