[SPARK-54022][SPARK-56617][SQL][TESTS] Add more CACHE TABLE tests and reorganize CACHE TABLE tests#55536
Open
longvu-db wants to merge 9 commits intoapache:masterfrom
Open
[SPARK-54022][SPARK-56617][SQL][TESTS] Add more CACHE TABLE tests and reorganize CACHE TABLE tests#55536longvu-db wants to merge 9 commits intoapache:masterfrom
longvu-db wants to merge 9 commits intoapache:masterfrom
Conversation
…reorganize cache tests Move existing cache-pinning tests to the bottom of DataSourceV2DataFrameSuite and add three new tests covering design doc Section [5] scenarios: - Scenario 3: external schema change is invisible to cached table - Scenario 4: session schema change invalidates and rebuilds cache - Scenario 5: external drop/recreate produces new table ID, query sees empty table Co-authored-by: Isaac
longvu-db
force-pushed
the
spark-dsv2-cache-scenario-5
branch
from
174cefd to
75db50d
Compare
Co-authored-by: Isaac
- Combine Scenarios 1+2 into single test following design doc flow - Remove separate "variant" test - Scenario 4: add external write after session schema change - Scenario 5: remove ID references from comments Co-authored-by: Isaac
- Restore all 3 original cache tests (moved to bottom) - New Scenario 1+2 test uses InMemoryBaseTable.withData() to simulate true external INSERT bypassing session CacheManager - Scenario 4 also uses withData() for external write simulation Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
longvu-db
changed the title
longvu-db
changed the title
longvu-db
changed the title
- Use loadTable with INSERT privilege to get original table (not copy) for withData calls, fixing copyOnLoad interaction - Split Scenarios 1+2 into separate tests to avoid data accumulation - All 8 cache tests pass locally Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Reorganize and expand the CACHE TABLE test coverage in
DataSourceV2DataFrameSuite.Moved tests: Three existing cache-pinning tests are relocated to the bottom of the suite, grouped under a section comment for discoverability:
"cached DSv2 table DataFrame is refreshed and reused after insert"(Scenario 2)"caching table via Dataset API should pin table state"(Scenario 1+2)"caching a query via Dataset API should not pin table state"New tests:
"cached table pinned against external schema change"): External ADD COLUMN via catalog API is invisible to the cached table."session schema change invalidates cache"): SessionALTER TABLE ADD COLUMNinvalidates and rebuilds cache with the new 3-column schema."cached table after external drop and recreate sees empty table"): External drop+recreate via catalog API produces a new table with a different ID; query sees the new empty table.Why are the changes needed?
The existing tests covered scenarios 1 and 2 (external data write pinning and session write invalidation), but did not cover:
These scenarios are important to verify the correctness of DSv2 cache behavior as described in the design doc.
Does this PR introduce any user-facing change?
No. This PR only adds and reorganizes tests.
How was this patch tested?
New and moved tests in
DataSourceV2DataFrameSuite.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-6)