[SPARK-52729][SQL] Add MetadataOnlyTable and CREATE/ALTER VIEW support for DS v2 catalogs

[cloud-fan]

What changes were proposed in this pull request?

This PR exposes a DS v2 API for metadata-only tables (read side), CREATE VIEW, and ALTER VIEW ... AS (write side) so that third-party v2 catalogs can participate in Spark's resolution and creation flows without reimplementing read/write themselves.

1. Read path — MetadataOnlyTable:

• New Table implementation that carries a TableInfo and delegates everything to it. Catalogs return it from loadTable to signal "Spark, interpret this via V1 paths" — data-source reads for file-source tables, view-text expansion for views (the latter via the perf opt-in described in section 4).
• Analyzer.lookupTableOrView and RelationResolution.createRelation detect MetadataOnlyTable and route through a new V1Table.toCatalogTable adapter to the existing V1 data-source / view machinery.

2. Shared DTO — TableInfo:

• TableInfo.Builder gains convenience setters that write reserved keys into properties: withProvider, withLocation, withComment, withCollation, withOwner, withTableType, plus withSchema(StructType). The read side (MetadataOnlyTable) and the write side (createTable(ident, TableInfo)) use the same struct. View-specific fields live on a typed subclass (see section 3) so they are not encoded as string properties.
• withProperties takes a defensive copy so convenience setters don't mutate the caller's map.

3. Typed view DTO — ViewInfo:

• ViewInfo extends TableInfo carries the view-specific fields that cannot be represented as string table properties: queryText, currentCatalog, currentNamespace (multi-part, never null; empty when no namespace was captured), sqlConfigs (unprefixed SQL config keys), schemaMode (BINDING / COMPENSATION / TYPE EVOLUTION / EVOLUTION), and queryColumnNames (mapping query output to the view's declared columns; empty in EVOLUTION mode).
• ViewInfo.Builder extends TableInfo.BaseBuilder<Builder> adds typed setters: withQueryText, withCurrentCatalog, withCurrentNamespace, withSqlConfigs, withSchemaMode, withQueryColumnNames. The inherited TableInfo.BaseBuilder setters (schema, properties, owner, comment, collation, etc.) are available on the same builder so view and table writes share one fluent API.
• The ViewInfo constructor stamps PROP_TABLE_TYPE = TableSummary.VIEW_TABLE_TYPE into properties() so catalogs and generic viewers reading PROP_TABLE_TYPE from the properties bag (e.g. TableCatalog.listTableSummaries default impl, DESCRIBE) classify the entry as VIEW without requiring authors to remember withTableType(VIEW).
• ViewInfo is the typed payload returned by ViewCatalog.loadView and accepted by createView / replaceView. It still extends TableInfo so a mixed catalog can opt into the perf path described in section 4 (returning MetadataOnlyTable(ViewInfo) from loadTable); pure view-only catalogs never see TableInfo directly because the typed builder covers everything they construct.

4. View support — ViewCatalog interface:

• A new ViewCatalog interface is the plugin-facing API for views. It is independent from TableCatalog: a connector implements just ViewCatalog (view-only catalog), just TableCatalog (table-only catalog), or both (mixed catalog like Hive / Iceberg / Unity Catalog). There is no capability flag — interface presence is the signal.
• API: listViews(namespace), loadView(ident) returning ViewInfo, createView(ident, ViewInfo), replaceView(ident, ViewInfo), dropView(ident), default viewExists(ident), default invalidateView(ident). No staging variant — replaceView is a single atomic-swap call.
• Mixed-catalog rule (documented on the class): tables and views share one identifier namespace. Each interface's methods behave as if the other kind didn't exist; the only cross-cutting invariant is that createTable rejects view-collisions and createView rejects table-collisions (one extra existence check the catalog already needs internally).
• Mixed-catalog perf opt-in: loadTable may return a MetadataOnlyTable wrapping a ViewInfo for a view identifier; Spark's resolver discriminates by instanceof ViewInfo and routes through view resolution without a follow-up loadView RPC. If the catalog instead throws NoSuchTableException for a view identifier, Spark falls back to loadView (one extra RPC on cold cache).
• Analyzer.lookupTableOrView (and RelationResolution.tryResolvePersistent): tries loadTable first only when the catalog is a TableCatalog (or the session catalog) — otherwise the underlying asTableCatalog cast would throw MISSING_CATALOG_ABILITY.TABLES for a pure ViewCatalog and mask the legitimate loadView fallback. On NoSuchTableException (or when loadTable is skipped), if the catalog is a ViewCatalog, calls loadView and synthesizes a ResolvedPersistentView from the resulting ViewInfo via V1Table.toCatalogTable(catalog, ident, viewInfo).
• Catalogs that are not ViewCatalog get MISSING_CATALOG_ABILITY.VIEWS from the resolver gate (for UnresolvedView) and from CheckViewReferences (for CREATE / ALTER VIEW), matching the previous capability-flag rejection.

5. Write path — DS v2 CREATE VIEW:

• DataSourceV2Strategy routes CreateView(ResolvedIdentifier(catalog, ident), …) to CreateV2ViewExec(catalog: ViewCatalog, …), which dispatches: createView for plain CREATE / IF NOT EXISTS; replaceView for CREATE OR REPLACE on an existing view (with a NoSuchViewException → createView fallback for the race where the view disappears between probe and replace); createView for CREATE OR REPLACE on a non-existent view. Cross-type collision (CREATE VIEW over a non-view table in a mixed catalog) is rejected up front with EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE.
• The exec builds the ViewInfo via a V2ViewPreparation trait reusing v1 ViewHelper helpers (aliasPlan, sqlConfigsToProps) to populate a ViewInfo.Builder with the current session's captured catalog/namespace and SQL configs. Cyclic-reference detection and auto-generated-alias rejection run once at analysis time in CheckViewReferences (see section 7).
• CreateView logical plan extends AnalysisOnlyCommand (same shape as V2CreateTableAsSelectPlan) so HandleSpecialCommand.markAsAnalyzed captures referredTempFunctions from AnalysisContext. The v1 rewriting path (ResolveSessionCatalog → CreateViewCommand) is unchanged.

6. Write path — DS v2 ALTER VIEW ... AS:

• AlterViewAs logical plan also extends AnalysisOnlyCommand so referredTempFunctions is captured for the non-session path.
• DataSourceV2Strategy routes AlterViewAs(ResolvedPersistentView(catalog, ident, _), …) to AlterV2ViewExec(catalog: ViewCatalog, …), which calls replaceView (the single atomic-swap entry point — no separate staging variant, since view REPLACE writes only metadata).
• A V2AlterViewPreparation trait (extends V2ViewPreparation) calls catalog.loadView(ident) once and uses the result to preserve user TBLPROPERTIES, comment, collation, owner, and schema-binding mode when constructing the replacement ViewInfo. Session-scoped fields (SQL configs, query column names) are re-emitted by buildViewInfo() from the active SparkSession, matching v1 AlterViewAsCommand.alterPermanentView. A racing DDL between analysis and exec (the view dropped, or replaced with a non-view table in a mixed catalog) surfaces NoSuchViewException / EXPECT_VIEW_NOT_TABLE rather than a stale-resolution error.
• ResolvedViewIdentifier.unapply (in ResolveSessionCatalog) replaces its assert(isSessionCatalog) with an if isSessionCatalog guard so non-session ResolvedPersistentView plans fall through to the v2 strategy instead of tripping the assertion.

7. Post-analysis check — CheckViewReferences:

• New rule wired into BaseSessionStateBuilder.extendedCheckRules. Rejects permanent views that reference temporary objects and rejects view bodies with auto-generated aliases for both CreateView and AlterViewAs (v2 paths). v1 CreateViewCommand / AlterViewAsCommand keep their existing exec-time safety net — Dataset-built commands can be constructed with isAnalyzed=true directly and bypass the analyzer's re-capture path.

8. Listing — SHOW TABLES / SHOW VIEWS:

• TableCatalog.listTables returns table identifiers only — views (if the catalog also implements ViewCatalog) are listed separately via ViewCatalog.listViews. listTableSummaries's default impl enumerates via listTables + loadTable and returns one summary per table. This is an intentional v2 divergence from v1 SHOW TABLES, which includes both tables and views; restoring the v1-parity output for SHOW TABLES on a v2 catalog (i.e. routing it through both listTables and listViews) is left as a follow-up so this PR's API surface stays narrowly scoped.
• SHOW VIEWS on a non-session ViewCatalog is routed through a new ShowViewsExec that enumerates via ViewCatalog.listViews(namespace). ResolveSessionCatalog.ShowViews skips (via guard) for ViewCatalog catalogs so they fall through to this strategy; non-session, non-ViewCatalog catalogs still hit the existing MISSING_CATALOG_ABILITY.VIEWS rejection. v2 catalogs have no temp views, so the isTemporary column is always false (mirroring v1, which only sets it true for local/global temp views).

Why are the changes needed?

A v2 Table is not always backed by a connector that implements read/write. Catalogs like HMS and Unity Catalog store only metadata and rely on Spark to interpret the table provider as a data source or to execute the view SQL. Previously the only way to achieve that was a hack around V1Table, which leaks private v1 types into v2 connectors (example: https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala).

Separately, v2 catalogs had no public way to handle CREATE VIEW or ALTER VIEW. ResolveSessionCatalog rejected CREATE VIEW on any non-session catalog with MISSING_CATALOG_ABILITY.VIEWS, so third-party catalogs could not own view lifecycle at all. The new ViewCatalog interface gives catalogs a clean view-shaped API (listViews / loadView / createView / replaceView / dropView) that is independent of TableCatalog: a view-only catalog implements just ViewCatalog (no TableCatalog boilerplate), a mixed catalog implements both, and the cross-type collision invariant is one extra existence check at createTable / createView time.

Does this PR introduce any user-facing change?

Yes to connector developers:

• Third-party v2 TableCatalog implementations can now return a MetadataOnlyTable from loadTable to delegate reads to Spark.
• Third-party v2 connectors can implement the new ViewCatalog interface to handle CREATE VIEW / CREATE OR REPLACE VIEW / CREATE VIEW IF NOT EXISTS / ALTER VIEW … AS / DROP VIEW / SHOW VIEWS, with view text, schema, captured current catalog+namespace, SQL configs, and temp-object-reference rejection handled the same way as for session-catalog views. View-only catalogs implement just ViewCatalog; mixed catalogs implement both TableCatalog and ViewCatalog.

No SQL-level or user-visible behavior change for existing deployments.

Remaining work (follow-up PRs)

This PR covers the core read path, CREATE VIEW (all shapes), ALTER VIEW ... AS, DROP VIEW, and SHOW VIEWS. The following view-scoped plans for DS v2 catalogs are not yet supported and are tracked for follow-ups. Until the follow-ups land, each currently surfaces a clean UNSUPPORTED_FEATURE.TABLE_OPERATION error (wired up in DataSourceV2Strategy and pinned by tests in DataSourceV2MetadataOnlyViewSuite), so users get a meaningful message rather than a generic planner failure:

• ALTER VIEW ... SET/UNSET TBLPROPERTIES — separate logical plans (SetViewProperties, UnsetViewProperties); need their own DataSourceV2Strategy cases backed by new TableChange routing.
• ALTER VIEW ... RENAME TO — RenameTable at the logical level; needs v2 view awareness and the catalog-side rename semantics.
• ALTER VIEW ... WITH SCHEMA BINDING — AlterViewSchemaBinding logical plan; needs the same treatment as ALTER VIEW AS (AnalysisOnlyCommand shape + v2 exec).
• DESCRIBE / SHOW CREATE TABLE / SHOW TBLPROPERTIES / SHOW COLUMNS on v2 views — currently route through ResolvedViewIdentifier which only matches session-catalog views; the v2 equivalents need dedicated handling.
• SHOW TABLES v1-parity output (include views) on a v2 catalog — TableCatalog.listTables now intentionally returns tables only; restoring v1-parity in the SQL layer (route SHOW TABLES through both listTables and listViews) is a separate piece of work.

How was this patch tested?

New DataSourceV2MetadataOnlyTableSuite covering:

• Reads: Metadata-only file-source reads (SELECT / INSERT / INSERT OVERWRITE, partitioned and non-partitioned); views with fully-qualified references + view SQL configs; views with unqualified references that expand via the stored current catalog+namespace.
• CREATE VIEW (plain TableCatalog): end-to-end CREATE, CREATE VIEW IF NOT EXISTS (no-op on existing), CREATE on existing (failure), CREATE OR REPLACE VIEW (replacement); user-specified columns (too-few / too-many); DEFAULT COLLATION propagation into ViewInfo.
• Pure-ViewCatalog end-to-end (no TableCatalog mixin): dedicated TestingViewOnlyCatalog fixture exercises view-text expansion on read + ALTER VIEW … AS against a pure ViewCatalog, ensuring the resolver's loadView fallback fires correctly when loadTable is skipped.
• CREATE VIEW over a non-view table (v1-parity): rejects CREATE OR REPLACE VIEW against a non-view table entry with EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE; rejects plain CREATE VIEW against a non-view table entry with TABLE_OR_VIEW_ALREADY_EXISTS; CREATE VIEW IF NOT EXISTS over a table is a no-op (matches v1 SQLViewSuite "existing a table with the duplicate name when CREATE VIEW IF NOT EXISTS").
• CREATE / ALTER / DROP / SHOW VIEW on a catalog that doesn't implement ViewCatalog: a dedicated TestingTableOnlyCatalog exercises the rejection on each path (expected MISSING_CATALOG_ABILITY.VIEWS).
• CheckViewReferences: permanent view referencing a temp function rejected; permanent view referencing a temp view rejected.
• ALTER VIEW ... AS: end-to-end body replacement; rejects temp-function reference; preserves user-set TBLPROPERTIES; preserves PROP_OWNER and SCHEMA EVOLUTION binding mode across the ALTER (v1-parity); missing view surfaces as AnalysisException.
• Multi-part captured namespace round-trip: unit-level test that ViewInfo.Builder.withCurrentCatalog(cat).withCurrentNamespace([db1, db2]) -> V1Table.toCatalogTable -> CatalogTable.viewCatalogAndNamespace preserves the full multi-part form, including namespace parts containing dots (which flow through structurally, not via any string encoding). A companion test pins the absent-branch: a ViewInfo built without withCurrentCatalog yields an empty viewCatalogAndNamespace.
• Unsupported v2 view DDL / inspection pinning: SET TBLPROPERTIES, UNSET TBLPROPERTIES, WITH SCHEMA, RENAME TO, SHOW CREATE TABLE, SHOW TBLPROPERTIES, SHOW COLUMNS, DESCRIBE TABLE against a v2 view all surface UNSUPPORTED_FEATURE.TABLE_OPERATION; DESCRIBE TABLE ... COLUMN surfaces a clean AnalysisException. Pins the current failure mode so a future regression to a generic planner error is caught in the diff.
• SHOW TABLES / SHOW VIEWS on a v2 catalog: SHOW TABLES returns tables only; SHOW VIEWS returns views only (isTemporary=false throughout for v2 catalogs); SHOW VIEWS ... LIKE filters on the view name; SHOW VIEWS against a non-ViewCatalog is rejected with MISSING_CATALOG_ABILITY.VIEWS.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

[cloud-fan]

I think the current view implementation which stores the original SQL text and a bunch of context is too convoluted to put into the public DS v2 API. It's better if the view text is context-independent. We are going to improve it in #51410

Before the improvement is done, we only allow to read DS v2 views that has context-independent SQL text.

[aokolnychyi]

Don't we store the current catalog and namespace in the view metadata? Do we expect the connectors to modify the view SQL text?

[cloud-fan]

My proposal is to let Spark modify the view text before saving it into the catalog, so that the catalog does not need to store the current catalog/namespace.

[aokolnychyi]

So all identifiers in the view text will always include the catalog name as the first name part and the table name as the last name part? How hard will it be to modify the original SQL text? Will it cause any surprises to the users if the original and the persisted SQL text differ?

[aokolnychyi]

So all identifiers in the view text will always include the catalog name as the first name part and the table name as the last name part? How hard will it be to modify the original SQL text? Will it cause any surprises to the users if the original and the persisted SQL text differ?

[cloud-fan]

Hive already did so and its view has both "view_text" and "original_view_text" fields. All identifiers should be fully qualified (with catalog name and namespace) in the view text.

[cloud-fan]

cc @aokolnychyi @gengliangwang

[gengliangwang]

how about SPARK_TABLE_OR_VIEW

[gengliangwang]

shall we explicitly mention what operations will be affected?(read/write/DDL/...)

[cloud-fan]

The capability has since been reshaped: TableCapability is gone in favor of the concrete MetadataOnlyTable for the read side, and a TableCatalogCapability.SUPPORTS_VIEW gate for the write side. The SUPPORTS_VIEW javadoc now spells out the affected operations explicitly (CREATE VIEW / CREATE OR REPLACE VIEW / CREATE VIEW IF NOT EXISTS via createTable, ALTER VIEW ... AS via dropTable+createTable or stageReplace, and the read-path round-trip through MetadataOnlyTable). PTAL and let me know if anything is still unclear.

[aokolnychyi]

I'd love to take a look on Monday.

[aokolnychyi]

I understand we may want to use the new API to expose views, but what about the case with Spark table? When would this be helpful?

[cloud-fan]

For example, UC does not want to rely on the Spark file source, but just set the table provider to the file source name and leave read/write to Spark. Today this is done by a hack with V1Table: https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala#L303

[aokolnychyi]

Ah, got it. The use case is catalogs that govern tables but not necessarily implement read/write logic.

Have we considered offering a generic V2 table implementation for built-in formats that would be accessible to external connectors? Essentially, a public version of V1Table that doesn't need to expose CatalogTable. If we go with the table capability approach, then each connector will have to implement a custom V2 table for it to be simply replaced as a Parquet table or view. Each connector would have to be aware of how the translation will be done in Spark. For instance, it seems like we assume that serdeProps will be with prefixed with option..

Just curious to know your thinking, I don't have a strong opinion here.

[cloud-fan]

I see, this makes sense. A concrete class is easier to use than a table capability.

[gengliangwang]

Shall we add this test case later?

[cloud-fan]

This will be fixed shortly after the PR is merged.

[gengliangwang]

This affects the table read/write operations but not DDL operations.

It seems a bit unclear. Before the change, DDL operations of DSV2 tables relies on Spark too.

[cloud-fan]

maybe we should just remove this line? DDL operations do not call read/write APIs of Table anyway.

[szehon-ho]

Makes sense, just wondering if some of it can be less bound to HMS concepts

[szehon-ho]

Just an opinion, serde is quite bound to HMS (?) and the newer DSV2 Catalog dont have that, will it make sense to abstract this (maybe something like 'storageProperties')? This is just for HMS table support?

[cloud-fan]

good point. I was trying to save the work of splitting serde properties from table properties, but API simplicity is more important.

[szehon-ho]

curious, do we not need to set inputFormat/outputFormat?

[cloud-fan]

V1Table itself does not access the input/output format when translating v1 table to v2 table. I think Hive table is never supported by DS v2 and we will not handle it here.

[szehon-ho]

Not entirely familiar with the end use case, but the API looks good now, thanks

[dongjoon-hyun]

Hi, @cloud-fan . Do you have a plan to proceed this forward?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[gengliangwang]

Re-review of the updated PR.

[gengliangwang]

The comment above says non-session views "fall through so they can be picked up by v2 strategies." Only AlterViewAs is actually picked up. ResolvedViewIdentifier is also used by SetViewProperties (line 176), UnsetViewProperties (line 179), AlterViewSchemaBinding (line 520), RenameTable-on-view (line 194), and DescribeRelation-on-view (line 198-199) — none of which have a v2 strategy case. After this PR a user can CREATE VIEW view_catalog.ns.v ... and then ALTER VIEW view_catalog.ns.v SET TBLPROPERTIES('k'='v') produces a generic planner "no plan for..." failure. Pre-PR, CreateView was rejected on non-session catalogs, so this orphan state was structurally unreachable; the pre-PR error path (MISSING_CATALOG_ABILITY.VIEWS) is no longer.

There is no test coverage for any of these five plans on a v2 view: DataSourceV2MetadataOnlyViewSuite has no test for SET TBLPROPERTIES, UNSET TBLPROPERTIES, WITH SCHEMA BINDING, RENAME TO, DESCRIBE, SHOW TBLPROPERTIES, or SHOW CREATE VIEW against a v2-catalog view (the only WITH SCHEMA EVOLUTION reference is on the CREATE VIEW path, line 462). The TODO at lines 36-37 of that suite acknowledges these as follow-ups, but nothing pins the current failure mode, so any future change (e.g., a reshape of planner error classes) can silently regress the UX further.

Two options:

1. Pin the current behavior — for each of the five plan types, add a test that runs the statement against a v2 view and asserts the error it throws today. Future changes then surface in the diff.
2. Close the gap up front — add explicit DataSourceV2Strategy cases (or a fall-through in ResolveSessionCatalog) that throw a clean UNSUPPORTED_FEATURE / FEATURE_NOT_YET_SUPPORTED naming the statement. Tests become one-per-plan and the UX doesn't regress between this PR and the follow-ups.

Option 2 is the cleaner closure given this PR is already landing the architectural change that enables the orphaning. Option 1 is the minimum safety net.

[gengliangwang]

viewText is read and assigned unconditionally. If a catalog returns a MetadataOnlyTable with PROP_VIEW_TEXT set but PROP_TABLE_TYPE is EXTERNAL/MANAGED (misconfiguration, or a future capability-composition catalog), the synthesized CatalogTable ends up with non-None viewText on a non-view — confusing downstream code that uses viewText.isDefined as an "is-view" proxy. Scoping the read to VIEW costs nothing:

Suggested change

	viewText = viewText,
	val viewText = if (tableType == CatalogTableType.VIEW) {
	props.get(TableCatalog.PROP_VIEW_TEXT)
	} else {
	None
	}

[gengliangwang]

Nice cleanup on the CheckViewReferences side with the new catalogAndIdent helper. The same case tc: TableCatalog if tc.capabilities().contains(TableCatalogCapability.SUPPORTS_VIEW) => tc; case _ => throw missingCatalogViewsAbilityError(catalog) pattern is still duplicated at lines 310-314 (CREATE VIEW) and 330-334 (ALTER VIEW AS). Similarly the TableIdentifier(ident.name, ident.namespace.lastOption, Some(catalog.name)) idiom for error-rendering is repeated in CreateV2ViewExec:60-64 and V1Table.toCatalogTable:153-156. Small helpers — e.g. CatalogV2Util.requireViewSupport(catalog) and an asLegacyTableIdentifier(catalogName) on IdentifierHelper — would eliminate the remaining drift risk. Non-blocking.

[gengliangwang]

TestingTableOnlyCatalog.loadTable always throws NoSuchTableException, so the ALTER fails at view resolution — the capability gate in DataSourceV2Strategy (line 330-333) is never reached. The test body's own comment acknowledges this. As a result the capability-gate rejection on the ALTER path has no real coverage.

Two fix options:

1. Rename to "ALTER VIEW on a missing view fails" — matches what it actually tests.
2. Better: extend TestingTableOnlyCatalog to store a MetadataOnlyTable view so the gate rejection is genuinely exercised. This would also catch regressions if SUPPORTS_VIEW is inadvertently added to the default capability set.

[gengliangwang]

The PR's multi-level-namespace correctness hinges on the QuotingUtils.quoted → parseMultipartIdentifier round-trip and the v1 unqualified-reference expansion both preserving the captured namespace. Today that's covered by (a) a builder-level serialization test (line 83) and (b) cycle-detection tests using ns1.inner.v-style identifiers. But there is no end-to-end read test that exercises the round-trip by actually resolving an unqualified reference inside a view whose captured namespace has >1 part. Add one: .withCurrentCatalogAndNamespace("spark_catalog", Array("db1", "db2")), create a table in that namespace, reference it unqualified in the view body, and check the result.

[gengliangwang]

LGTM