[SQL] Support fallback resolver for unknown V2 catalog names

[jarredhj0214]

Problem

We are using Spark 3.3.2 with Apache Gravitino and observed that all top-level V2 catalogs need to be registered eagerly through Spark configuration:

spark.sql.catalog.<catalog_name>=<catalog_class>

This appears to be a general limitation of Spark's V2 catalog lookup model rather than a version-specific bug. I also checked the current catalog loading path in master, and the same static-registration model still applies.

When Spark resolves a multipart identifier such as:

SELECT * FROM catalog_a.db.table

CatalogManager.catalog("catalog_a") eventually calls Catalogs.load("catalog_a", conf). If spark.sql.catalog.catalog_a is not configured, Spark throws CatalogNotFoundException immediately.

Spark supports lazy instantiation for configured catalogs, but it does not currently support lazy discovery for catalog names managed by an external catalog service.

Motivation

External catalog services, such as Apache Gravitino or similar enterprise catalog services, may manage a large number of catalogs. Since Spark requires each top-level catalog name to be configured before analysis, connectors have to eagerly materialize all visible catalog names into Spark configuration during application startup.

This has several drawbacks:

• Spark application startup becomes slower as the number of catalogs grows.
• Catalogs created after Spark application startup are not naturally discoverable.
• Connectors have to preload catalog names only to satisfy Spark's static registration model.
• The actual catalog implementation may still be initialized lazily, but the top-level catalog name must be eagerly registered.

For example, an external catalog service connector may currently need to list visible catalogs at driver initialization time and register entries like:

spark.sql.catalog.catalog_a=...
spark.sql.catalog.catalog_b=...
spark.sql.catalog.catalog_c=...

even if the Spark application only accesses one of them.

Proposal

Introduce an optional fallback resolver for unknown V2 catalog names.

One possible API shape could be:

trait CatalogResolver {
  def resolveCatalog(name: String, conf: SQLConf): Option[CatalogPlugin]
}

Resolvers could be configured statically, for example:

spark.sql.catalog.fallbackResolvers=com.example.ExternalCatalogResolver

Then CatalogManager.catalog(name) could follow this order:

1. Return the cached catalog if it has already been loaded.
2. Try the existing spark.sql.catalog.<name> based loading path.
3. If the catalog is not configured, invoke fallback resolvers in configured order.
4. If a resolver returns a CatalogPlugin, cache it in CatalogManager.
5. If no resolver resolves the catalog, throw the existing CatalogNotFoundException.

The default behavior would remain unchanged when no fallback resolver is configured.

Example

With a fallback resolver configured, a user could query:

SELECT * FROM iceberg_prod.db.table

without predefining:

spark.sql.catalog.iceberg_prod=...

The resolver would receive iceberg_prod, query an external catalog service, construct or load the proper CatalogPlugin, initialize it, and return it to Spark.

Compatibility

This should be fully backward compatible:

• No behavior changes unless fallback resolvers are explicitly configured.
• Existing spark.sql.catalog.<name> configuration keeps priority.
• Existing CatalogNotFoundException behavior remains when no resolver matches.
• Resolved catalogs can follow the same caching and lifecycle behavior as other V2 catalogs.

Open Questions

• Should the resolver return an initialized CatalogPlugin, or return catalog class/options and let Spark initialize it through the existing Catalogs.load path?
• Should resolver configuration live in SQLConf or SparkConf?
• Should multiple resolvers be supported, with the first successful resolver winning?
• Should Spark expose refresh/invalidation for dynamically resolved catalogs, or should the initial proposal only cover discovery?
• Should this be limited to SQL analysis on the driver, or should there be explicit constraints to avoid catalog resolution from executor-side paths?

Summary

Spark already supports lazy instantiation of configured V2 catalogs. This proposal is to add optional lazy discovery for unknown V2 catalog names, which would help integrations with external catalog services avoid eagerly registering every visible catalog at application startup.

[jarredhj0214]

I can also create a SPARK JIRA if that is preferred.