feat: add catalog config to separate directory and REST type catalogs

[jackye1995]

No description provided.

[jackye1995]

@yanghua please take a look if you have time, the updated user experience based on my understanding of the discussion so far (this mainly focuses on the Spark SQL experience without the need to register temp view):

For Lance directories, user is expected to start session with:

spark =
        SparkSession.builder()
            .appName("spark-lance-connector-test")
            .master("local")
            .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceCatalog")
            .config("spark.sql.catalog.lance.type", "dir")
            .config("spark.sql.catalog.lance.path", dbPath)
            .getOrCreate();

which will map the default Spark namespace to contain tables in dbPath, so user can run:

spark.sql("SELECT * FROM t1")

spark.sql("SELECT * FROM default.t1")

Another user experience for Lance directory:

spark =
        SparkSession.builder()
            .appName("spark-lance-connector-test")
            .master("local")
            .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceCatalog")
            .config("spark.sql.catalog.lance.type", "dir")
            .config("spark.sql.catalog.lance.paths.ns1", dbPath1)
            .config("spark.sql.catalog.lance.paths.ns2", dbPath2)
            .getOrCreate();

This will map Spark namespace ns1 to dbPath1, ns2 to dbPath2, so user can run:

spark.sql("SELECT t1.c1, t2.c2 FROM ns1.t1 as t1, ns2.t2 as t2 WHERE t1.c3 = t2.c3")

For REST catalog, user should run something like:

spark =
        SparkSession.builder()
            .appName("spark-lance-connector-test")
            .master("local")
            .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceCatalog")
            .config("spark.sql.catalog.lance.type", "rest")
            .config("spark.sql.catalog.lance.uri", "https://my.lancecatalog.com")
            .getOrCreate();

which will connect to the corresponding endpoint (this is pending the HMS server impl to write unit tests)

[jackye1995]

If we agree with the general direction, I can remove the CreateNamespace part to get the updated structure merged first. And there is another follow up fix needed for the LanceDataSource.extractCatalog which is currently hard-coded to lance and needs to be updated.