Merge branch 'main' of https://github.com/TobikoData/sqlmesh into sung/init-refresh

Author: sungchun12

docs/integrations/engines/duckdb.md

@@ -18,6 +18,7 @@
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                                                                                                                                                      |  list  |    N     |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                                                                                                                                                                |  dict  |    N     |
 | `secrets`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
+| `filesystems`   | Configuration for registering `fsspec` filesystems to the DuckDB connection.                                                                                                                                                                                                |  dict  |    N     |
 
 #### DuckDB Catalogs Example
 
@@ -256,4 +257,36 @@ After configuring the secrets, you can directly reference S3 paths in your catal
 
 Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html).
 
-> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+
+##### File system configuration example for Microsoft Onelake
+
+The `filesystems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet).
+
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    gateways:
+      ducklake:
+        connection:
+          type: duckdb
+          catalogs:
+            ducklake:
+            type: ducklake
+            path: myducklakecatalog.duckdb
+            data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files
+        extensions:
+          - ducklake
+        filesystems:
+          - fs: abfs
+            account_name: onelake
+            account_host: onelake.blob.fabric.microsoft.com
+            client_id: {{ env_var('AZURE_CLIENT_ID') }}
+            client_secret: {{ env_var('AZURE_CLIENT_SECRET') }}
+            tenant_id: {{ env_var('AZURE_TENANT_ID') }}
+            # anon: False # To use azure.identity.DefaultAzureCredential authentication 
+    ```
+
+
+Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options. 

docs/integrations/overview.md

@@ -9,20 +9,20 @@ SQLMesh supports integrations with the following tools:
 * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli)
 
 ## Execution engines
-SQLMesh supports the following execution engines for running SQLMesh projects:
+SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses - example usage: `pip install "sqlmesh[databricks]"`):
 
-* [Athena](./engines/athena.md)
-* [Azure SQL](./engines/azuresql.md)
-* [BigQuery](./engines/bigquery.md)
-* [ClickHouse](./engines/clickhouse.md)
-* [Databricks](./engines/databricks.md)
-* [DuckDB](./engines/duckdb.md)
-* [MotherDuck](./engines/motherduck.md)
-* [MSSQL](./engines/mssql.md)
-* [MySQL](./engines/mysql.md)
-* [Postgres](./engines/postgres.md)
-* [GCP Postgres](./engines/gcp-postgres.md)
-* [Redshift](./engines/redshift.md)
-* [Snowflake](./engines/snowflake.md)
-* [Spark](./engines/spark.md)
-* [Trino](./engines/trino.md)
+* [Athena](./engines/athena.md) (athena)
+* [Azure SQL](./engines/azuresql.md) (azuresql)
+* [BigQuery](./engines/bigquery.md) (bigquery)
+* [ClickHouse](./engines/clickhouse.md) (clickhouse)
+* [Databricks](./engines/databricks.md) (databricks)
+* [DuckDB](./engines/duckdb.md) (duckdb)
+* [MotherDuck](./engines/motherduck.md) (motherduck)
+* [MSSQL](./engines/mssql.md) (mssql)
+* [MySQL](./engines/mysql.md) (mysql)
+* [Postgres](./engines/postgres.md) (postgres)
+* [GCP Postgres](./engines/gcp-postgres.md) (gcppostgres)
+* [Redshift](./engines/redshift.md) (redshift)
+* [Snowflake](./engines/snowflake.md) (snowflake)
+* [Spark](./engines/spark.md) (spark)
+* [Trino](./engines/trino.md) (trino)