SQL API for other query engines to read/write duck lake

[flxliu]

I'm proposing a new set of SQL-based APIs for DuckLake — similar in spirit to delta-kernel, but expressed in SQL — to make it easier for other query engines (e.g., Spark) and tools to interoperate with DuckLake.

For example, on the write path, we could provide an SQL API that allows external engines to write Parquet files and then commit them to DuckLake, similar to Iceberg’s add_files:

SELECT ducklake_append_data_files('table_name', ARRAY['file1.parquet', 'file2.parquet']);

On the read path, we could expose an SQL API that returns the list of Parquet files to scan, based on the current query predicates.

By building these capabilities as SQL functions, any programming language that duckdb supports, could integrate with DuckLake easily - import duckdb, load ducklake extension, then call a SQL.