Support Adding Hive-Partitioned

[xevix]

ducklake_add_data_files supports adding parquet files, but if a table has used ALTER TABLE .. SET PARTITIONED BY to specify partitions, using ducklake_add_data_files will not pull in the partitions from the path.

I believe this is a known issue but wanted to formally bring it up since I would love to be able to query existing open datasets on S3, but using ducklake as a friendly wrapper for querying.

e.g.

-- Attach to S3
ATTACH 'ducklake:~/noaa.ducklake' AS noaa (DATA_PATH 's3://noaa-ghcn-pds/', METADATA_SCHEMA 'parquet');

-- Create a table with the same structure but no data (metadata only)
CREATE TABLE noaa.by_year AS FROM 's3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=TMAX/*.parquet' LIMIT 0;
-- Set the partitioning (which won't touch the existing data dir)
ALTER TABLE noaa.by_year SET PARTITIONED BY (year, element);

-- Make DuckLake aware of some or all partitions, reading out the YEAR and ELEMENT columns from the path
CALL ducklake_add_data_files('noaa', 'by_year', 's3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=TMAX/*.parquet');

-- Now query data as usual
FROM noaa.by_year WHERE year = 2024 AND element = 'TMAX';

[gahtan-syarif]

There's already a PR for adding hive partitioned data files: #252

[xevix]

Thanks! I searched through discussions and issues but missed PRs. Great to see it’s on the way.