Feature request: Improved automatic schema evolution

[biellls]

Both Snowflake and Databricks schema evolution support automatic schema evolution when loading files into a table with a COPY statement. As far as I can tell in ducklake you need to figure out yourself which columns you need to add, delete, rename or promote and run the ALTER statements yourself. It would be hugely useful to have this automated to support ELT patterns.

[kelf]

Any plans on this?

[djouallah]

this is a very big pain, ideally if you insert new data, the schema should add those columns automatically

[ekkuleivonen]

Yeah, I’ve been running into this from the downstream side too.

I’m playing with a small DuckLake CDC layer here: https://github.com//

It doesn’t do automatic schema evolution, but it tries to make schema changes easier to react to: consumers stop at the schema boundary, inspect the diff, migrate their sink, then continue on the new shape.