Support Embeddings

[hpvd]

Embeddings are here to stay.
It may be a good idea to support them one future day in Ducklake too.

fields of usage:

1. If you offload your Data from DB to an lake for long time storage and usage, more and more Datasets contain embeddings and often have stored theses directly in their database (almost all Databases today have support for this or are just getting this build in..).
   With this the question arise, why a lake is not the right place to have these stored too?

2. As discussed, Ducklake is somehow different to other Lakes (speed, efficiency etc) and with this new possible usecases arise.
   see e.g. Adding indexes #389 (reply in thread)
   Many Data uscases today include AI and with this many rely on embeddings...
   To booster these new usecases, the support for embedding may be the extraordinary fuel...

-> What do you think?

Of course we are not the first to think of this "vector lake", but maybe Ducklake is the perfect basement to make this happen

What I hope this enables: open vector data lakes
So what would a real vector data lake look like? Vector embeddings should be stored in Parquet, or Lance, as a native data type. Metadata formats such as Delta Lake or Iceberg should support rich indices to be added by the user, and support versioning on these indexes. Query engines such as Trino and SparkSQL should be able to do nearest neighbour search on the vector data, just like how they are able to filter or join relational data.

https://blog.lancedb.com/why-dataframe-libraries-need-to-understand-vector-embeddings-291343efd5c8/

Vector data lakes combine the scalability of traditional data lakes with the intelligence of vector databases

https://www.linkedin.com/pulse/vector-data-lakes-powering-future-ai-search-jathin-gangi-qqvdc/

This may give some further ideas/background on implementation:

The Best Way to Use Text Embeddings Portably is With Parquet and Polars

https://minimaxir.com/2025/02/embeddings-parquet/

[Alex-Monahan]

Howdy!
DuckLake supports lists and other nested types. What is the functionality that you are interested in? Would a list of floats meet your needs?
https://ducklake.select/docs/stable/specification/data_types#nested-types

[hpvd]

Hi!
because of the importance of embeddings, one could think/discuss about making them a dedicated first class citizen e.g.

• maybe introducing a dedicated vector column (fixed length)
• think about adding approximate index like IVFFlat
• have a look at howto efficiently adding (new version of ) embeddings afterwards
• ...

[hpvd]

if one wants to go even further, one could add even more advanced / efficient indexes like filtered DiskANN
see https://github.com/microsoft/DiskANN

[hpvd]

just to give it some visibility (was already mentioned in first post):

The Best Way to Use Text Embeddings Portably is With Parquet and Polars

https://minimaxir.com/2025/02/embeddings-parquet/