Design to lower cloud costs of ducklake (TCO)

[hpvd]

Since cost is always a topic and since ducklake is going for "large scale" with huge amounts of data, cost is even more important and tiny things may have noteable impact.

Ducklake is already a very efficient design.

But maybe we can optimize this further or/and
open optional configurations to calibrate the cost/performance sweet spot for

• your different usescases and
• your type of S3 you are using (standard, glacier, express... which have different cost characteristics)

Where do cost have their origin (Cost types)?

• A) running metadatabase
• B) put, get... to S3
• C) S3 storage
• D) ...

Ideas:

1. Adding indexes to reduce query work may lower cost types B and may raise costs of type A, see Adding indexes #389
2. think about batch writing to S3 may lower cost Type B)
3. ...

Which Points/Ideas/approaches do you see?