Support for a NoSQL DB as a catalog DB

[Maelstrom6]

NoSQL databases like Google Cloud Firestore meet the requirements of being a catalog DB:

• It supports ACID transactions
• It has the idea of primary keys (document ID)

But in addition, it is far more cost effective than SQL databases. Instead of paying $20 per month for a micro instance, you only pay for when you query the database and batch writes should be able to write straight to S3.

Reads would be at the cost of reading the catalog (maybe 2 or 3 documents?) and the S3 bucket. Probably a similar situation for writes depending on how the catalog deals with batch writes vs tiny one-line commits.

There is no server management required. But NoSQL databases also have this cool property of infinite horizonal scaling. So this might allow for hundreds of thousands of duckdb client connections at once?

Of course, these requirements I just got from watching the video and there may have been other additional requirements like needing to perform more complex joins.

However, if this problem could be solved, this would democratize terabyte-scale cloud SQL databases for everyone and completely turn the current state of the market on its head. I think this is a revolutionary idea that would free many penny pinchers, hobbyists and small teams from the cost bump of having to host a DB in the cloud for DuckLake.