completeness of implementation?

[adriangb]

hi @suremarc, I was wondering if the implementation here is complete? I was trying to grok it and came across e.g. https://github.com/datafusion-contrib/datafusion-materialized-views/blob/main/src/rewrite.rs which seems to be an empty file. I wanted to confirm what the state of the repo is before I break my brain. Thanks for your work on this!

[suremarc]

Hey @adriangb, so far the materialized module is pretty much complete, which is the piece related to incrementally maintaining materialized views. The rewrite module is for "query rewriting" which is more about exploiting materialized views to speed up queries. Everything in the materialized module is nearly ready, I'm cooking up an integration test which should serve dually as an example of how to use it, but I'm still ironing out some small bugs.

After finishing up the integration test I plan to port the query rewriting code to that empty file you mentioned.

[alamb]

BTW saw a great talk from @JanKaul about materialized views in DataFusion at

• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 apache/datafusion#12988

He even had some nice relational algebra (thanks to @jakthom for the pictures)

It was not 100% clear to me where that implementation was but I wanted to point it out in this repo as a source of potential collaboration

https://github.com/JanKaul/frostbow

[JanKaul]

The implementation is here:

https://github.com/JanKaul/iceberg-rust/tree/main/datafusion_iceberg%2Fsrc%2Fmaterialized_view

I have to update the documentation and upload the slides somewhere.

[alamb]

The implementation is here:

https://github.com/JanKaul/iceberg-rust/tree/main/datafusion_iceberg%2Fsrc%2Fmaterialized_view

I have to update the documentation and upload the slides somewhere.

Perhaps you can post the slides to

• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 apache/datafusion#12988

[suremarc]

Hey @alamb @JanKaul, sorry I've been traveling the last two weeks and I didn't see your comments. (I can't seem to figure out how to get notifications for this repo 😞)

I think collaboration is a great idea. I would note that I think the materialized view implementations differ in some key ways:

• The iceberg materialized view implementation does fully incremental changes, i.e. computes exact deltas (where feasible), while the implementation here relies on recomputing and overwriting partitions, never insert/delete.
• The iceberg implementation is specific to iceberg (though maybe it could be generalized, and I assume the relational algebra bits are reusable), while the implementation here is for ListingTable or ListingTable-like tables.

I think it would be interesting if we could house both IVM implementations and put them behind a unified interface, then hook it up so that users could refresh their tables through SQL commands. This would provide a great out-of-the-box experience for materialized views in DataFusion, and it would let us use the query rewriting implementation for any kind of materialized view.

Another thing we could try to do is abstract the storage layer, and have each IVM implementation list which storage operations it needs (insert, delete, overwrite partition, full overwrite, read from snapshot/time travel?). Then each IVM implementation can work regardless of the storage backend, as long as their respective operations are supported.