Support window operations on Structured Streaming data sources

[edurdevic]

At the moment, Spark does not support window functions (like lag or head) on a stream.

It would be great if you could add support for those type of operation.

[tnixon]

Thanks for the suggestion, @edurdevic. There are some windowing functions that are supported on streams, see:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows

However, we are limited in what we can provide without fundamentally modifying the core of the Spark streaming engine.

We are looking to add streaming support for some Tempo operations including as-of-joins and resampling. @Sonali-guleria can comment further on our efforts in this area and whether operations like lag or head are possible.