Kedro for streaming

[astrojuanlu]

I tried to look for a place where we were collecting ideas about this, but found none.

Early 2021 prototype by @deepyaman https://github.com/deepyaman/kedro-streaming

Have you tried to use Kedro for streaming or are you interested in doing so? Leave your comment below 👇

[astrojuanlu]

https://medium.com/quantumblack/kedro-goes-streaming-34e1094c354c

[astrojuanlu]

A user https://linen-slack.kedro.org/t/27082217/hi-all-i-am-new-to-kedro-and-have-gone-thru-intro-docs-so-th#ec525ce7-d70e-4577-a4a5-deb47ee1ed9c

Most of the examples in the docs are with static datasets (csv, sql etc) but don’t see any examples for ticking datasets.

Something that explains how the pipeline will work if the datacatalog has a realtime event stream source, eg Kafka etc.

Can someone direct me to docs or examples?

[astrojuanlu]

Another user https://linen-slack.kedro.org/t/10274126/is-there-a-good-way-on-using-kedro-with-structured-streaming#d9f8f9fe-cb09-4b5b-a594-177351819ac8

Is there a good way on using Kedro with structured streaming? Customizing runner or doing other similar things?

I just started studying streaming and would be nice to have some good scaffolding on this. Also making the overall project structure similar between batch and streaming would be interesting in a learning curve as people would only see one single structure.

[astrojuanlu]

Most recent https://linen-slack.kedro.org/t/28560045/hi-are-there-any-plans-to-make-kedro-natively-built-for-stre#72d9307e-835d-45ac-9f17-a191af566c7d

Hi, are there any plans to make Kedro natively built for streaming (spark Streaming for reading and writing streaming data) without using using custom nodes and hooks?
the disadvantage that I have to write a lot of spark logic to handle streaming

[astrojuanlu]

Earlier discussion #912