Subscribe to streaming sources (continuous ingest)

[amotl]

Dear @karakanb and @turtleDev,

while and after adding support for CrateDB (GH-238), we are currently exploring all the many capabilities of ingestr. I recently used the Kinesis source and @hlcianfagna used the Kafka source today, both sinking data into a CrateDB destination.

Both of us discovered that the ingest process currently stops at the end of the stream. Do you think it is viable to expand features in this area to be able to consume streams continuously from relevant stream sources aka. "subscribe to" them?

If it's out of scope for ingestr, we would need to look into other top dog polyglot frameworks like FastStream (previously FastKafka and Propan), but it would be so cool if ingestr could be the versatile swiss-army-knife-like toolkit we would just use in all the situations. Would you be open to accept patches to enhance ingestr in this area?

With kind regards,
Andreas.

[karakanb]

Hi @amotl, I appreciate the thoughtful approach here, thank you.

When we initially designed ingestr, it was positioned to be a batch loading tool. You integrate your source, run ingestr in regular intervals, and it just loads the data, simple as that.

With that being said, I don't see anything fundamentally wrong with ingestr supporting streaming usecases in a long-running fashion as well. We would want that to be an explicit behavior, e.g. behind a --stream flag, so that it is clear what it will do.

If you have a proposal about how this could be implemented, I would be happy to discuss. My preference would be either having a quick chat together to think outloud on this (as well as get to know each other), or a written down document before submitting patches.

Would you be up for that?