Launching docker container for each dataset

[titansmc]

Hi,
We would like to have it running in our k8s cluster, is it possible that we spin up a new container for each dataset?
Thanks.

[alokito]

HI @titansmc ! This is a feature that I would like to enable, but it hasn't been prioritized by my company, and I don't really have the time or k8s expertise to do it on my own (though I am working on the latter part!). I will leave this ticket open in hopes that I or someone else gets to it. With the "ItemSource" interface we now have flexible data item sources, allowing both filesystem and S3 locations. Ideally we would implement a similar kind of flexible compute.