Local execution monitoring

[tiborsekera]

Are there any plans to support:

• monitoring of the elyra pipeline, when running locally?
• command line interface, so that one can run pipeline via bash script or so?

Or, is any of that already possible?

[kevin-bates]

Hi @tiborsekera - thanks for your questions.

Pull request #1246 will address the second item and is nearly ready. If you'd like to take that branch for a spin, that would be helpful.

We might need further information as to what kind of monitoring you'd like relative to your first item.

[tiborsekera]

Hi @kevin-bates !

Thank you for the reply!

Reg. monitoring:
Currently, to now that the locally executed pipeline is running, which step is running, etc. I have to use some custom logging and other libraries. It would be great to be able to see, at least via command line, that the pipeline is running. Even better, to have some elyra_pipeline.log file, where one can see whether the pipeline is running, crashed, which step is running, etc.

[kevin-bates]

Thanks. I think it would be good to define what local execution really means. I believe this should be viewed as a means of prototyping the pipeline and primarily a development aid. However, it does seem like an execution ledger could be something useful that would contain node execution timings and failure information pertaining to the executed pipeline.

Definitely feels like something we should discuss further.

Does anyone else have ideas of where we should head with this?