How to export from kiara to a jupyter notebook?

[caro401]

the ability to export (what?) from kiara to a jupyter notebook exists already(?). We should probably tell people it exists. I see 2 parts to this how-to:

• end user facing/conceptual stuff

  • why would you want to do that? for use in publications?
  • when would you choose this kind of export over eg just lineage and some resulting data?
  • what does the jupyter output contain?

• how to actually do it

  • where is the code to do that? core? a plugin? which one?
  • how do you do it? I guess this is something a mini-app would need to offer, so I'd need to know how to use it via the python API. Are there other contexts/ways that people would need to do it

I can't write this, because I didn't know the functionality existed.

I guess first input is required from @makkus to show where the code is, check it is currently working, and explain how to use it. Then @MariellaCC is best placed to explain to an end user why they'd want to do it?

[makkus]

Just quickly (will put more info in here later): this does export of a declarative pipeline structure into a jupyter notebook. This is not relevant if people use the Python API as far as I can see, it's only relevant if you have such a pipeline structure in some way. In theory, this could also be used to render a value lineages (the reverse pipeline) as jupyter notebook, but that hasn't been implemented (and there might be some things I'm missing that would cause it to not work) because it was not a requirement so for. The assumption for this was that the frontend would have some sort of pipeline config/structure reflecting the current state of the user workflow session, and that structure could be automatically rendered into (very simple, but working) Python code.

Those pipelines can in theory be exported to anything you can imagine, like a streamlit mini-app, or a static html page that describes the stages/steps that are contained and in which order, etc.

[caro401]

General export is a much broader question, please open a separate issue to discuss this.

In this issue, I'm looking for concrete documentation on precisely how to export from kiara to a jupyter notebook using existing code from you, assuming that there is existing functional code.

If that's not clear from my original issue, please feed back to me in Slack about how to write better issues.

[makkus]

Sorry, I don't understand, what do you mean with 'export from kiara to a jupyter notebook'? My answer from above relates to the notebook export thing we were talking about on slack. If you meant something else, then I'm sorry, I misunderstood. Can you please elaborate? Thanks!

[caro401]

Ok so I fundamentally don't understand what this notebook export feature is meant to be. Please can you or @MariellaCC fix my assumptions about what this feature actually is, and redo the documentation request.

As I said, I wasn't aware this feature existed at all, I just did my best to interpret what you all were talking about so we could attempt to make some progress and share some knowledge

[makkus]

As I said, I'll write more about all this later. I just want to make sure I understand what you were asking. My above answer was a first (apparently failed) attempt to outline the general thing. But I guess it was what you wanted to know after all (since its what we talked about on slack), so I'll expand on that once I'm back working officially.

[MariellaCC]

@caro401
Concerning the code and where it was in Kiara: this might be outdated, but in this doc: https://dharpa.org/kiara.documentation/latest/workflows/ there is a mention of notebook "rendering" (what we called "export" in the requirements) with Kiara. If I remember correctly, in the past it worked with a kiara render command from the CLI.

Concerning users, a new feedback session from the team would be needed indeed on this, after we figure out if/how it works.

[makkus]

Ok, so, I've updated the jupyter notebook template to reflect some of the code practices we've talked about, if I forgot something let me know.

I'm the wrong person to ask about the background or why this is required, but as I understood it the team wanted some way to create code that would be equivalent to what happened when a user used the GUI to run a workflow. That would then be published along the paper, or could be used within the GUI to be displayed in some way or other. But it's probably a good idea if someone else chips in here and explains the reasoning behind it. @stakats , @MariellaCC ?

The need to render some internal kiara model into something else has come up for other internal stuff (which would in most cases not be relevant for any kiara end- or api-user, if it comes up we can document it seperately), which is why I decided to have a sort of internal 'mini-framework' for this, to enable some code-reuse etc.

Just in case you are interested and to get an idea, you can see a list of all available renderers via:

kiara render list-renderers

(not all of those are functional)

And to filter only the one we are interested here, you'd do:

kiara render --source-type pipeline --target-type jupyter_notebook list-renderers

This should give you all the usage information you'd need. If something is missing, let me know and I'll amend the docs.
For this to work, you need kiara version >=0.5.7, and kiara_plugin.jupyter version >=0.5.0.

You need to have some sort of pipeline structure for this to work, obviously. I guess I could add support to render this by looking into a values lineage tree, and converting that into a pipeline, but that wasn't a requirement so far, so there is no code. But if it is needed, let me know.

[MariellaCC]

I'm the wrong person to ask about the background or why this is required, but as I understood it the team wanted some way to create code that would be equivalent to what happened when a user used the GUI to run a workflow.

My understanding is that initially, the proposition was that non-coder users would assemble and run workflows in a UI, and would then be able to export their workflow in the form of Python code in a Jupyter notebook. The resulting notebook could then be attached to publications, or used as such for publication (JDH). The exported notebook was anticipated as a tool for workflow code sharing and reproducibility/auditing by peers. In the early brainstorming sessions, it wasn't anticipated that the engine used to run the code in the UI, and export the notebook, would be apparent in the resulting notebook, but the project changed quite a lot since then.

Evolution on the requirements for this functionality should be collected by adding this question as an agenda item in a team meeting in my opinion.

[makkus]

Ah, before I forget, there is some example invocation in the kiara.examples repo if you want to create a notebook from a non-trivial pipeline and see what it looks like:

https://github.com/DHARPA-Project/kiara.examples#render-a-jupyter-notebook-from-a-pipeline-file