kiara features

[makkus]

Here is a condensed version of the main kiara features. Most of them you should already have come across if you have used kiara in the past and went through the existing tutorials I wrote (information in those would be complementary to this -- I might not have included stuff here that is contained in there). This information here that can be found in more details via (mainly) the KiaraAPI source code or discovered via the cli (using the --help option o the kiara command and sub-commands).

As we've discussed before, the more important features are the ones that come from frontend requirements via the use-cases, those should be the 'official' ones. The backend won't be our product, the frontend(s) will be.

So there is a good chance I have missed some obvious, necessary features (since I haven't gotten real frontend requirements/specs yet), and in some cases I did deliberately not implement anything (like notes/comments) because the design of necessary API endpoints would depend entirely on how the frontend implements a specific feature. To that end, it's also important to note (again) that this API is the result of me guessing the endpoints a frontend might want to use, and designing them in a way so that changes can be made relatively easy. Which also means that I sort of assumed we can adjust the endpoints once frontend development gets on its way, and 'real' requirements crop up.

Obviously, that won't be possible every time, but given the constraints in the project it's something I tried to account for. So if you come across requirements that are not implemented, or slightly different to existing endpoints, let me know and we can discuss and adjust.

[makkus]

Introspection

kiara lets frontends introspect basically every aspect of its internal workings, if something is not exposed, then there is a good chance it can be added.

Objects that can be inspected include the current environment the backend is running (Python env, OS, etc.), which plugins are installed and their details (documentation, which modules do they contain, data-types, operations, pipelines, archives, ..), the current config and runtime config. It also lets the the frontend inspect which contexts are available (because the user created them some time before) and their details (like archives it contains), which context is currently used (and its details), and it will let users create and change to new contexts. A context is basically just a workspace with its own data values and aliases.

In most cases, kiara has two different methods to retrieve information about each of the internal objects, one quick one that contains only basic data, and one expensive one that contains basically everything kiara knows about the object.

For details which exact internal objects can be inspected, read the KiaraAPI source, it probably does not make sense to duplicate the information here.

[makkus]

Modules, Operations, Pipelines

Those are arguably the most important concepts within kiara, and introspection is available for all of them. Custom operation instances can be created via the API, a list of all pre-registered operations can be retrieved, and the details of each operation can be inspected. The same goes for pipelines, and modules. There are also methods to retrieve a filtered list of operations, pipelines, and modules, which is useful for specific use-cases like only presenting operations to the user that work with a specific file-type.

In addition, the API lets the user register pipeline stuctures, which in turn can then be used like any other operation. Pipeline config can be a local (to the backend) file, or a dictionary containing the pipeline config data directly.

kiara also has so-called operation types, which are basically categories operations can be sorted into. One operation can be of multiple types. One such category would be 'pipeline', for example, indicating to you that this operation contains multiple sub-operations. Another use is for operations that should have a known, same interface, regardless of their input data type, which could be useful in specific frontend use-cases mabye.

[makkus]

Data / Values

kiara has its own data types, which as everything else can be inspected. kiara data types wrap around Python data types, and provide functionality that is used by the introspection features, as well as for purposes like validation, serialization, and deserialization. Also, in some cases to parse strings into the actual data, as strings are often the only way for users to directly input data (like in the case of the CLI, or yaml/json config files).

kiara lets frontends 'register' data, which involves creating said wrapper around it, after validating against the data type it is supposed to represent, calculating it's hash, and assigning it a globally unique value id.

A kiara context keeps track of data using so called 'data-archives' / 'data-stores', which are classes that implement the necessary management features. Curently kiara has two types of stores: sqlite and filesystem, up until now 'filesystem' was the default, but that may/will change in the future (transparently to the user). Stores keep track of the values they contain, and are responsible for serializing and deserializing them. In most cases the details here are not important for end-users/front-end devs, as the API abstracts away the details of the stores. That might change in the future.

kiara also has a feature that lets users assign meaningful (to them) aliases to values, similar to how filesystems work for Operating Systems (filenames -> inodes). This might or might not be useful for a specific frontend, it's ceratinly possible to create one without using this feature.

The metadata that is contained in a kiara value includes its 'pedigree' (direct ancestor values & module type used to create it), 'lineage' (all ancestor values & module types used to create it), details about the environments it was created in (not fully implemented), its data type, pre-computed data-type specific properties, the hash for its serialized form, and a few other things I probably forgot.

From the next version onwards kiara also supports exporting one or several values into a file that can be shared with other people, and used by them to import those values (incl. their metadata) into their own contexts.

The API supports listing the ids of all values in a context, retrieving details about all of them, a filtered list of them (e.g. only values that are of a specific data type). Deleting of values is not supported atm, as it's surprisingly complex to implement and not a high priority for the use-cases I came across.

[makkus]

Job management

kiara contains a component that manages when/how the jobs the user wants to run are run. In most cases a frontend would use the 'queue_job' endpoint, which returns a reference (uuid) to the job, and which can be used to retrieve the status and eventually result of the job. The job manager also has introspection features, like listing all jobs, and retrieving details about a specific job. There also exists a 'run_job' endpoint, which blocks (so probably not suitable for a UI frontend), and returns the result of the job directly. In addition, there are also 'queue_manifest' and 'run_manifest' endpoints, which are lower-level and not recommended for use by frontends.

[makkus]

Configuration

The basic aspects of a kiara backend can be configured, this is split up into base configuration (KiaraConfig) and runtime_config (KiaraRuntimeConfig). Which configuration options are available can be looked up in the source code, or retrieved via introspection (get familiar with the cli and it's --help option to find what you are looking for, any further questions I'm happy to help with as always).

In most cases, configuration should not be necessary as the defaults should be sufficient, but if you have special requirements, then check out the configuration model classes.

[makkus]

Let me know if you think I forgot anything, that's entirely possible.

[makkus]

Also, happy to expand on anything that is unclear.

[MariellaCC]

The backend won't be our product, the frontend(s) will be.

just a quick comment about that: I think this shifted since the early stages of the project, since 2 years ago we defined a user persona of "modules creators" (via community plugins)

(there is a discussion about that here: #5)

[makkus]

What I meant is that with just the backend, kiara is fairly use-less. We want people to eventually do research with it. Just writing a module without anyone ever using it obviously doesn't make any sense, so module creators can't be our main persona.

For doing research with kiara, we need to have at least one frontend, even if its just a very thin one like using the backend via jupyter (and the requirements that this would put on the backend).

[makkus]

And this means that whatever features are important, they need to be defined/arrived at via frontend requirements. Otherwise there is no justification for any of them to exist.

[MariellaCC]

For doing research with kiara, we need to have at least one frontend, even if its just a very thin one like using the backend via jupyter (and the requirements that this would put on the backend).

Sure this is not contradictory at all, but I think it is important to not forget about these module creators (who know data analysis/statistics python ecosystem and not software engineering python ecosystem, which are 2 very distinct areas of expertise even if both fields are technical/computing related and that Python is a common word)

These users are important since they would bridge the gap for modules that are non existent and needed, as the dharpa team won't have the possibility to anticipate all needed mdoules.

[caro401]

How do these somewhat abstract things map to what a user (of any kind) can actually do with kiara? Given a (future, imagined, all-powerful) UI or python expertise, what kind of tasks can I do, or what things can I achieve using kiara?

My best guess from the above is as follows, but as you can tell from the amount of question marks, I really don't have a clear picture of what's currently possible

• I can import data into kiara, and it (validates? and then) stores it (persistently? where?) with some metadata? with an optional meaningful name?) (but I can't delete it?)
• I can use operations to make changes to this data, and the changed data is stored (non-destructively to original data?, with optional meaningful names?) and the changes are tracked (in lineage which I can view?)
• I can export the data (and lineage? metadata?) when I've made changes to my data (coming soon?)
• I can write my own plugins to provide operations and data types not included in core kiara, if I know python
• I can share workflows/pipelines/bits of data? with colleagues/myself on a different computer?
• I can create different contexts in kiara, which let me store data in different places? isolated from other data? (why would I want to do this?)
• I can configure kiara if I have particular needs (to be different in what kinds of ways?)
• I can ask kiara things about itself, its configuration and the things it knows about? (why is this useful?)
• I can run operations/"jobs" asynchronously (and therefore multiple operations/"jobs" at once?), but it's complicated to do?
• I can (/will be able to?) make notes on what i did in kiara and why

Does this list cover everything kiara can currently actually do?

And does this cover all the end-user needs that have been identified in the various user research/surveys that have been done?

[makkus]

Ok, again, the main point I'm trying to get across is that I did never got 'real' requirements from someone responsible for the frontend experience. I had to 'translate' the end-user needs we collected into a backend design, without knowing about the thing between, what is between the end-user and the backend API. I hope it's clear that this is not how things ideally work, but that was out of my control, and this is why the list of features I gave you is fairly generic/abstract. For some of the requirements (notes for example) I found it impossible to come up with an implementation without knowing more about how a frontend intends to use it. Does that make sense?

I can answer all of the questions above, but before that I'd like to make sure we're on the same page about our basic premise:

• can everyone appreciate what I said about how I designed the API, and that I always considered it as preparation for until 'real requirements' come in, and since it never had any real validation from a real frontend, it can't be considered the 'source of truth'

• and that I don't consider myself a good enough programmer to design an API without any real frontend specs/requirements work that can be used instantly to build a frontend without having to look at actual use-cases/user requirements, and expect it will cover everything that will come up (or contain things that are not needed)

• that I did my best to anticipate the most likely requirements that will come up using different strategies

• but that I really really need you guys to help me fleshing out the 'real' requirements, and usage patterns coming from actual frontends that will be used by actual users, designed using actual requirements and user-stories (can be the same ones I used as guidance)

If we can all agree on that premise in some way, that would be great. Then I'll go through all of the questions above and comment on them, assuming behind all of them is an an actual use-case and reason why those things should be possible.

And I'd suggest that we come up with descriptions / wireframes / or whatever details we have about our two (? not sure about the topic-modeling one) mini-apps. We know what data will go into them, and what users want to do with them (users in this case being Caitlin and Lorella/Mariellal), and we use that to come up with a list of specs/requirements the backend needs to satisfy?

As I said, it's fairly likely that some of the stuff kiara can do is not necessary at all, and would not be used at all by such a mini-app. Like, aliases, we might not need at all. Sharing workflows/pipelines, I can't really see how that would be important for a mini-app, but if you can show me a usage flow for that, I'm more than happy to implement it or change an existing implementation.

I guess the short of it is that I really need your help with all that, and that I dont' have all the answers, and also that we should probably ask ourselves whether we should talk about all the possible features we could have or that exist, or limit our discussion to only what is needed for our specific next goal, the mini-apps. And those should have their own requirements inbuilt, totally independent from what exists atm, right?

[makkus]

Sure this is not contradictory at all, but I think it is important to not forget about these module creators (who know data analysis/statistics python ecosystem and not software engineering python ecosystem, which are 2 very distinct areas of expertise even if both fields are technical/computing related and that Python is a common word)

I feel like I spent a huge amount of time making this easy, writing tutorials on how to create modules targeted at that audience, creating the plugin template to make it easy to get started, investigating ways to make it easier to create a Python env (pixi), etc. To be honest, I'm running out of ideas (and time) to spend a lot more on this, and I was hoping that the docs sub-project could take off some of the load and others could jump in and clean up what already exists. If there is something you want specifically me to do, I'm also happy to do it, but as I said, my own ideas are sort of running out, and I could really need some help there...

These users are important since they would bridge the gap for modules that are non existent and needed, as the dharpa team won't have the possibility to anticipate all needed mdoules.

Again, I feel like I've always had that in mind, and tried to make that as easy as possible. There is a discussion to be had about the quality of the modules we ship 'officially', but that would be independent of this.

[MariellaCC]

I feel like I spent a huge amount of time making this easy, writing tutorials on how to create modules targeted at that audience, creating the plugin template to make it easy to get started, investigating ways to make it easier to create a Python env (pixi), etc.

there was not a feature request at all in my previous message, this was just to acknowledge the existence of such back-end users, nothing more