Implement "package contents" docs for plugins

[caro401]

Each plugin needs to be able to publish documentation about what modules, operation, data types it contains. This is equivalent to this overview page and the details pages for each "thing" eg this one for operations in the existing mkdocs-based documentation.

Ideally (pending results of @MariellaCC's survey) internally-developed plugins should be able to host this part of their docs on a single central docs site, to help users be able to find what they need in one place and not get lost. Externally-developed plugins need to be able to host this kind of documentation standalone.

Ideally, we should be able to always provide documentation for all versions of a plugin, with the user seeing the most up-to-date version by default.

A proposed technical solution to this is as follows:

• each time a plugin publishes a version, it commits a json file called vX.Y.Z into a docs directory. This contains all the content needed for the 'package contents' bit of the plugin's docs at that version, as generated by asking kiara for that data via its API.
• Each plugin also comes with a svelte-kit app that gets all its content from these json files, and knows how to render them to make a standalone docs site for that specific plugin. It mostly uses components from a shared component library. This can be published to github pages for standalone plugin docs of external plugins.
• if the plugin is internal, when it makes a new version, also PR the main docs repo to add its new json file in a known place. (and also maybe update a changelog file?)
• the main docs site also has a sveltekit app which renders the 'package contents' bit for each plugin in mostly the same way, using the json files per version, but all the internal plugins together in a single site. It never needs to run any python itself for this part of the docs, because all the information it needs from kiara about a specific version is in that version's json file.

JSON is proposed as the initial file format, because this is easy to work with from javascript, and is what Kiara's API currently produces. This could be changed in future if it turns out these files are too big and a binary format (sqlite/arrow etc) would be a better fit. Static files may also be of use for "environment metadata" within kiara.

This collection of JSON files and minimal sveltekit app, publishable to github pages automatically would replace the current mkdocs solution in plugins. Steps that need to be done are

• define the format of the generated JSON file - @makkus this is something you could handle once the pydantic migration is done?
• make a way to generate this JSON file automatically for a specific plugin at a specific version - a make command in the plugin?
• make a prototype sveltekit app including components that renders these JSON files into a docs site for a single plugin
• make the github action to generate JSON and deploy docs site on a plugin release
• extract components into an NPM package, publish that(?) and use that from this docs site
• make a sveltekit-based version of the "main" docs site, add JSON data from the plugin and render it at the appropriate permalink
• make an easy way for internal plugin authors to add data for new versions of their plugin to the main docs repo (github action to open a PR when a new plugin release is made?)
• update the plugin template repo to use javascript rather than mkdocs for their standalone docs sites #38
• update remaining plugins to use this strategy

@makkus have I missed anything obvious here? Which plugin do you think should be used to prototype in?

In the future, we could use this static approach to capture examples and test data from plugins, to record the python code and output for tutorials and guides at specific plugin versions. This avoids any concerns about how to call complex sequences of python code from a javascript context.

[makkus]

I can get started with this, and try to create an initial json file, schema and ideally Typescript model classes for you to use.

Once we have that, we can start thinking about the other questions?

[makkus]

Ok, I implemented a first version of creating a data structure that contains all that info (available in kiara >=0.5.5). I haven't added it to any CI runs yet, but I will do so in the future and probably create a json artefact with it whenever a tagged version is released.

It can be used like so from the commandline:

kiara info plugin explain kiara_plugin.tabular --format json

To get the json-schema for the data, you can do:

kiara info plugin explain kiara_plugin.tabular --format json-schema

Initially I also wanted to auto-create Typescript classes, but the library I used for that broke with the pydantic 2 release. I hope that the json-schema is good enough for now.

For development, I'd suggest you just write some example data into a json file and use that to retrieve any information you need. It should be trivial to convert that to a CI workflow where we first auto-write the json file, then pick it up from the doc render step.

[caro401]

I've just merged #34 which takes the JSON files from kiara info plugin explain and renders some pages based on that. I'll add automation/documentation of how to generate the JSON files in #30 , and open a separate issue on the plugin template for the remainder of the work for standalone plugin docs for non-core/supported plugins