Document `juju-exec` in ops charming practice

[dimaqq]

We should probably document juju-exec as part of charming practice
Dima Tisnek to open an issue to have us add that in the docs, David to post on the issue where the new content should go.
How to manage the workload :: machine :: asynchronous operations
Should we recommend a specific charm lib?
Presumably, we’d recommend to juju-exec the charm code, over bash scripts that call hook commands directly.
For reconcilers: pretty easy, just reconcile (except we insist on listening on specific events)
For delta charms: should we expose user-defined / fake Juju events in Ops?
Note that a complex charm may use multiple libs that ultimately juju-exec
Or shall we tell chambers to juju-exec as update-status?
Or how can a delta change listen for juju-exec invocations?
How to add event payload (env vars? temp file?)
Do we know for a fact that juju-exec is mutually exclusive wrt. Juju event dispatch?

[dimaqq]

• validate that there's mutual exclusion between juju-exec run and this or other charm's real Juju hooks
• validate that juju-exec completes synchronously (in which case the script that calls it could create temp dir, dump some data for the charm, call juju-exec and cleanup afterwards)
• where does this go in our docs? (something on par with when and how to use Pebble custom notices)
• concrete example how to juju-exec this unit's dispatch
• recommend what event to abuse (config-changed? update-status?)

[dwilding]

Thanks for tracking this. For now, I recommend putting the info as a section in How to run workloads with a machine charm. Later, I'd like that doc to be split into a how-to about managing system packages and a how-to about juju-exec.

When we publish info about juju-exec, we could also consider adding an explanation doc about handling async processes, linking to the juju-exec doc and our docs about Pebble notices & health checks.