Proposal: Event tracking

[ManevilleF]

In our crux app we quickly got the need for the shell to know when an Event is finished, and potentially get an output from it.

A typical example would be the shell sending a SyncWithRemote event which the core process by doing a bunch of database and http calls.
The core could build a state machine in the view model to expose the progress with a value like sync: Pending/Done but it doesn't scale well and view models can get pretty complex.

So we built a system similar to the following:

pub enum Event {
   Init,
   SyncWithRemote,
   DoSomething
}

pub struct TrackedEvent {
  pub event: Event,
  pub id: Option<String>
}

the App::Event is actually TrackedEvent.

We added a Track capability, able to notify the shell with the following operations:

// Status of a tracked event
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize, facet::Facet)]
#[repr(C)]
pub enum OperationStatus {
    /// The event was started by the core
    Started,
    /// The event was successful
    Success,
    /// The event failed -- error message
    Error(String),
}

/// Notify the shell of the status of a tracked event
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize, facet::Facet)]
#[repr(C)]
pub struct TrackOperation {
    /// Identifier of the tracked event
    pub id: String,
    /// New status of that event
    pub status: OperationStatus,
}

The simplified logic is that if the shell specifies an TrackEvent::id we are going to wrap the associated Command with some tracking

Command::new(|ctx| {
  ctx.notify_shell(TrackOperation { id, status: OperatioStatus::Started });
  let result = {
      // The actual event processing return a `Result`
  }.await;
  match result {
      Ok(()) -> ctx.notify_shell(TrackOperation { id, status: OperationStatus::Success }),
      Err(e) -> ctx.notify_shell(TrackOperation { id, status: OperationStatus::Error(e.to_string()) })
  } 
}

This way the shell is notified when:

• The core actually starts polling the associated Command
• The core finishes processing and gives a status

Concretely it allows many things for the shell:

• Display loading screens
• Show sucess toasts
• Given context (The original user instruction) to failures instead of a simple error capability.

This works well as long as the shell has to specify unique ids itself based on an Event + some entropy to avoid duplicates but we can go even further by adding data to the Success variant.

Doing so unlocks some cool features:

• Being able to query the core without having to be based on the view model
• Returning relevant data

An example could be that for a calendar app the Event::CreateCalendarItem could return the actual new calendar item unique id and avoid for the shell the necessity to do some comparisons between the previous view model and the new one to find the "new" item that was created
This goes even further if we are in a workspace environment with potentially multiple new calendar items popping up. The shell is often confused on what is a result of an instruction it sent and what is coming from something else

I think crux could benefit from having a tracking mechanism built-in, would that make sense ? If so I could work on an RFC for that

[charypar]

Hey @ManevilleF! Thanks for the detailed proposal!

I've encountered a similar need before, and I have to say I am of two minds about it 😅 If I understand it right, it essentially makes it possible to wrap the core in an async API on the shell side, where in TypeScript, for example, you could return a promise which resolves once the relevant TrackOperation is received.

On one hand, I do understand the utility of this. And I can also see how making a capability like that first class (or even making it built in) would make thigns like nice developer tools possible.

On the other hand, I've previously discouraged that kind of API, in favour of a strictly passive UI observing a more detailed ViewModel – effectively what you describe here:

The core could build a state machine in the view model to expose the progress with a value like sync: Pending/Done but it doesn't scale well and view models can get pretty complex.

In my mind, the need for promise like APIs suggest that the UI is being too active and making too many decisions. The quoted example below:

This goes even further if we are in a workspace environment with potentially multiple new calendar items popping up. The shell is often confused on what is a result of an instruction it sent and what is coming from something else

... is exactly the sort of situation which should be really naturally covered in a strictly reactive UI that is just observing a more detailed view model and adapting. Having the decision making and UI triggers split across the shell and core hurts the testability and portability and consistency across platforms.

I think my initial gut feeling is that the tracking capability as you describe it above would be a great candidate for contributed capabilities. I appreciate its useful at times, especially in migration and gradual adoption scenarios, but personally I don't really want to encourage that style of interaction, over the more circular data flow of events in, view model out, no explicit relationship between the two.

This is clearly just my opinion, I'd very much like to discuss this further to understand the needs and the complexity you mentioned better, and also let @StuartHarris and others chime in. This isn't a clear cut decision at all.

[mhedgpeth]

Thanks for the detailed proposal; this is great!

This is something that in year 1 of my crux journey I would have loved and embraced immediately. But in year 2 and with LLMs the way they are, I have gone uber-state machine, as you can see in the examples/weather in my recent contributions, with Outcome, etc. It's just been better for me to have the complete reasoning and invariants modeled right there in the app's model. It feels to me like the "rusty" way - if it compiles it works. And yes I do have a lot of Pending/Done stuff. But you'd be surprised how much goes beyond that, and is really nice to model with rust enums.

But that's as I got really advanced with rust and crux. Once the "rust" way became second nature after almost a year of using it, I doubled down on making the Model states difficult to represent badly, with liberal use of enums.

That said I agree with this would lower the barrier to entry for a lot of these situations and not force such tedious onboarding (especially without LLM usage) that could trip up users, or even just people who chafe against such strictness coming from more permissive UI models.

[ManevilleF]

Thanks for your answers !

I get that the idiomatic crux way is to have the UI being purely reactive, but I think that is not enough as it doesn't scale on large apps that target multiple platforms. In our case we target mobile, web and desktop.

I think at some point in development your app reaches a level of maturity where tracked or managed events becomes mandatory. This does not replace anything in the way the UI is made, it's complementary.

Purely on the core side you will want to:

• Have telemetry and metrics on the processing of events, for this you need some abstraction on the core side that give you a processing starting point and an output you can construct metrics from, especially if an error ocurred and you also want to have time measurements
• Have retry strategies, if an event fails based on an recoverable error you want to be able to re-inject that event in the core with enough metadata so that you don't loop indefinetely and have relevant metrics
• Be able to communicate information to the shell without the need of the view model, I gave the example of the calendar item but this is even more true when you work on mobile app backgrounding

Let's give an example:

Scenario:

You build a calendar app, we focus on the CreateCalendarEntry event that the shells send to create a new calendar entry.
Calendar entries can also come from 2 different sources:

• From remote as your app syncs with whatever is on the server
• From a consequence of that event, for repeating calendar entries (every week or so)

Calendar item creation can fail for many reasons, the biggest one would be HTTP (rate limiting/offline network). And we want the app to retry the failing operations based on a defined retry strategy (Exponential backoff, 3 retries) and we want to upload metrics and errors to our temeletry provider

The UI requires that:

• As a calendar item is getting created we want a toast/spinner
• If something fails we want that toaster red with the associated error
• The created calendar item should be highlighted and focused

Taking in the purely UI approach with no tracking or managed events:

• You need to define an entire enum of possible tracked operations that you display in the view model so that the toast can be displayed, I guess some secion of InProgress with a hashmap of that enum and some state

What if 2 calendar items are getting created concurrently ? How do you distinguish ? What is the discriminant ? and more importantly who defines it ?

• When it is created and fails, you can put the error in the InProgress state

When do you clear this error ? Do you create a new event for this ?
But what of retries ? Since there is no tracking do we expect the Shell to re-trigger the original event itself and the retry logic lives in the shell ?

• When creation succeeds, a new calendar item is created in the view

What do we do wit the InProgress ?
How does the core define which CalendarItemView to highlight ? and for how long ? How do you handle unicity ?
If 2 events for creations were sent, as the creation process does not block the UI, we have no guarantee of order on which one will be created first, as per the way the crux runtime works, but we do need to highlight only the last one

With tracking we have:

• The TrackedEvent with the id field defined by the shell and an optional timestamp defined by the shell if we want to measure the time from the shell perspective
• As the operation starts the shell receives the OperationStatus::Started, a good trigger to display the toast
• As the operation finishes the shell receives either Error or Success with the CalendarItemId as the one to highlight based on the original user instruction
• On the core side as we get a Result<Output, Error> in our managed event, within the Command we can:
  • compute the metrics
  • Decide if we want to retry the event, based on a retry strategy, then we don't send anything to the shell, as the operation is stil Started, that's the big gain from this, the tracking works well with retries and event re-injection
  • Send telemetry events

We have different UI/UX requirements between Web, mobile and desktop, and the metrics requirements a system like this becomes very useful and it scales very well. Especially in case of backgrounding apps, or with web apps that have very strict requirements especially for navigation as they need to alter the browser URI and basing the URI from the view model creates a lot of jitter.

I don't think this replaces a reactive UI, this is more of an observability issue. When you send an instruction you want to be able to subscribe to notifications from that instructions or subscribe to the state of that instruction if not the output.
The ability to correlate a User facing instruction (CreateCalendarItem) with an HTTP error is priceless for the UX, the debugging, and for the observability.
This is also a very compact system with a single capability, and there is no need to do breaking API change (We have 3 consuming shells, altering the view model or serialization is "expensive") every time you want to add a new possible toast

---

But that's as I got really advanced with rust and crux. Once the "rust" way became second nature after almost a year of using it, I doubled down on making the Model states difficult to represent badly, with liberal use of enums.

For context we are a team of senior rust engineers and we've been using crux for almost 2 years now, this is not me having trouble with the framework or the hexagonal architecture concept, but alleviating something we consider a real limitation in the current state of the library

I'm making this proposition because I think the way we are handling this is not straightforward and did require some work to work properly and be absolutely safe and correct I made sure that it is impossible to receive an OperationStatus::Started and never get the success/failure, so upstreaming it or making it a contributed library as @charypar mention could be useful to other who might find similar issues.

Also this is no a "Take All or Nothing" proposal, I'm mixing together a lot of pieces of our tracking system which does honestly quite a lot (Tracking, metrics, retriability)

It feels to me like the "rusty" way - if it compiles it works. And yes I do have a lot of Pending/Done stuff. But you'd be surprised how much goes beyond that, and is really nice to model with rust enums.

Small note on that, but we tend to try and use enums less and less, because we target WASM and the stack usage overhead for complex enums is having a significant impact, but this is because the app is particularily large

[StuartHarris]

I'm not against exploring this further. I've felt the need myself for lifecycles when building complex Crux apps (@charypar you probably remember 😅).

I would be interested in an RFC if you're up for writing one, @ManevilleF. I think we should frame it differently though. Instead of "let's add a tracking sidecar", we could think of it like "every shell-originated event already implies a scope of work, let's name it and observe its lifecycle".

So something like the concept of a virtual "arena", which contains the event that is sent in from the shell, and any effects/events it produces. The arena would effectively be "the transitive closure of that Command, including follow-up events generated by resolving effects within it." The executor already has this graph so we'd just be giving it an identity and a settled signal.

We could tag inbound events with an id from a slab at the bridge boundary, propagate it through effect requests and events generated by effect resolutions, and notify the shell when the arena is settled.

It might also enable features like "cancel everything caused by that event", dev tooling support, including causality inspection and timelines.

We'd need to think about an escape hatch, so that if you raised an internal event that didn't really belong in the arena, or for long-running subscriptions, you could opt out (e.g. ctx.detach(...)).

Not sure about pushing data out to the shell this way though. But a settled notification could trigger a view model read.

[ManevilleF]

I really like this approach @StuartHarris , I didn't think of using the command graph for this but now that you mention it it's obvious we should use it.

I can take care of writing an RFC but I don't want to prevent you of doing it if you have a clear idea of how it can be done.

I like that any effect would be aware of that slab, meaning that my issues with the classic Error effect or the telemetry could work out of the box and if we want to extend through other effects then it's basically free.
A thing that would be useful is to allow some custom id, or have the root event discriminant in the command context. Because a generated id looses all meaning for the observability.

Maybe a cool feature would be to allow for CommandContext extensions ? Through a generic parameter that defaults to () ? It would allow apps to build an extension for their observability needs like a shell defined "start" timestamp for telemetry ?

[mhedgpeth]

There is a community meeting this Tuesday afternoon CET perhaps we can discuss then?

[ManevilleF]

Absolutely