Skip to content

Should signal loudly when internal channel capacity exhausted #209

Open
@pnkfelix

Description

@pnkfelix

After writing the docs for the parameters that control the internal channel capacity for the ConsoleLayer -> Server communication, I realized that I don't think the current UI does anything to relay that channel exhaustion may have occurred. The system reacts to this scenario by throwing events away until capacity is freed up.

Thus, the output may be misleading (e.g. showing tasks that never joined, when in fact what happened is that their join event was dropped) if some of the events have been dropped. But the console UI doesn't say that this has happened. (I think its currently up to the human user to notice "something's weird about this output" and dig deeper into why.)

I suggest we attack this in two ways, concurrently:

  1. We keep a single boolean for the channel, on whether it had ever been exhausted or not. When the channel is exhausted and that flag is false, then after capacity is subsequently freed up, the first event we will push to it is a special event signalling "an arbitrary number of events may have been dropped from this point on", so that the client UI can reflect that in some sort of warning about the model's accuracy from that point on in the console output.
  2. We add an extra channel, solely for communicating with high priority when the first channel is exhausted (but that the events that are currently in the channel have not yet been dropped -- i.e., that they still yield an accurate model of the runtime). This way, if the problem is that the app is simply generating events at too high a frequency, the end developer will hopefully at least get a notification about it before waiting to process the entirety of the event.

With part 1 in place, the console UI should be able to render sets of tasks (i.e. every task that was still alive at the time of channel exhaustion) as having potentially inaccurate output. (Ideally we'd be able to indicate which outputs should not be relied upon, but I'd be happy with just a general warning at the top.)

With part 2 in pace, the console UI should be able to signal exhaustion very soon after it occurs, before the model becomes inaccurate. Thus, end developers who are still debugging their app can start investigating the problem without waiting for the events currently in the channel to drain completely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-instrumentationArea: instrumentation.C-consoleCrate: console.C-subscriberCrate: console-subscriber.E-mediumEffort: medium.S-featureSeverity: feature. This is adding a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions