Skipping of demands at SObjectizer's shutdown

[eao197]

This issue was described in Marco Arena's SObjectizer Tales – 28. If I had a magic wand…. The problem is that when SObjectizer's shutdown is initiated then all agents will handle all demands already in agents' queues. It might take a long time and there is no way at the moment to drop such demands to speed up the shutdown procedure.

I like the idea of telling SObjectizer that an agent agrees to skip remaining demands on shutdown:

class my_agent final : public so_5::agent_t
{
public:
    my_agent(so_5::agent_context_t ctx)
        : agent_t(ctx + drop_events_at_shutdown), ...

The question is "how to implement it?"

I'm afraid there is no way to tell something like "drop all demands except evt_finish for all bound agents" to every dispatcher. It's because there are dispatchers that do not have their own demand queues (like Asio's based dispatchers from so5extra companion project). Moreover I don't want to ask a dispatcher to analyze extracted demand because it would make writing of custom dispatchers harder.

But it's possible to speed up demand handling when shutdown is in progress. It means that demands will be extracted from queues the usual way, but they will be skipped (except for evt_finish).

The simplest way is to switch an agent to a special state (maybe the state used for so_deactivate_agent() is appropriate for this purpose too). In such a state an agent has no subscriptions and every demand (except evt_finish) will be ignored the usual way. There is no need to add some new flags to the agents and check those flags during the demand handling procedure.

But switching the agent to that special state will lead to calling on_exit handler for the current agent state (even states if an agent is a hierarchical FSM). I don't know if it is good or not.

Another approach is to just drop all agent's subscriptions and filters but do not change its current state. In such a case demands will be ignored, but on_exit handlers won't be called.

---

At the moment of writing I prefer an approach with reusing a special state from so_deactivate_agent. And the required behavior of an agent may be described as:

class my_agent final : public so_5::agent_t
{
public:
    my_agent(so_5::agent_context_t ctx)
        : agent_t(ctx + auto_deactivate_at_shutdown), ...

That means that an agent asks the SObjectizer to automatically deactivate the agent at the shutdown.

Such an approach avoids introducing a new concept to SObjectizer, because we already have such a thing as the "deactivated agent" and the method so_deactivate_agent.

---

I would like to collect some feedback before I start work on this feature.

[eao197]

It seems that there could be two kinds of deactivation of agents:

• transparent or exposed. It means that when an agent is switched to a special state and all on_exit handlers are called for the current state(s) of the agents. This is the default behavior for now;
• silent or covered. It means that the agent stays in its current state(s) but all its subscriptions and filters are dropped. No on_exit handlers are called.

The functionality of so_deactivate_agent can be extended by providing another so_deactivate_agent overload:

enum class agent_deactivation_t
{
  exposed,
  covered
};

void
so_deactivate_agent(agent_deactivation_t mode) noexcept;

It also allows to specify mode to auto_deactivate_on_shutdown:

class my_agent final : public so_5::agent_t
{
public:
    my_agent(so_5::agent_context_t ctx)
        : agent_t(ctx + auto_deactivate_at_shutdown(agent_deactivation_t::exposed)), ...

With such an approach a user can select which kind of deactivation is preferable.

[ilpropheta]

Hi @eao197!
Thanks for taking time to look at this.

First of all, I believe it's acceptable to skip demands. This is essentially what happens when you buffer data into the agent and ignore them during shutdown.

Personally, I prefer simplicity and consistency, so if the agent is being deactivated on shutdown, invoking on_exit is the right thing to do. However, That said, I'm not a regular user of on_exit handlers myself, so I'm not sure whether this approach might limit the broader use of the feature.

[ilpropheta]

I just noticed this comment before mine, apologies for missing it earlier. I like the idea of selecting an appropriate deactivation mode, but I’d like to share a few thoughts:

• the naming feels a bit unclear: terms like exposed and covered don’t clearly convey their intent;
• I wonder if SObjectizer could offer a sensible default that works well in most cases, so users wouldn’t have to dive into the details unless necessary;
• switching to a deactivated state implies a state transition. In that context, does it really make sense for covered to exist? Perhaps this isn't a matter of auto_deactivate_at_shutdown(agent_deactivation_t::covered) but rather a different kind of policy, something like ignore_messages_at_shutdown, for example. My point is: if you’re transitioning states, then on_exit should be called for the current state. Otherwise, it’s not truly a state change. As I understand it, so_deactivate_agent() does call on_exit for the current state, correct?

Probably this is just syntactic sugar and the internals won't change.

What do you think?

It seems that there could be two kinds of deactivation of agents:

• transparent or exposed. It means that when an agent is switched to a special state and all on_exit handlers are called for the current state(s) of the agents. This is the default behavior for now;
• silent or covered. It means that the agent stays in its current state(s) but all its subscriptions and filters are dropped. No on_exit handlers are called.

The functionality of so_deactivate_agent can be extended by providing another so_deactivate_agent overload:

enum class agent_deactivation_t
{
exposed,
covered
};

void
so_deactivate_agent(agent_deactivation_t mode) noexcept;
It also allows to specify mode to auto_deactivate_on_shutdown:

class my_agent final : public so_5::agent_t
{
public:
my_agent(so_5::agent_context_t ctx)
: agent_t(ctx + auto_deactivate_at_shutdown(agent_deactivation_t::exposed)), ...
With such an approach a user can select which kind of deactivation is preferable.

[ilpropheta]

Addendum: internally, you might have so_deactivate_agent(agent_deactivation_t) but I wouldn't expose this to the user since so_deactivate_agent(covered) doesn't really transition the agent to another state. I would only call this from the strategy handling ignore_messages_at_shutdown. The point is: if so_deactivate_agent implies a state transition, I am wondering if so_deactivate_agent(covered) would be semantically correct. It looks like another function.

[eao197]

What do you think?

First of all I don't like any of my terms used above (transparent/exposed, silent/covered). They are just attempts to find something appropriate. It's always hard to me to find a good identifier and there is an old programmers joke I heard when was a student -- "there are unsolved problems in the software development, for example: predictions of development time and choosing of good identifiers" :)

The main idea behind transparent/exposed is the complete deactivation with:

a) dropping all subscriptions and delivery filters;
b) switching to a special state.

Because so_deactivate_agent leads to switching the state and this switch is visible (via on_exit handlers) this deactivation is named as transparent or exposed. It means that this deactivation is obvious/visible.

But it seems to me that sometimes it's better to partially deactivate an agent: dropping all subscriptions/filters, but keeping the agent in its current state. So this deactivation is not visible, because on_exit won't be called. That is why I used terms silent/covered.

May be complete or partial will be more appropriate. I hope the right variants will be found soon.

The next thing is default values for auto_deactivate_on_shutdown. I thought about this version of auto_deactivate_on_shutdown:

[[nodiscard]]
auto_deactivate_on_shutdown_indicator_t
auto_deactivate_on_shutdown(
  agent_deactivation_t mode = agent_deactivation_t::exposed);

So if a user simply calls ctx + auto_deactivate_on_shutdown he/she will get the default behavior via ordinary call to existing so_deactivate_agent().

As I understand it, so_deactivate_agent() does call on_exit for the current state, correct?

Yes, you're right.

Thanks for your thoughts, in many ways they overlap with my current doubts.

---

Another variant of naming:

this->so_deactivate_agent(); // The usual way.

this->so_deactivate_agent(so_5::keep_state); // Drop subscriptions/filter, keep the current state.

agent_t{ctx + auto_deactivate_on_shutdown() + ...}; // Ordinary deactivation of agent on shutdown.
   // It means that this->so_deactivate_agent() will be called for agent on shutdown.

agent{ctx + auto_deactivate_on_shutdown(so_5::keep_state) + ...}; // Deactivate without changing the state.
    /// It means that this->so_deactivate_agent(so_5::keep_state) will be called.

[ilpropheta]

It's always hard to me to find a good identifier and there is an old programmers joke I heard when was a student -- "there are unsolved problems in the software development, for example: predictions of development time and choosing of good identifiers" :)

I know this one "There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors" :)

Thanks for the detailed explanation. It makes sense to me.

I think that so_5::keep_current_state (or so_5::keep_this_state) would be express the intent even more. But it's just personal preference and maybe it does not fit with current SObjectizer's naming conventions.

[eao197]

I know this one "There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors" :)

It's famous nowadays, but I in doubts that "cache invalidation" was a serious problem in early 1990s :)

so_5::keep_current_state

It's definitely better than just so_5::keep_state.

[ilpropheta]

It's famous nowadays, but I in doubts that "cache invalidation" was a serious problem in early 1990s :)

😁 fair enough!

[eao197]

A new thought: skipping the remaining work may be needed not only on SObjectizer's shutdown, but also when agent's coop is being deregistered.

Suppose a case when a new coop created for handling an incoming request (and handling of the request may take a long time). Then we want to cancel request processing and deregister the coop created for this request. But if some agent has pending demands in its demand queue then all of those demands have to be processed even if we don't need the result.

So I think that auto_deactivate_at_shutdown can be renamed to auto_deactivate_on_dereg. Thus we'll get a functionality of skipping pending demands on deregistration of agent's cooperation.

Because the shutdown of SObjectizer initiates deregistration of all cooperations the auto_deactivate_on_dereg will work for SObjectizer's shutdown too.

An open question: what if we don't need to skip demands on ordinary deregistration of agent's coop, but demands have to be skipped on SObjectizer's shutdown? But I wonder: why ordinary deregistration of agent's coop should differ from deregistration in the case of shutdown?

[eao197]

An underwater rock: the so_deactivate_agent() can't be called when agent's coop is deregistered (no matter because it happens). The call to so_deactivate_agent() should be made by the agent itself on the worker thread to that agent is bound. But coop deregistration may be initiated from anywhere, especially in the case of SObjectizer's shutdown.

It seems that skipping of pending demands can be implemented, but it will be done without so_deactivate_agent machinery.

[ilpropheta]

But I wonder: why ordinary deregistration of agent's coop should differ from deregistration in the case of shutdown?

Shutdown might be considered a special case, but it's still just one of several possible reasons for deregistration. In my view, if SObjectizer supports skipping pending demands more generally in the context of coop deregistration, that could be useful. I'm not sure whether the intention is to offer users more fine-grained policies to distinguish behavior during shutdown versus other forms of deregistration, but that might be worth considering. Surely, there would be use cases for both.

An open question: what if we don't need to skip demands on ordinary deregistration of agent's coop, but demands have to be skipped on SObjectizer's shutdown?

This reminds me that SObjectizer gives the user the opportunity to make custom actions at the shutdown when processing stop guards. However, I'm not sure whether that mechanism could be useful here (personally, I prefer a more declarative approach such as creating an agent with + auto_deactivate_on_dereg , or similar).

An underwater rock: the so_deactivate_agent() can't be called when agent's coop is deregistered (no matter because it happens). The call to so_deactivate_agent() should be made by the agent itself on the worker thread to that agent is bound. But coop deregistration may be initiated from anywhere, especially in the case of SObjectizer's shutdown.

It seems that skipping of pending demands can be implemented, but it will be done without so_deactivate_agent machinery.

So, does this make it impractical to support skipping pending demands during coop deregistration? It’s not a major concern for me, I'm just asking to better understand the implications.

[eao197]

So, does this make it impractical to support skipping pending demands during coop deregistration?

No. It means that so_deactivate_agent won't be used for this skipping. Because of that I'm thinking about:

my_agent::my_agent(context_t ctx) : so_5::agent_t{ctx + skip_demands_on_dereg() + ...}

instead of:

my_agent::my_agent(context_t ctx) : so_5::agent_t{ctx + auto_deactivate_on_dereg() + ...}

It also means that so_5::keep_current_state can't be used with hypothetical skip_demands_on_dereg, because skip_demands_on_dereg won't used so_deactivate_agent under the hood.

[ilpropheta]

Ok, gotcha. I think that something like skip_demands_on_dereg is spot on. And this also resolves every doubt on calling on_exit that is not necessary since the agent state is not changed.

[eao197]

And this also resolves every doubt on calling on_exit that is not necessary since the agent state is not changed.

IIRC, SObjectizer automatically switches agents to the default state after completion of so_evt_finish:

sobjectizer/dev/so_5/agent.cpp

Line 1356 in 10eb34c

d.m_receiver->return_to_default_state_if_possible();

So if agent wasn't deactivated explicitly by manual call of so_deactivate_agent() then on_exit will be called anyway.

[ilpropheta]

Ok, then the behavior stays consistent.