idle sessions exceed the max idle time by many minutes before being removed

[rokroskar]

Idle sessions do not get removed when reaching the idleness limit, but some time later (sometimes more than 10 minutes later).

[olevski]

This is just how the kubernetes controller time functions.

We use RequeueAfter (in some cases) to delay adding the event back into the queue to controller uses to reconcile the CR. But all this guarantees is that the event will be processed at the delay OR LATER. And if the controller is busy or there are lots of CRs then the processing delay (in addition to the queuing delay) will grow. That is what we are seeing.

For example a large class coming and launching sessions all at once can easily fill up the queue and cause the processing delay to grow.

The queue also has some rate limiting and protections to make sure it does not blow up the memory / whole thing. So it could be even though we use RequeueAfter sparingly that we are hitting limits some of the rate limits on the queue.

[olevski]

The current delay in amalthea 0.22.0 is 10 seconds and it is added as a delay to requeuing only when the session is fully running. In all other cases we dont put a delay.

What we can try is:

• Requeue after a long time (i.e. 30 mins) if the session is hibernated. If it is resumed an event will be queued immediately because the CR changed. We cannot set this to zero because we have a timer to delete the session if it is hibernated for too long.
• In all other keep the 10 sec delay. This is just to avoid the queue getting constantly hammered when there are lots of sessions.