Feature request: expose per-session memory usage, and session-count perfdata, in check_users

[Fantu]

Is your feature request related to a problem? Please describe.

I monitor Windows RDS/terminal servers passively (NSCP pushing to Icinga 2 via the Icinga API). On those machines the recurring problem is a single user session eating an unreasonable amount of memory: leaking applications, or sessions left disconnected for days that keep everything resident. A recent real case was a disconnected session sitting at almost 18 GB.

That is exactly the information the Task Manager shows in its Users tab (memory per logged-on user, active and disconnected alike), but I could not find a way to get it out of NSClient++.

Looking at 0.14.1:

• check_users (CheckSecurity) reports user, client, session_state and session_type, which is great for who is connected, but has no resource fields.
• check_process has working_set. On current master it also has username and uid, added after 0.14.1, which is real progress towards this — but there is still no session id, and no way to aggregate the matched processes per user inside the check. (Note for other readers: check_process also has a field called user, which is user-mode CPU time, not the process owner.)

So the total memory of a session is still not reachable from the built-in checks, even though both halves of the information now exist in different checks.

Describe the solution you'd like

Optional resource fields on check_users, so that each returned session can carry its resource usage alongside the existing user / session_state / session_type.

I would suggest making it opt-in, off by default — something like resources=true — rather than always collecting it. Most users of check_users probably only care about how many sessions exist and in what state, and they should not start paying for data they do not use. It would also keep the door open for extending the set of fields later without changing the default behaviour.

With that, the check I am after would look roughly like:

check_users resources=true "warn=working_set > 8G" "crit=working_set > 12G" "detail-syntax=%(user) (%(session_state)) ram=%(working_set)"

and would produce one perfdata series per session, which is what makes the slow growth of a leak visible over time rather than only at the moment it becomes critical.

Memory is the field I actually need, so if only one thing gets added, that would be it. CPU per session would be the natural companion, and in principle the rest of what Task Manager shows per user (disk, network) could be useful too — but only insofar as the numbers can be taken from data NSClient++ already has. Which leads to the real question below.

A second, smaller ask: numeric perfdata for the session counts

Independently of the resource fields above — and useful even if they are never added — check_users currently emits no performance data at all, not even the number of sessions. Its fields are all registered with add_string_var() and none carries an add_perf(), so a scheduled check_users returns a status and a text line, and nothing that can be graphed.

Plain counters would be worth having on their own:

• total sessions;
• one per session_state (active, disconnected, ...);
• one per session_type (console, rdp, ...).

These are fixed labels with bounded cardinality, so they behave well over time in an RRD/TSDB backend. That is worth stating explicitly, because it is the opposite of what a per-session series does: session and user labels change as people log on and off, so a per-session perfdata set leaves a trail of dead datasources behind it. If the resource fields above do land, that trade-off is probably worth it for working_set — but the aggregate counters are cheap, safe, and answer a different question.

And they answer a useful one. "How many disconnected sessions are accumulating on this server?" is the trend that leads to the memory problem described above, and it is visible from counters alone, with no resource collection involved.

Is the data already available internally?

I have no idea how the session enumeration is implemented, so this may be naive, but from the outside it looks like NSClient++ may already be holding most of these numbers:

• check_process already enumerates processes with their working_set, so the memory side might be a matter of grouping an existing enumeration by session rather than collecting anything new.
• The CheckSystem collector already samples PDH continuously into its buffer, and Windows exposes a Terminal Services Session performance object with per-session counters — so the CPU (and possibly disk/network) side might be reachable from data that is already being collected anyway.

If either of those is true, this feature is mostly about exposing existing numbers under check_users, which is what would make it worth doing. If instead it means adding a dedicated collection pass just for this, it is probably not worth the cost, and I would rather have a clear "use the alternatives" answer than a feature that makes the check expensive for everyone.

Describe alternatives you've considered

• A session id on check_process, completing what username and uid already started on master: combined with filter_perf it would let the aggregation be done on the client side. This would probably be more flexible — but it is also more work to configure for what is a fairly common question ("how much memory is this user using?").
• The Terminal Services Session PDH object via check_pdh, which does expose per-session counters. This looks workable but has two drawbacks for my scenario: the instances are identified by session, not by user name, so the result still needs to be correlated with check_users to be readable by a human; and the counter names are localized, so the configuration has to be written with expand-index to survive non-English Windows installations.
• An external PowerShell script summing WorkingSet64 per session. This works, but it means paying the cost of starting PowerShell on every scheduled run, on machines where the whole point of the built-in checks is that they are cheap.

Additional context

I mainly wanted to check whether this is something you would consider, or whether one of the alternatives above is the intended way to answer the question "how much memory is this user using?". A "not worth it" is a perfectly good answer. The two asks are independent: the perfdata counters would be useful on their own even if the resource fields are declined.

The other thing that bothered me about check_users on a terminal server, the single very long output line, was filed separately as #1370 and is already implemented — thanks for that.

Tested on 0.14.1, Windows Server 2019. The missing perfdata is still the case on current master: the four fields registered in check_users.cpp are all add_string_var() and none carries an add_perf().

[mickem]

Sounds good, but I need to investigate the impact a bit... So likely not till after NWC