plugin: reset active limit counts for associations with an RPC?

[cmoussa1]

I was thinking this morning about how it might be sort of nice to have some sort of functionality in the plugin to reset the active limit counts for all of the associations without having to unload and reload the plugin. This could be a sort of "break glass in case of emergency" type functionality offered by the plugin, but probably not needed to be used very often at all. It could work something like:

1. An RPC gets issued to the plugin that signals that all of the active limit counts (current running/active jobs, current resources) should be reset for all associations.
2. The plugin iterates through all of the associations and sets all of the active limit counts to 0.

You would probably need to be very careful here though, because if there are active/running jobs when this is executed, and the plugin does not check that the current counts are > 0, it could potentially reduce the count to negative numbers and mess them up.

I haven't fully fleshed this idea out yet, but just jotting down some quick thoughts. Will need to do some experimenting here.

[grondo]

This is a good idea, but as you mention, I'm not sure it could be done in a way that isn't susceptible to races.

If there was a way for a jobtap plugin to synchronously get a list of active jobs, then you could recalculate the limits on demand. Maybe I'll open an issue in flux-core if you think that would be useful.

[cmoussa1]

Oh that's an interesting thought! Yeah, maybe if also during the reset, it could recalculate the limits.

The other thought is that there are flux-accounting-specific dependencies added/removed on jobs depending on these counts, so potentially another wrinkle to think about when recalculating/resetting these counts.