Do not block access for ALL users when the license user limit is exceeded #192

[KebabRonin]

Mostly API changes to better support this functionality in Auth Extensions.

• DefaultLicensor.java, DefaultLicenseManager.java - Some additional APIs which accept extension IDs as String instead of ExtensionId
• UserCounter.java - Added an in-memory cache of active (non-disabled) users on the instance, storing the users in the order of their creation date
• AuthExtensionUserManager.java - Added interface for auth extensions, with mostly unused methods

[KebabRonin]

How to store the users managed by an Auth extension (like AD)?

The idea for this fix would be to add a mechanism so once the user limit is exceeded for an extension handling users, newer users are disabled automatically, and re-enabled once the license is upgraded. This should be done so we do not block access to the instance.

For this, we need the list of users which are part of an extension, to manage it in different situations. On every save of a page that contains a XWiki.XWikiUsers object, check:

1. if the license user limit is exceeded -> get all AD users and block those over the limit
2. when the license user limit has been upgraded / downgraded -> get all AD users and block / unblock users
3. an user is manually enabled -> check if changes are needed for AD users
4. an AD user is manually enabled -> prevent the change

The question right now is how should we handle this list of users, how to store / mark them, in order for it to be reliable without impacting performance. We identified 3 ways:

---

Solr

The idea would be to add a custom solr field using SolrEntityMetadataExtractor (example in Admin Tools).

In short, for a page containing XWiki.XWikiUser class, we would add a custom field in SOLR in order to mention if it's a AD / EntraID / .. user (details to be decided after). Like this, when we need to get AD users we can run a fast SOLR query.

PRO:

• Should be faster and more efficient

CON:

• Depends on unpredictable (re)indexing operations - could have unexpected results until the indexing is done and users' active statuses are recomputed. E.g. The reindex operation is still in progress when a new XWikiUser page is added or modified. On this moment we would need to query how many AD users we have, if the limit is exceeded. But since solr is reindexing, we won't have complete results.

QUESTIONS

• What exactly is the result of a solr query while solr is reindexing, do we simply get the current result from the index, or an error that mentions the index is not ready?
• Can we wait for reindex to be done (it can take more than a day for some instances)?
  • Is it a problem that temporarily the license limit could not be enforced, as it will be enforced at the next user operation?
  • Could it be used as a loophole by users, to be able to add multiple users?

---

Database Query (HQL)

PRO:

• Information always reflects reality

CON:

• Need to run the same query on all subwikis
• Could be slow

---

In-Memory cache

PRO:

• Easy access from java

CON:

• Storing all users in memory may become costly for large instances - even if the cache is stored on disk when not in use
• Need to implement cache updates/invalidation

---

Implementation detail:
Each Auth extension will probably need to implement a component which describes which users are managed by an app, and which users are still allowed after a license expiration.

[michitux]

An in-memory cache for all users feels perfectly fine. As an optimization, you could disable the feature (and the cache) when the license is unlimited, which should avoid issues on huge instances. Otherwise, we control what kind of licenses we provide so we can test the memory usage for the license limits we provide. But as usernames shouldn't be more than 100 bytes, even 1 million of them wouldn't consume more than 100MB, which doesn't sound much for an instance of 1 million users.

Solr should never fail queries while indexing is running, it just returns the results from the last time commit was called. Since XWiki 16.9.0RC1, you can wait for Solr to index everything that has been submitted to the indexing queue before your call, see SolrIndex#waitReady. Note that we already index all XObject properties, it might already be possible with this to search for users managed by the AD extension without any extra attributes.

I would be very careful with automatically enabling users as you need to be careful that you don't enable users that have been intentionally disabled by the admin. Instead, I would rather suggest blocking the login itself if that's possible. I wouldn't worry so much about admins temporarily bypassing restrictions, I would more worry about user experience. When you're starting using the wiki and your first experience is that after a few seconds your user is deactivated/on the next login your user is deactivated and you don't understand why, that's bad. There should be clear and understandable error messages for the affected users.

[KebabRonin]

Since the cachedSortedUsers is populated from the database, dummy XWikiDocuments can't be used to index into the set directly (equals fails because it checks for a bunch of fields like version, author, etc.).

This is the reason why the map is needed.

[oanalavinia]

I added some comments. Some are just for small things, but there are also questions related to the functionality / architecture. I will continue the review once those are clarified, since some comments might not apply anymore after (I think it's the case now as well)

I had another question related to this functionality. Will this affect other authenticators? For example, if I also have EntraID app installed, what's the behavior?

[abrassat]

Installing the OIDC extension with this licensor version works, I didn't see any issues logging in. You can't have both OIDC and AD authenticators active at the same time, but that's expected.

OIDC has no license/user limit checks as far as I can tell, so nothing should break, you can still log in with too many users on the instance.

[oanalavinia]

Could you see how we could do some performance tests? We would need to know that this doesn't fail on an instance with a loot of users

[abrassat]

I'm guessing this depends on the JVM allocated memory. Iirc, this comment mentioned that the cache size should be ok (at least proportionally to the instance size).

I'll try to create a test with ~10.000 users on the docker instance, but I'll have to see if the test instance can really support 10k users.

[oanalavinia]

Would be nice if we could have it. I know there were some script to create multiple users / wikis
This could be done as an extra step after you finish the code part :)

[oanalavinia]

This looks ok on my side, let me know if maybe you are not sure of some aspects. Thanks for the work!

Small note that the performance tests are still to be done + there is still xwikisas/application-activedirectory#112 which is related