[Feature Request] Fetch-level filtering in user sync — avoid downloading entire tenant

[krishnaGuptaGit]

Summary
When "User creation restriction" or "Group Object ID" is configured in the Microsoft 365 Integration sync settings, the plugin still fetches all users from the Azure AD tenant via the Graph API /users endpoint. Filtering is applied locally after all users have been downloaded. In large multi-company tenants (200K+ users), this causes severe performance issues, memory exhaustion, and API throttling — even when the admin only needs to sync a small subset of users.

Request: Apply the configured restrictions at the Microsoft Graph API level ($filter parameter and/or group members endpoint) during the fetch stage, so only relevant users are ever downloaded.

Current Behavior
Admin configuration:

"User creation restriction" is set to filter by companyName for specific companies
"Group Object ID" is configured with a security group containing only target users.

Expected Behavior

1. "User creation restriction" (plain text) should generate a $filter query
   When the admin sets a plain-text restriction like Company A on field companyName, the plugin should call:
   GET /v1.0/users?$filter=companyName eq 'Company A'&$select=...&$top=999
   For comma-separated values like Company A, Company B, Company C:
   GET /v1.0/users?$filter=companyName eq 'Company A' or companyName eq 'Company B' or companyName eq 'Company C'&$select=...&$top=999
   This way the Graph API returns only matching users — no local filtering needed.
2. "Group Object ID" should use the group members endpoint
   When a Group Object ID is configured, the plugin should call:
   GET /v1.0/groups/{group-id}/members/microsoft.graph.user?$select=...&$top=999
   Instead of:
   GET /v1.0/users?$select=...&$top=999
   This fetches only group members directly, not the entire tenant.
3. Regex restrictions should use a hybrid approach
   When "Value is a regular expression" is enabled, the plugin should attempt to extract literal components from the regex pattern and build a partial $filter to reduce the dataset, then apply the full regex locally on the smaller result set.
   For example, /^Company (A|B|C)$/ on companyName could generate startsWith(companyName, 'Company') as a partial server-side filter, then verify the full regex match locally.
   If no filterable component can be extracted, fall back to current behavior but log a performance warning.
4. Combined filters should work together
   When both Group Object ID and User creation restriction are set, the plugin should fetch from the group members endpoint AND apply the $filter:
   GET /v1.0/groups/{group-id}/members/microsoft.graph.user?$filter=companyName eq 'Company A'&$select=...

Impact
This issue affects any organization that:

Has a large Azure AD tenant with multiple companies/divisions
Needs to sync only a subset of users to Moodle
Experiences sync timeouts, memory errors, or slow cron execution

The performance improvement would be proportional to the filtering ratio. In our case: ~93% reduction in API calls, data transfer, and sync time (214K → 15K users).

Error handling
All filter-related errors should fall back gracefully to current behavior (fetch all + local filter). The sync process should never fail completely due to a filter issue.
Delta sync
The same filtering should apply to delta queries. If filter settings change between syncs, the delta token should be invalidated and a fresh filtered full sync should run.
Backward compatibility
When no restrictions or group IDs are configured, behavior remains identical to the current version.

Workarounds Attempted
Workaround Result
Set "User creation restriction" to company names Users still fetched at API level — filter only applied locally after download
Set "Group Object ID" with a security group All 214K users still fetched — group membership only checked after download
Dynamic security group in Azure AD Group works correctly in Azure, but plugin ignores it at fetch stage

Additional Context
Microsoft Graph API fully supports $filter on the /users endpoint for properties like companyName, department, mail, userPrincipalName, and others
The /groups/{id}/members endpoint is the standard way to fetch group members without querying the full directory
Both endpoints support pagination via @odata.nextLink, which preserves query parameters including $filter
Delta queries (/users/delta) also support $filter on select properties

I am happy to contribute a pull request for this if the maintainers agree with the approach.

[weilai-irl]

Hi @krishnaGuptaGit

Thank you for making the suggestion.

We are in the final stage in the 202601 release testing at the moment. There's a code freeze, so this change won't be part of this release even if it's ready today. I'll revisit this after the release. It should be out in the next few weeks.

BTW, one of the biggest changes in the upcoming 202601 release is an attempt to improve the performance of the user sync task. The changes will reduce task running time, memory use, Graph API call amounts and payload size, as well as Moodle DB load. The work is carried out as part of #2930. Please keep an eye on it too and see if it solves at least some of your issue.

Regards,
Lai

[weilai-irl]

Hi @krishnaGuptaGit

I have made an initial implementation of this feature. The way it was done is to add a new setting to store the object ID of the group that contains the M365 users to sync. This setting is used in the main user sync task, as well as the photo/timezone sync task. The reason that the existing "User creation restriction" setting is not used is because the setting applies only to account creation - a user can still exist in Moodle even if it doesn't meeting the "User creation restriction" setting, e.g. created manually, uploaded from CSV etc.

Unfortunately this feature cannot be included in this upcoming 2026-02 release due to quite a few other changes updated the same files, making merging changes in this one infeasible. This will be merged and tested in the next release.

Regards,
Lai