Enhance ScubaGear to automatically detect if a specific policy would have passed if the conditional access policy was defined in the config

[tkol2022]

Prerequisites

• This issue has an informative and human-readable title.

💡 Summary

The number one question we get related to the Entra baseline is users saying that they failed a ScubaGear policy check even though their conditional access policy is configured exactly as instructed in the baseline document.

9 times out of 10 it is because they have exclusions (users, groups, guest, application) in their conditional access policy and they have not defined those in the configuration file.

This issue proposed some ideas to reduce the occurrence of this problem:

• Have ScubaGear automatically check if a policy would have passed if the config exclusions were defined.
• For each policy where ScubaGear detected the scenario in the first bullet, display a note to the user in the PowerShell command output and in the HTML report to make the user aware that they can rectify the problem by defining the exclusions.
• Put this information in a specific field in the JSON output in a structured way so that CISA can also track the problem in the backend Scuba Analytics.

Motivation and context

The scenario where a user configured a conditional access policy correctly but simply did not put the exclusions in the config file, causes multiple problems:

1. It causes CISA to get skewed results for policy check Pass rates in Scuba Analytics. Many of the tenants that failed certain policy checks actually configured their policies correctly but as mentioned earlier, they did not properly declare the exclusions in the config.
2. It causes a bad experience for users because they get really confused. "I am following what you told me to do, but ScubaGear keeps failing me". I lost track of how many times I have heard this.

Implementation notes

1. To be defined by the assignee.

Acceptance criteria

• Implementation items above have been completed.

[tkol2022]

Design Enhancement Suggestion

In this new design, there are two types of failures for conditional access policies, which are described below. Here are the possible Pass & Fail scenarios:

• Pass - A conditional access policy was found that matches the configuration in the baseline and all excluded Apps, Groups, Users, Guests are defined in the config file. This is the same way our current code works. No changes.
• Fail type 1 - No conditional access policy was found that matches the configuration in the baseline.
• Fail type 2 - A conditional access policy was found that matches but it contains exclusions that are not defined in the config file.

I don't have a specific proposal on how we handle the two failure types, but one approach is to add a new field to the Rego output. The policy would still Fail but the new field would contain a standard indicator describing "why" it failed. The importance of having this indicator is that:

1. On the back-end in Scuba Analytics, we can get more accurate numbers. Right now the analytics data is skewed because we have counts of policies Failing checks but we don't know if they actually have a CAP configured correctly. I have seen some tenant JSONs where this is the case and from answering BOD questions that provides more evidence.
2. The output report can use the new field to display a message to the user so that they can address the problem.

@ahuynhECS @mitchelbaker-cisa Thoughts?

[FollyBeachGurl]

this will move to the next milestone. @FollyBeachGurl