Skip to content

Commit d1ab335

Browse files
AshjaniAshjani
committed
docs(compliance): add manual control classification and threshold justification
1 parent 727dbb3 commit d1ab335

2 files changed

Lines changed: 160 additions & 0 deletions

File tree

‎docs/compliance/confidence_threshold_justification.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Confidence Scoring Threshold Justification
2+
3+
## What this document is
4+
5+
When an auditor uploads a screenshot or document as evidence for a
6+
manual control, AutoAudit extracts the text from that file and checks
7+
how many of the control's keywords appear in it. That gives a match
8+
percentage. This document explains how that percentage gets turned
9+
into a suggestion — pass, review, or fail — and why the bar is set
10+
differently depending on how serious the control is.
11+
12+
## The basic thresholds
13+
14+
| Match percentage | Suggestion | What it means |
15+
|---|---|---|
16+
| >= 80% | Suggest pass | Enough keywords found — the right settings page was probably captured |
17+
| 50–79% | Flag for review | Some keywords matched but not enough to be confident — auditor should take another look |
18+
| < 50% | Suggest fail | Too few keywords found — the evidence does not clearly show compliance |
19+
20+
These are starting values. The algorithm adjusts them up or down
21+
based on how serious the control is.
22+
23+
## Why severity changes the bar
24+
25+
Not every control carries the same risk if it gets marked wrong.
26+
27+
Getting a high severity control wrong in the pass direction — telling
28+
an auditor the tenant is compliant when it is not — can have serious
29+
real-world consequences. So the algorithm demands more keyword matches
30+
before it suggests pass on those controls.
31+
32+
Getting a low severity control wrong in the fail direction — flagging
33+
something as non-compliant when it actually is fine — just creates
34+
unnecessary work for the auditor. That is a much smaller problem, so
35+
the bar can be a little lower.
36+
37+
**A concrete example on the high end — control 1.1.2:**
38+
This control checks that two emergency access accounts exist. If the
39+
scoring algorithm incorrectly marks this as passed when no break-glass
40+
accounts are set up, the tenant has no fallback if all primary admin
41+
accounts get locked out. That is a genuine security crisis. The
42+
algorithm requiring a higher keyword match percentage before suggesting
43+
pass is the right call.
44+
45+
**A concrete example on the low end — control 5.1.2.5:**
46+
This control checks whether the stay signed in option is hidden on
47+
the login page. If the algorithm incorrectly flags this as failed, an
48+
admin spends five minutes double-checking a low-risk setting. That is
49+
the entire cost of getting it wrong. A lower threshold here is fine.
50+
51+
## Connection to the AutoAudit Risk Matrix
52+
53+
This approach comes directly from the risk thinking already documented
54+
in the AutoAudit Risk-Impact Prioritisation Matrix. R-06 in that
55+
matrix covers unauthenticated FastAPI endpoints — a gap that looked
56+
minor but was actually exploitable. The lesson from R-06 is that
57+
assuming something is compliant without proper verification is where
58+
real security problems start.
59+
60+
Tighter thresholds on high and critical controls exist for exactly
61+
that reason. The algorithm should be conservative when the cost of
62+
being wrong is high.
63+
64+
## Thresholds per severity level
65+
66+
| Severity | Suggest pass | Flag for review | Suggest fail |
67+
|---|---|---|---|
68+
| Critical | >= 90% | 60–89% | < 60% |
69+
| High | >= 80% | 50–79% | < 50% |
70+
| Medium | >= 70% | 40–69% | < 40% |
71+
| Low | >= 60% | 30–59% | < 30% |
72+
73+
These values are implemented in
74+
`backend-api/app/services/confidence_scorer.py` in the
75+
`PASS_THRESHOLDS` and `REVIEW_THRESHOLDS` dictionaries. If the team
76+
decides to adjust any of these values after testing, update both this
77+
document and the code at the same time so they stay in sync.

‎docs/compliance/manual_control_classification.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# CIS M365 v6.0.0 — Manual Control Classification
2+
3+
## Purpose
4+
5+
AutoAudit scans Microsoft 365 tenants automatically and checks them
6+
against the CIS M365 Foundations Benchmark v6.0.0. But not every
7+
control can be checked by code. Some settings have no API, some live
8+
in admin portals that only a human can open, and some require a
9+
judgment call that a script cannot make.
10+
11+
This document lists the 14 controls that fall into that category,
12+
explains why each one cannot be automated right now, and specifies
13+
what an auditor needs to provide as evidence.
14+
15+
Each row in the control register below maps directly to one record in
16+
the ControlVerificationTemplate table that the backend builds.
17+
Without this classification, that table has no content and auditors
18+
have no guidance when they open a pending manual control.
19+
20+
## Two sub-categories
21+
22+
Not all 14 controls are manual for the same reason. It helps to split
23+
them into two groups.
24+
25+
**Truly manual — no API exists:**
26+
1.1.2, 1.3.8, 2.2.1, 2.4.3, 5.1.2.1, 5.1.2.4, 5.1.2.5, 5.1.2.6,
27+
5.1.8.1, 5.2.4.1, 8.4.1
28+
29+
These settings either have no public API at all, are only available
30+
through internal Microsoft APIs that app registrations cannot reach,
31+
or require a human policy decision that cannot be expressed as a
32+
configuration value. They will remain manual until Microsoft exposes
33+
the relevant API surface.
34+
35+
**Manually verified for now — automation candidates:**
36+
- **7.2.8** — The SharePoint collector exists in the AutoAudit
37+
codebase but is incomplete and raises `NotImplementedError` when
38+
called. CIS actually marks this control as potentially automatable.
39+
Once a developer finishes the collector, this control can move from
40+
manual to automated with no changes needed to the template table.
41+
- **9.1.1–9.1.12** — Microsoft Fabric does have a tenant settings
42+
API. The blocker is that AutoAudit has not yet confirmed that
43+
app-only authentication works against Fabric admin endpoints. Until
44+
that is tested and working, auditors verify these 12 controls
45+
manually. They are the most likely controls to be automated next.
46+
47+
## Control register
48+
49+
| control_id | severity | service | why manual | evidence_type |
50+
|---|---|---|---|---|
51+
| 1.1.2 | high | EntraID | Which accounts are designated as break-glass is an organisational policy decision, not a config value any API can read | screenshot |
52+
| 1.3.8 | medium | Sway | Microsoft provides no API for Sway external sharing settings | screenshot |
53+
| 2.2.1 | high | EntraID | Defining which accounts to monitor is a human decision, there is no API that can confirm this is set up correctly | screenshot |
54+
| 2.4.3 | medium | Defender | MCAS configuration must be verified through the security portal , no stable API exposes its enabled state and policy configuration | screenshot |
55+
| 5.1.2.1 | medium | EntraID | The relevant endpoint only exists in the Microsoft Graph beta API, which is not stable enough for production use | screenshot |
56+
| 5.1.2.4 | medium | EntraID | This setting is only accessible through an internal Azure API that is not available to app registrations | screenshot |
57+
| 5.1.2.5 | low | EntraID | Microsoft does not expose this setting through the Graph API | screenshot |
58+
| 5.1.2.6 | low | EntraID | Same as 5.1.2.4 — internal Azure API only, no app registration access | screenshot |
59+
| 5.1.8.1 | high | EntraID | Verifying password hash sync requires checking on-premises AD Connect directly, there is no cloud API for this | screenshot |
60+
| 5.2.4.1 | medium | EntraID | SSPR settings are not exposed through the Graph API | screenshot |
61+
| 7.2.8 | medium | SharePoint | The collector exists but raises NotImplementedError, automation candidate once the collector is completed | screenshot |
62+
| 8.4.1 | medium | Teams | CIS marks this as manual; Teams app permission policy configuration is only accessible through the Teams admin portal | screenshot |
63+
| 9.1.1–9.1.12 | medium | Fabric | Fabric API auth is untested in AutoAudit, automation candidates once Fabric app-only auth is confirmed working | screenshot |
64+
65+
## Notes on the automation candidates
66+
67+
**7.2.8 — SharePoint external sharing:**
68+
The `sharepoint.spo_tenant` collector is already registered in
69+
AutoAudit but the implementation is not finished. A developer raising
70+
a `NotImplementedError` is a placeholder, not a permanent blocker.
71+
Once someone completes the collector, this control slots straight into
72+
the automated scan with no other changes needed. The template provided
73+
here covers the interim period.
74+
75+
**9.1.1–9.1.12 — Fabric tenant settings:**
76+
These twelve controls all live in the same place, the Fabric admin
77+
portal under Tenant settings. Microsoft does have an API for this,
78+
which is why these controls are marked as candidates rather than
79+
permanently manual. The problem is that AutoAudit has not yet
80+
validated that its app registration can authenticate against the
81+
Fabric admin endpoints. Once that is confirmed, all twelve controls
82+
can be automated in one go. Until then, auditors use the templates
83+
in this document to verify them manually through the portal.

0 commit comments

Comments
 (0)