Security Checkup

Security Checkup is a framework-driven compliance assessment for your Microsoft 365 tenant. Unlike a one-time scan, it maintains a living assessment — the relay auto-detects what it can from M365, and you manually attest the rest (requirements met by other tools like Okta, Duo, CrowdStrike, etc.). The result is an always-current compliance score per framework.

How it works

Choose one or more compliance frameworks (CIS M365 v6.0.1, CIS D365/Power Platform, CISA SCuBA M365 Baselines)
Calibrant creates your assessment with all framework controls listed as Pending
Run a scan — the relay collects M365 configuration data via Managed Identity
Controls with M365-automatable checks are updated to Auto Pass or Auto Fail
For controls not auto-detected (organizational controls, or requirements met by other tools), manually attest with a note and optional tool name
Your compliance score updates as you complete attestations and re-scan

Scans update your assessment, not replace it. Manual attestations persist across scans — only the auto-detected controls are refreshed. If you attest "MFA is handled by Okta", that attestation remains until you change it.

Prerequisites

Relay deployed and online — follow the Relay Setup guide
Managed Identity permissions granted — same permissions as Tenant Healthcheck. Follow Step 4 of the Relay Setup guide.
Professional or Enterprise plan — Security Checkup is an additional product slot. Select it from your product selection in Settings.

Supported frameworks

CIS Microsoft 365 Foundations Benchmark v6.0.1

The most directly actionable framework for M365 — every control maps to a specific M365 configuration setting. ~140 controls across 9 sections (Admin Center, Defender, Purview, Intune, Entra ID, Exchange, SharePoint, Teams, Microsoft Fabric). Controls are rated Level 1 (minimum recommended) or Level 2 (high-security environments). You can target L1 or L2 from the assessment settings.

Approximately 70 of 140 controls are auto-detected by the relay scan. The remaining 70 are organizational controls (e.g. "define emergency access accounts", PIM configuration, access reviews) that require manual attestation.

CIS Dynamics 365 / Power Platform Benchmark v1.0.0

A separate CIS benchmark covering Microsoft Dynamics 365 and Power Platform environments. 15 controls across 4 sections (Accounts & Authentication, Permissions, Data Management, Logging & Auditing). All controls are manual attestation with the exception of DLP policy checks which use the Power Platform BAP API.

CISA SCuBA M365 Security Baselines

CISA's Secure Cloud Business Applications (SCuBA) project provides security configuration baselines for Microsoft 365. 134 controls across 8 M365 products: Entra ID, Defender, Exchange Online, Power Platform, SharePoint Online, OneDrive, Teams, and more.

Approximately 100 of 134 controls are auto-detected by the relay scan. The remaining controls are organizational or process controls designed for manual attestation. Each SCuBA control also shows its cross-mapping to the relevant CIS M365 control.

Control statuses

Status	Meaning	Counts toward score?
Auto Pass	M365 scan confirmed this requirement is met	Yes (met)
Auto Fail	M365 scan confirmed this is NOT met in M365	Yes (not met)
Attested	You confirmed this is handled (possibly by another tool)	Yes (met)
Not Met	You acknowledged this is not in place	Yes (not met)
N/A	Not applicable to your organization	Excluded from score
Pending	Not yet assessed	Yes (not met)

Compliance score

The score formula is:

Score = (Auto Pass + Attested) / (Total − N/A) × 100

Pending and not-met controls count against the score, giving you an incentive to either attest or explicitly mark as N/A. Per-section scores are calculated the same way for each framework section or SCuBA product baseline.

Manual attestation

When a control is not auto-detectable (or your organization satisfies it through a non-M365 tool), click the control to expand it and choose a status:

Attested — add a note describing how the requirement is met, and optionally name the tool (e.g. “Okta”, “Duo”, “CrowdStrike”). This note is included in CSV exports.
Not Met — acknowledge the gap. Useful for tracking known issues you plan to remediate.
N/A — mark controls that genuinely don't apply to your organization (e.g. CMMC controls for a non-defense organization).

For controls that auto-failed (M365 doesn't have the setting configured), you can still attest — for example, if the check finds IMAP is enabled in Exchange but your organization blocks IMAP through a third-party mail gateway, you can attest that with a note.

Credit cost

Each scan costs 3 credits regardless of how many categories are selected. Creating and updating an assessment (including manual attestations) is free.

Exporting results

From the assessment detail page, use the Export CSV button to download all controls with their current status, evidence/notes, and tool names. Columns include: Framework, Section, Control ID, Title, Level, Status, Evidence/Note, Tool.

Troubleshooting

Entra ID controls show "Unavailable" — the relay may not have the Graph permissions for Conditional Access and Identity Protection. Re-run the permission grant script and ensure Policy.Read.All and IdentityRiskyUser.Read.All are included.
Power Platform DLP shows an error — the Managed Identity must be registered as a Power Platform management application via the grant permissions script. This is optional; other categories are unaffected.
Scan stuck in "Collecting" — same as Tenant Healthcheck. Check relay logs and restart if needed.
Score didn't update after attestation — the score recalculates immediately on save. Refresh the page if the number appears stale.