Security Checkup

Security Checkup is a framework-driven compliance assessment for your Microsoft 365 tenant. Unlike a one-time scan, it maintains a living assessment — the relay auto-detects what it can from M365, and you manually attest the rest (requirements met by other tools like Okta, Duo, CrowdStrike, etc.). The result is an always-current compliance score per framework.

How it works

Choose one or more compliance frameworks (CIS M365, NIST CSF 2.0)
Calibrant creates your assessment with all framework controls listed as Pending
Run a scan — the relay collects M365 configuration data via Managed Identity
Controls with M365-automatable checks are updated to Auto Pass or Auto Fail
For controls not auto-detected (organizational controls, or requirements met by other tools), manually attest with a note and optional tool name
Your compliance score updates as you complete attestations and re-scan

Scans update your assessment, not replace it. Manual attestations persist across scans — only the auto-detected controls are refreshed. If you attest "MFA is handled by Okta", that attestation remains until you change it.

Prerequisites

Relay deployed and online — follow the Relay Setup guide
Managed Identity permissions granted — same permissions as Tenant Healthcheck. Follow Step 4 of the Relay Setup guide.
Professional or Enterprise plan — Security Checkup is an additional product slot. Select it from your product selection in Settings.

Supported frameworks

CIS Microsoft 365 Foundations Benchmark v4.0

The most directly actionable framework for M365 — every control maps to a specific M365 configuration setting. ~59 controls across 7 sections (Admin Center, Entra ID, Exchange Online, Purview, SharePoint/OneDrive, Teams, Defender). Controls are rated Level 1 (minimum recommended) or Level 2 (high-security environments).

Approximately 44 of 59 controls are auto-detected by the relay scan. The remaining 15 are organizational controls (e.g. "define emergency access accounts") that require manual attestation.

NIST Cybersecurity Framework 2.0

The universally recognized risk management framework, updated in 2024 with a new Govern function. 41 subcategories across 6 functions: Govern (GV), Identify (ID), Protect (PR), Detect (DE), Respond (RS), Recover (RC).

Approximately 30 of 41 controls are auto-detected. The remaining 11 are organizational or process controls (incident response plans, recovery procedures, etc.) designed for manual attestation. Each NIST control also shows its cross-mapping to the relevant CIS M365 control.

Control statuses

Status	Meaning	Counts toward score?
Auto Pass	M365 scan confirmed this requirement is met	Yes (met)
Auto Fail	M365 scan confirmed this is NOT met in M365	Yes (not met)
Attested	You confirmed this is handled (possibly by another tool)	Yes (met)
Not Met	You acknowledged this is not in place	Yes (not met)
N/A	Not applicable to your organization	Excluded from score
Pending	Not yet assessed	Yes (not met)

Compliance score

The score formula is:

Score = (Auto Pass + Attested) / (Total − N/A) × 100

Pending and not-met controls count against the score, giving you an incentive to either attest or explicitly mark as N/A. Per-section scores are calculated the same way for each framework section or NIST function.

Manual attestation

When a control is not auto-detectable (or your organization satisfies it through a non-M365 tool), click the control to expand it and choose a status:

Attested — add a note describing how the requirement is met, and optionally name the tool (e.g. “Okta”, “Duo”, “CrowdStrike”). This note is included in CSV exports.
Not Met — acknowledge the gap. Useful for tracking known issues you plan to remediate.
N/A — mark controls that genuinely don't apply to your organization (e.g. CMMC controls for a non-defense organization).

For controls that auto-failed (M365 doesn't have the setting configured), you can still attest — for example, if the check finds IMAP is enabled in Exchange but your organization blocks IMAP through a third-party mail gateway, you can attest that with a note.

Credit cost

Each scan costs 3 credits regardless of how many categories are selected. Creating and updating an assessment (including manual attestations) is free.

Exporting results

From the assessment detail page, use the Export CSV button to download all controls with their current status, evidence/notes, and tool names. Columns include: Framework, Section, Control ID, Title, Level, Status, Evidence/Note, Tool.

Troubleshooting

Entra ID controls show "Unavailable" — the relay may not have the Graph permissions for Conditional Access and Identity Protection. Re-run the permission grant script and ensure Policy.Read.All and IdentityRiskyUser.Read.All are included.
Power Platform DLP shows an error — the M365 CLI login with Managed Identity requires additional Power Platform API permissions. This is optional; other categories are unaffected.
Scan stuck in "Collecting" — same as Tenant Healthcheck. Check relay logs and restart if needed.
Score didn't update after attestation — the score recalculates immediately on save. Refresh the page if the number appears stale.