Scanning
This guide covers how to start a scan, what happens during execution, how scores are calculated, and how to read and export your results.
Starting a scan
- Navigate to Healthcheck in the dashboard
- Ensure the prerequisites banner shows your relay is online
- Select the categories you want to scan (all are selected by default)
- Click Launch Scan
Each scan costs 2 credits regardless of how many categories are selected. You can run scans on individual categories if you want to check a specific area quickly.
What happens during a scan
- Command dispatch — Calibrant generates PowerShell scripts for each selected category and sends them as commands to your relay
- Relay execution — the relay runs each script using its Managed Identity, collecting configuration data from Microsoft Graph, Exchange Online, Teams, SharePoint, and Power Platform APIs
- Data return — raw results are returned to Calibrant and parsed into structured findings
- Evaluation — each finding is evaluated against health rules and assigned a status (pass, fail, not licensed, or not evaluated)
- AI analysis — Claude generates an executive summary and prioritized remediation recommendations
Most scans complete in 30–90 seconds depending on how many categories are selected. You can watch progress in real time — the UI shows which categories have finished collecting.
Understanding scores
Scores range from 0 to 100 and are calculated per-category and overall. The scoring system is severity-weighted — not all checks count equally. Critical and high-severity findings carry significantly more weight than low-severity issues, reflecting their real-world impact on your tenant's security and operational posture.
How weighting works
Every health rule has a severity level. When your score is calculated, each passing or failing check contributes points proportional to its severity. A single critical failure won't tank your entire score, but multiple high-severity failures will pull it down meaningfully. Low-severity items still matter, but they won't overshadow more important findings.
| Severity | Impact | Examples |
|---|---|---|
| Critical | Highest impact | Modern auth disabled |
| High | Significant impact | Too many global admins, anonymous sharing open |
| Medium | Moderate impact | Guest invite restrictions not set, IMAP enabled |
| Low | Minor impact | No org branding, empty distribution groups |
| Info | No impact | Informational observations (license inventory, org name) |
What's excluded from scoring
Your score only reflects checks that were actually evaluated. The following are excluded from both the numerator and denominator — they don't help or hurt your score:
- Not licensed — the feature requires a license tier your tenant doesn't have (e.g., Conditional Access requires Entra ID P1)
- Not evaluated — data couldn't be collected, usually due to a missing permission or transient API error
- Info-only checks — informational observations with no pass/fail criteria
This means you are never penalized for features you don't own or for checks that couldn't run. Your score reflects your actual, actionable configuration gaps.
Score ranges
| Score | Label | Guidance |
|---|---|---|
| 90–100 | Excellent | Tenant is well-configured. Address remaining low-severity items at your pace. |
| 80–89 | Good | Strong posture with a few gaps. Review high-severity failures first. |
| 60–79 | Needs Attention | Multiple meaningful gaps. Prioritize critical and high-severity remediations. |
| 40–59 | Poor | Significant configuration gaps. Address critical findings immediately. |
| 0–39 | Critical | Major configuration issues across multiple areas. Urgent remediation recommended. |
Reading results
After a scan completes, the results page shows:
- Scan coverage — how many checks were evaluated, passed, failed, and skipped
- Overall score — your aggregate score across all categories
- Category scores — individual scores for each scanned category (click a category to filter findings)
- AI executive summary — a generated analysis of your tenant's health posture
- Prioritized recommendations — AI-generated action items ranked by effort and impact
Finding statuses
| Status | Meaning |
|---|---|
| Pass | Check passed — this configuration meets best practices |
| Fail | Check failed — review the remediation steps to fix this |
| Not Licensed | Feature requires a license your tenant doesn't have |
| Not Evaluated | Data couldn't be collected — check relay permissions |
Use the filter tabs to narrow findings by status (Failing, Passing, Not Evaluated, Not Licensed) or by category. Expand any finding to see detailed information and remediation guidance.
Exporting results
Two export formats are available from the results page:
- CSV — spreadsheet-friendly export of all findings with rule ID, category, severity, status, message, and remediation
- Raw JSON — the full relay output logs for each category, useful for auditing or integration with other tools
Troubleshooting
- Relay shows offline on the Healthcheck page — go to Connections and check that your relay deployment has a green Online indicator. If it's offline, view the relay logs via Azure Run Command:
az vm run-command invoke --name calibrant-relay-vm --resource-group calibrant-relay-rg --command-id RunPowerShellScript --scripts "Get-Content C:\calibrant-relay\relay.log -Tail 30" - Scan fails with “Access Denied” — the Managed Identity is missing one or more permissions. Re-run the permission grant script and wait 5–10 minutes for propagation.
- Exchange or Teams category fails but others pass — the directory roles (Exchange Administrator, Teams Administrator) may not have propagated yet. Wait 10 minutes and retry.
- Power Platform shows no data — the relay's Managed Identity must be registered as a Power Platform management application. See Step 4 of the Relay Setup guide for the BAP API registration step. This category is optional; skipping it does not affect other category scores.
- Scan stuck in “Collecting” — relay commands expire after 10 minutes. If the relay goes offline mid-scan, the scan will be marked failed automatically. Check relay logs and restart the service if needed:
az vm run-command invoke --name calibrant-relay-vm --resource-group calibrant-relay-rg --command-id RunPowerShellScript --scripts "Restart-Service CalirantRelay"