Calibrant Relay Setup

The Calibrant Relay is a small service you deploy in your own Azure subscription. It handles two things for you:

• Agent Optimizer — test your Copilot Studio agents as real users (personas), including authenticated test scenarios
• Tenant Healthcheck — run M365 configuration audits using PowerShell against your tenant via Managed Identity
• Security Checkup — CIS and CISA SCuBA framework assessments with automated M365 configuration checks

The relay is a Windows VM running Windows Server 2022 Core (no desktop GUI, minimal attack surface). It uses a Managed Identity for all M365 authentication — no passwords, no service account credentials, no domain join required. The VM makes outbound-only HTTPS connections to Calibrant and Microsoft APIs. No inbound ports are open.

Deployment takes about 10–15 minutes (mostly waiting for Azure to provision the VM and install software).

Step 1 — Install the Azure CLI

If you already have the Azure CLI installed, skip to Step 2.

# Windows (PowerShell or Command Prompt)
winget install Microsoft.AzureCLI

# Mac
brew install azure-cli

# After installing, sign in:
az login

Verify you're pointed at the right subscription:

az account show --query "{name:name, id:id}" -o table

# Switch subscriptions if needed:
az account set --subscription "Your Subscription Name"

Step 2 — Create a relay in Calibrant

Go to Connections and scroll to the Calibrant Relay section. Click Add Relay, give it a name (e.g. "Production Relay"), and click Create Relay.

You'll see a Relay API Key — copy it now.

Step 3 — Create a resource group

az group create --name calibrant-relay-rg --location eastus

Pick the Azure region closest to your users — eastus, westeurope,australiaeast, etc.

Step 4 — Deploy the relay VM

Download the Bicep template and run the deployment command. Replace cal_YourKeyHere with the API key you copied in Step 2, and set a secure password (you'll almost never need it — it's just required by Azure for the VM admin account).

# Download the Bicep template
curl -O https://www.calibrant.ai/relay/main.bicep

# Deploy (takes 10-15 minutes)
az deployment group create \
  --resource-group calibrant-relay-rg \
  --template-file main.bicep \
  --parameters calibrantApiKey='cal_YourKeyHere' adminPassword='YourSecurePassword123!'

When the deployment finishes, you'll see output including a managedIdentityPrincipalId. Copy this value — you'll need it in Step 5 to grant M365 permissions.

What gets deployed:

• A Windows Server 2022 Core VM (no GUI, Standard_B2s — 2 vCPU, 4 GB RAM)
• A User-assigned Managed Identity (for M365 authentication)
• A Virtual Network with a Network Security Group (outbound-only, all inbound denied)
• A Log Analytics workspace for diagnostics
• A Custom Script Extension that automatically installs Node.js, PowerShell 7, all M365 modules, and the relay service

// Calibrant Relay — Windows VM deployment with Managed Identity
// No domain join required. No inbound ports. Outbound HTTPS only.
// Full source: https://github.com/fusedad/calibrant/blob/main/relay/deploy/main.bicep

@secure()
param calibrantApiKey string
@secure()
@minLength(12)
param adminPassword string
param calibrantApiUrl string = 'https://www.calibrant.ai'
param location string = resourceGroup().location
param pollIntervalMs string = '600000'
param adminUsername string = 'calibrant-admin'
param vmSize string = 'Standard_B2s'
param relayZipUrl string = 'https://www.calibrant.ai/relay/calibrant-relay.zip'

resource identity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'calibrant-relay-identity'
  location: location
}
// ... (NSG, VNet, NIC, VM, Extensions)
// Full template at the URL above

Step 5 — Grant M365 permissions to the Managed Identity

The relay's Managed Identity needs permission to read your M365 tenant configuration. This is a one-time setup.

Download and run the permission grant script:

↓ Download grant-healthcheck-permissions.ps1

# In a pwsh 7 terminal on your local machine (not the VM):
./grant-healthcheck-permissions.ps1

The script will open a browser for sign-in (requires a Global Administrator account), grant all necessary permissions, and print a validation report showing every permission as [OK].

For the full list of what the script grants and why, see Permission details below.

Step 6 — Verify the relay is connected

Go to Connections in Calibrant. Within 30 seconds of the bootstrap completing, the relay should show as Online with a green indicator.

If it's still showing offline after a minute, check the relay logs:

az vm run-command invoke \
  --name calibrant-relay-vm \
  --resource-group calibrant-relay-rg \
  --command-id RunPowerShellScript \
  --scripts "Get-Content C:\calibrant-relay\relay.log -Tail 30"

Step 7 — Add test personas (for Agent Optimizer)

Go to Connections → Test Personas. Personas are user accounts the relay impersonates when testing your Copilot Studio agents. Create them once and assign them to any agent.

1. Click Add Persona and enter the user's UPN (e.g. jane.doe@contoso.com)
2. Click Authenticate — a device code appears
3. The user signs in at aka.ms/devicelogin with that code
4. Name, department, and job title are auto-populated from Microsoft Graph

Viewing logs

# Last 50 lines of relay log (run from your local terminal)
az vm run-command invoke \
  --name calibrant-relay-vm \
  --resource-group calibrant-relay-rg \
  --command-id RunPowerShellScript \
  --scripts "Get-Content C:\calibrant-relay\relay.log -Tail 50"

# Check service status
az vm run-command invoke \
  --name calibrant-relay-vm \
  --resource-group calibrant-relay-rg \
  --command-id RunPowerShellScript \
  --scripts "Get-Service CalirantRelay"

Environment variables

Pre-installed tools

PowerShell modules

• ExchangeOnlineManagement — mailbox policies, DKIM, transport rules, connectors
• Microsoft.Graph — users, groups, Entra ID policies, SharePoint settings
• MicrosoftTeams — Teams policies, federation, meeting settings
• PnP.PowerShell — SharePoint site administration
• Microsoft.Online.SharePoint.PowerShell — SPO service administration

All modules authenticate automatically via Managed Identity before your script runs.

Power Platform

Power Platform checks (environment inventory, DLP policies, flow failures) use the Power Platform BAP API directly via REST. The Managed Identity is registered as a Power Platform management application during the permission grant step (Step 5), which authorizes it to call admin APIs.

Microsoft Fabric / Power BI

Fabric tenant settings checks (CIS Section 9) use the Fabric admin REST API (https://api.fabric.microsoft.com/v1/admin/tenantsettings) via Managed Identity. The grant permissions script assigns the Fabric Administrator Entra ID role. You must also configure settings in the Fabric admin portal.

Step 1: Create a security group in Microsoft Entra ID (e.g. FabricApiAccess) and add the calibrant-relay-identity service principal as a member.

Step 2: In the Fabric admin portal, enable these two settings (they are in different sections of Tenant settings):

1. Tenant settings > Developer settings > Service principals can call Fabric public APIs — enable, select "Specific security groups", and add your FabricApiAccess group
2. Tenant settings > Admin API settings > Service principals can access read-only admin APIs — enable, select "Specific security groups", and add the same group

Settings take up to 15 minutes to propagate. Fabric checks will show as "Not Evaluated" until the settings take effect.

Permission details (Step 5)

The grant script assigns 22 Microsoft Graph app roles, 1 Exchange Online app role, and 5 Entra ID directory roles:

Troubleshooting

• Relay shows Offline after deployment — the bootstrap script may still be running. It takes 10–15 minutes to install all software. Check progress in the Azure portal: go to your VM → Extensions + applications → CustomScriptExtension.
• Bootstrap failed — check the extension logs in the portal or run:

  az vm run-command invoke --name calibrant-relay-vm --resource-group calibrant-relay-rg --command-id RunPowerShellScript --scripts "Get-Content 'C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\1.10\CustomScriptHandler.log' -Tail 50"

• PowerShell cmdlets return Access Denied during healthcheck — the Managed Identity permissions may not have propagated yet. Wait 10 minutes after running the grant script and try again.
• Service starts but shows no agents in logs — check that the API key is correct. Re-run the deployment with the correct key, or update it via Run Command:

  az vm run-command invoke --name calibrant-relay-vm --resource-group calibrant-relay-rg --command-id RunPowerShellScript --scripts "[System.Environment]::SetEnvironmentVariable('CALIBRANT_API_KEY', 'cal_NewKey', 'Machine'); nssm set CalirantRelay AppEnvironmentExtra 'CALIBRANT_API_KEY=cal_NewKey'; Restart-Service CalirantRelay"