Primary Local + Azure Gateway Runbook¶

Operator runbook for the validated hybrid topology: local Primary + AKS Gateway

Document Version: 1.0.0
Last Updated: 2026-04-13
License Tier: Enterprise+

1. Scope¶

This runbook describes the exact deployment model where:

Primary backend runs locally (Docker Compose)
Gateway runs in Azure AKS (MAZEVAULT_MODE=gateway)
Gateway bootstrap and runtime are controlled by the Primary
Azure uses Managed Identity + Workload Identity for Key Vault secrets

2. Critical behavior: local vs Azure¶

docker compose -f docker-compose.yml up --build rebuilds and starts local containers only.

It does not update Azure.

To propagate code/config to Azure gateway, you must:

Build and push image to ACR
Apply Kubernetes manifests or Helm values to AKS
Restart rollout and verify pod runtime

3. Mandatory Entra callback¶

Use this redirect URI in the Entra App Registration used by MazeVault SSO:

https://<BACKEND_OR_PUBLIC_API_HOST>/api/v1/auth/sso/entra/callback

Using /auth/entra/callback causes redirect mismatch in this architecture.

4. Bootstrap token from Primary¶

TOKEN=$(curl -sk -X POST https://PRIMARY_URL/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@mazevault.com","password":"YOUR_PASSWORD"}' | jq -r '.token')

curl -sk -X POST https://PRIMARY_URL/api/v1/gateways/bootstrap \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"azure-gateway-gwc-01"}'

Save the returned token as GATEWAY_BOOTSTRAP_TOKEN.

5. Build and publish gateway image¶

docker build -t acrmzvtest.azurecr.io/mazevault-backend:latest \
  -f maze-core/backend/Dockerfile \
  maze-core/backend

docker push acrmzvtest.azurecr.io/mazevault-backend:latest

6. Managed Identity + Workload Identity (required)¶

6.1 Enable AKS OIDC + workload identity¶

az aks update \
  --resource-group rg-mazevault-<env> \
  --name aks-mazevault-<env> \
  --enable-oidc-issuer \
  --enable-workload-identity

6.2 Federated credential subject must match service account exactly¶

Gateway service account in this deployment:

system:serviceaccount:mazevault:mazevault-gateway-sa

Create/update federated credential:

az identity federated-credential create \
  --identity-name id-mazevault-<env>-<region> \
  --resource-group rg-mazevault-<env> \
  --name fic-mazevault-gateway-sa \
  --issuer <AKS_OIDC_ISSUER_URL> \
  --subject system:serviceaccount:mazevault:mazevault-gateway-sa \
  --audiences api://AzureADTokenExchange

If subject does not match, AKS pod mount fails with AADSTS700213.

7. Key Vault secret delivery method¶

Use Secrets Store CSI Driver + SecretProviderClass for gateway runtime secrets.

kubectl apply -f maze-core/deploy/k8s/azure-test/gateway-secrets-provider.yaml
kubectl apply -f maze-core/deploy/k8s/azure-test/gateway-deployment.yaml
kubectl rollout restart deployment/mazevault-gateway-npr -n mazevault
kubectl rollout status deployment/mazevault-gateway-npr -n mazevault --timeout=300s

Verify sync:

kubectl get secretproviderclass -n mazevault
kubectl get secretproviderclasspodstatus -n mazevault
kubectl get secret mazevault-gateway-config -n mazevault

8. Service Principal for provisioning (Terraform/CI)¶

Use a dedicated Service Principal for infrastructure provisioning only.

Required minimum roles at subscription or resource-group scope:

Contributor
User Access Administrator

Required Azure provider registrations (one-time per subscription):

az provider register --namespace Microsoft.ContainerService --wait
az provider register --namespace Microsoft.KeyVault --wait
az provider register --namespace Microsoft.DBforPostgreSQL --wait
az provider register --namespace Microsoft.Cache --wait
az provider register --namespace Microsoft.ContainerRegistry --wait
az provider register --namespace Microsoft.ManagedIdentity --wait

8.1 OBO user RBAC (required for Azure discovery)¶

For users signed in via Entra SSO (OBO), assign:

Reader on each subscription that should be discoverable
Key Vault Secrets User (or Secrets Officer for write scenarios) on each target Project Key Vault

Example:

az role assignment create \
  --role "Reader" \
  --assignee <user-object-id-or-upn> \
  --scope /subscriptions/<subscription-id>

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee <user-object-id-or-upn> \
  --scope /subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault-name>

Without subscription Reader, OBO discovery endpoints can return HTTP 200 with zero subscriptions.

9. Post-deployment checks¶

kubectl get pods -n mazevault -l app=mazevault-gateway
kubectl logs -n mazevault deploy/mazevault-gateway-npr --tail=120

Expected behavior:

Gateway starts in gateway mode
Local license onboarding is skipped
Polling to primary tasks endpoint returns HTTP 200

10. Troubleshooting quick map¶

AADSTS700213: federated credential subject mismatch (mazevault-gateway-sa vs another SA)
401 on gateway tasks: verify gateway token middleware compatibility and headers
missing gateway_write_queues: apply DB migration 000111_gateway_write_queues
stale gateway health noise: remove old gateway records from primary DB and keep one active gateway
ManagedIdentityCredential timeout to 169.254.169.254 on local Primary: expected when MI is tested from Docker on localhost (no IMDS)
automatic fallback from local Primary integration test/sync to Azure gateway: not available in current release