Skip to content

Blob storage calls fail with 403 AuthorizationPermissionMismatch, app unhealthy

For affected Forge 3.2.7 through 3.8.3 apps, upgrade to 3.8.4 or later, re-apply the affected environment, and restart the app. The fix grants blob access to the app registration service principal that the runtime uses, instead of the User-Assigned Managed Identity (UAMI).

🚨 Symptom

Every data-plane call to Azure Blob Storage returns:

Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

The Azure_BlobServiceClient health check reports Unhealthy, which fails the App Service health probe and can pull the instance out of rotation. Look under App Service > Health check / Instances in the Azure Portal, or for a 403 span connecting to azure_blobServiceClient in Dynatrace distributed tracing.

Because the failure happens at runtime, terraform apply still reports success. A successful apply alone does not verify blob access.

Uploads or downloads proxied through APIM can also surface as an unrelated-looking 500 from the frontend, for example:

{ "status": 500, "title": "Internal Server Error", "detail": "No such host is known.", ... "errorReason": "BackendConnectionFailure" }

That specific APIM error is a separate test-app mocking/backend-routing symptom, fixed by sending x-mocking: false. Do not diagnose this RBAC issue from the APIM error alone: check for the unhealthy blob health check and the blob client's AuthorizationPermissionMismatch response.

📌 Applies to

Use this fix for saif-resources/modules/storage with enable_blob_storage = true on Forge 3.2.7 through 3.8.3. Check the version compatibility matrix before upgrading.

  • Identify the affected Terraform configuration and workspace, storage account, and App Service before applying changes.
  • Confirm the app registration service principal is the runtime identity. Do not grant more access to the UAMI to work around this mismatch.
  • Arrange permission to apply the environment and restart the affected app. If you need the temporary manual role assignment below, confirm permission to assign roles at the storage-account scope.

Before applying or restarting

Review the affected environment's planned changes and coordinate the App Service restart with its owner. Both actions change the running environment; do not run them against a different workspace or app.

✅ Fix

  1. Upgrade to Forge 3.8.4 or later.

  2. From the affected environment's Terraform configuration, re-run terraform apply with the correct workspace selected. This replaces the stale UAMI role assignment with Storage Blob Data Contributor for the app registration service principal. Contributor, not Owner, is the least-privilege role that still covers every read and write path Forge apps use; do not escalate to Owner to resolve a lingering 403.

    terraform apply

  3. Restart the App Service so the blob health check re-evaluates against the new identity.

  4. Allow time for RBAC propagation before treating a lingering 403 as a failure. Azure RBAC role assignment changes can take up to 10 minutes to take effect.

If you need blob access restored before you can upgrade, grant the app registration service principal Storage Blob Data Contributor on the storage account directly (for example with az role assignment create) as a temporary unblock. Coordinate that manual assignment with the Terraform workspace owner before applying the upgrade; do not assume the apply will remove it automatically.

🔬 Verify

Repeat the application calls that failed and check the App Service health probe. The 3.8.4 fix validation observed:

App Service "Azure_BlobServiceClient" health check reports Healthy.
A blob upload (POST) and a blob listing (GET) both return 200.

If the 403 remains after propagation, check the role assignment on the affected storage account and confirm it targets the app registration service principal before applying or restarting again.

🧠 Cause

Since Forge 3.2.7, the platform pins the runtime DefaultAzureCredential to EnvironmentCredential via the AZURE_TOKEN_CREDENTIALS and AZURE_CLIENT_ID app settings emitted by modules/identity/outputs.tf, for Aspire 13.2 compatibility. From that point on, the app's runtime data-plane identity is the app registration service principal, not the UAMI.

The storage module kept granting Storage Blob Data Owner to the UAMI. The service principal held no blob role, so requests failed while the UAMI's assignment sat unused. The corrected assignment follows the platform identity convention: UAMI for control-plane operations such as ACR pull and Key Vault/App Config references, and the app registration service principal for data-plane operations. The cosmosdb and ai-project modules already followed that convention.