--- id: SAIFTRBL0004 moved_from: - guides/troubleshooting/blob-storage-403-authorizationpermissionmismatch.md title: "Blob storage calls fail with 403 AuthorizationPermissionMismatch, app unhealthy" description: Restore runtime blob access for affected Forge apps by upgrading to 3.8.4 or later, applying the corrected role assignment, and restarting the app. tags: - troubleshooting - terraform --- # Blob storage calls fail with 403 AuthorizationPermissionMismatch, app unhealthy For affected Forge **3.2.7 through 3.8.3** apps, upgrade to **3.8.4 or later**, re-apply the affected environment, and restart the app. The fix grants blob access to the app registration service principal that the runtime uses, instead of the User-Assigned Managed Identity (UAMI). --- ## 🚨 Symptom Every data-plane call to Azure Blob Storage returns: ```text Status: 403 (This request is not authorized to perform this operation using this permission.) ErrorCode: AuthorizationPermissionMismatch ``` The `Azure_BlobServiceClient` health check reports `Unhealthy`, which fails the App Service health probe and can pull the instance out of rotation. Look under **App Service > Health check / Instances** in the Azure Portal, or for a 403 span connecting to `azure_blobServiceClient` in Dynatrace distributed tracing. Because the failure happens at runtime, `terraform apply` still reports success. A successful apply alone does not verify blob access. Uploads or downloads proxied through APIM can also surface as an unrelated-looking 500 from the frontend, for example: ```text { "status": 500, "title": "Internal Server Error", "detail": "No such host is known.", ... "errorReason": "BackendConnectionFailure" } ``` That specific APIM error is a separate test-app mocking/backend-routing symptom, fixed by sending `x-mocking: false`. Do not diagnose this RBAC issue from the APIM error alone: check for the unhealthy blob health check and the blob client's `AuthorizationPermissionMismatch` response. --- ## 📌 Applies to Use this fix for `saif-resources/modules/storage` with `enable_blob_storage = true` on Forge 3.2.7 through 3.8.3. Check the [version compatibility matrix](../reference/version-compatibility.md) before upgrading. - Identify the affected Terraform configuration and workspace, storage account, and App Service before applying changes. - Confirm the app registration service principal is the runtime identity. Do not grant more access to the UAMI to work around this mismatch. - Arrange permission to apply the environment and restart the affected app. If you need the temporary manual role assignment below, confirm permission to assign roles at the storage-account scope. !!! warning "Before applying or restarting" Review the affected environment's planned changes and coordinate the App Service restart with its owner. Both actions change the running environment; do not run them against a different workspace or app. --- ## ✅ Fix 1. Upgrade to Forge **3.8.4** or later. 2. From the affected environment's Terraform configuration, re-run `terraform apply` with the correct workspace selected. This replaces the stale UAMI role assignment with `Storage Blob Data Contributor` for the app registration service principal. Contributor, not Owner, is the least-privilege role that still covers every read and write path Forge apps use; do not escalate to Owner to resolve a lingering 403. ```powershell terraform apply ``` 3. Restart the App Service so the blob health check re-evaluates against the new identity. 4. Allow time for RBAC propagation before treating a lingering 403 as a failure. [Azure RBAC role assignment changes can take up to 10 minutes to take effect](https://learn.microsoft.com/azure/role-based-access-control/troubleshooting#symptom---role-assignment-changes-are-not-being-detected). If you need blob access restored before you can upgrade, grant the app registration service principal `Storage Blob Data Contributor` on the storage account directly (for example with `az role assignment create`) as a temporary unblock. Coordinate that manual assignment with the Terraform workspace owner before applying the upgrade; do not assume the apply will remove it automatically. --- ## 🔬 Verify Repeat the application calls that failed and check the App Service health probe. The [3.8.4 fix validation](../release-notes/3.8.4.md) observed: ```text App Service "Azure_BlobServiceClient" health check reports Healthy. A blob upload (POST) and a blob listing (GET) both return 200. ``` If the 403 remains after propagation, check the role assignment on the affected storage account and confirm it targets the app registration service principal before applying or restarting again. --- ## 🧠 Cause Since Forge 3.2.7, the platform pins the runtime `DefaultAzureCredential` to `EnvironmentCredential` via the `AZURE_TOKEN_CREDENTIALS` and `AZURE_CLIENT_ID` app settings emitted by `modules/identity/outputs.tf`, for Aspire 13.2 compatibility. From that point on, the app's runtime data-plane identity is the **app registration service principal**, not the UAMI. The `storage` module kept granting `Storage Blob Data Owner` to the UAMI. The service principal held no blob role, so requests failed while the UAMI's assignment sat unused. The corrected assignment follows the platform identity convention: UAMI for control-plane operations such as ACR pull and Key Vault/App Config references, and the app registration service principal for data-plane operations. The `cosmosdb` and `ai-project` modules already followed that convention. --- ## 📚 Related - [PR #1027: fix(terraform): grant storage Blob Data Contributor to app registration SP](https://github.com/saif-corp/forge/pull/1027) - [3.8.4 release notes](../release-notes/3.8.4.md) - [Composable Terraform resources: identity conventions](https://github.com/saif-corp/forge/blob/main/docs/_internal/composable-terraform-resources.md) - [Storage account dev guide](../build/data/storage-account.md)