Pipeline Troubleshooting

Common Azure DevOps pipeline issues and solutions for Forge projects.

---

📋 Overview

This guide covers common pipeline failures in Azure DevOps and how to diagnose and resolve them.

Looking for what a pipeline does?

This guide is about fixing failures. For what each pipeline deploys, what triggers it, and which pipeline to run for a given change, see the Pipelines reference.

---

🏗️ Pipeline Architecture

Forge Pipeline Structure

A repository contains only the files its templates added. The full set for an application repository, with the condition under which each is present:

.azdo/
├── azure-pipelines-api.yml             # API deploy            — API
├── azure-pipelines-api-pr.yml          # API PR validation     — API
├── azure-pipelines-auth.yml            # Auth deploy (manual)  — API
├── azure-pipelines-auth-pr.yml         # Auth PR validation    — API
├── azure-pipelines-web.yml             # Web deploy            — front end
├── azure-pipelines-web-pr.yml          # Web PR validation     — front end
├── azure-pipelines-sub.yml             # Subscription deploy   — event subscription
├── azure-pipelines-sub-pr.yml          # Subscription PR       — event subscription
├── azure-pipelines-pr-slot-cleanup.yml # PR slot teardown      — API or standalone web
├── azure-docs.yml                      # Docs publish (manual) — docs site
├── azure-docs-pr.yml                   # Docs PR validation    — docs site
├── dependabot-scan.yml                 # Daily scan (cron)     — API services
└── vars/
    ├── base.yml                        # Shared variables      — always
    ├── api.yml                         # API variables         — API
    ├── auth.yml                        # Auth variables        — API
    ├── web.yml                         # Web variables         — front end
    ├── sub.yml                         # Subscription variables— event subscription
    └── docs.yml                        # Docs variables        — docs site

Event service and package repositories use a different layout — a single azure-pipelines.yml and azure-pipelines-pr.yml pair. See the Pipelines reference for the full inventory by repository type.

Pipeline Templates

Forge uses centralized templates from SAIF/pipeline-templates, referenced as refs/heads/releases/v3:

Template	Purpose
azure-dotnet-api-v3.yml	.NET API build and deploy
azure-dotnet-api-pr-v3.yml	.NET API PR validation
azure-react-web-v3.yml	React web app build and deploy
azure-react-web-pr-v3.yml	React web PR validation
azure-auth.yml	Auth configuration deployment
azure-auth-pr.yml	Auth PR validation
azure-dotnet-sub-v2.yml	Event subscription deploy
azure-dotnet-sub-pr-v2.yml	Event subscription PR validation
azure-docs-v2.yml	Documentation site publish
azure-docs-pr-v2.yml	Documentation PR validation
azure-pr-slot-cleanup.yml	PR deployment slot teardown

---

❌ Common Pipeline Failures

1. Template Reference Errors

Error: Template reference not found or Unable to find template

Cause: Missing or incorrect ref for templates repository

Solution:

# Correct template reference
resources:
  repositories:
    - repository: templates
      type: git
      name: SAIF/pipeline-templates
      ref: refs/heads/releases/v3  # Must specify version

Known issue: event-service PR pipeline template not found

Repositories scaffolded from saif-event-service on Forge 3.8.2 or earlier may reference azure-event-service-pr-v2.yml@templates, which does not exist in SAIF/pipeline-templates. The Forge template source has been corrected for future scaffolds, but existing repositories keep their generated file. In the extends: block of .azdo/azure-pipelines-pr.yml, change template: azure-event-service-pr-v2.yml@templates to template: azure-event-service-pr.yml@templates.

2. Variable Group Access

Error: Variable group 'X' is not authorized for use

Cause: Pipeline not authorized to access variable group

Solution:

1. Go to Azure DevOps → Library → Variable Groups
2. Click on the variable group
3. Go to "Pipeline permissions"
4. Authorize the pipeline

3. Agent Pool Issues

Error: No agent pool found or No hosted parallelism

Cause: Agent pool not available or parallelism quota exhausted

Solution:

# Use hosted agent
pool:
  vmImage: 'ubuntu-latest'

# Or specific agent pool
pool:
  name: 'SAIF-AgentPool'

4. .NET SDK Not Found

Error: SDK 'Microsoft.NET.Sdk' not found

Cause: .NET SDK version not installed on agent

Solution:

# Ensure UseDotNet task runs first
- task: UseDotNet@2
  inputs:
    version: '10.x'
    includePreviewVersions: true

5. Node.js Version Mismatch

Error: node: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_X.XX' not found

Cause: Node.js version incompatible with agent OS

Solution:

# Use UseNode task
- task: UseNode@1
  inputs:
    version: '22.x'

6. Docker Build Failures

Error: Cannot connect to Docker daemon

Cause: Docker service not running or insufficient permissions

Solution:

1. Verify agent has Docker installed
2. Check service account has Docker permissions
3. Use hosted agent with Docker pre-installed

7. Terraform State Lock

Error: Error locking state or state is locked by another process

Cause: Previous pipeline run crashed without releasing lock

Solution:

# Force unlock (with caution)
terraform force-unlock <lock-id>

# Or wait for lock timeout

8. Artifact Publishing Failures

Error: Failed to publish artifact

Cause: Artifact path doesn't exist or permissions issue

Solution:

# Ensure build produces artifacts
- task: PublishBuildArtifacts@1
  inputs:
    pathToPublish: '$(Build.ArtifactStagingDirectory)'
    artifactName: 'drop'

---

🔍 Diagnostic Approaches

1. Enable Verbose Logging

variables:
  System.Debug: true

2. Check Pipeline Logs

Look for these key sections:

• Initialize job - Agent setup issues
• Checkout - Repository access issues
• Build tasks - Compilation errors
• Test tasks - Test failures
• Deploy tasks - Deployment errors

3. Run Locally

Reproduce the issue locally:

# Simulate pipeline environment
$env:BUILD_BUILDNUMBER = "1.0.0"
$env:BUILD_SOURCESDIRECTORY = (Get-Location).Path

# Run the same commands
dotnet build
dotnet test

4. Check Terraform State

# List workspaces
terraform workspace list

# Check state
terraform state list
terraform show

---

🔎 Reading Build Logs from the CLI

Use saif pipeline logs <build> to stream build log output straight into your terminal — no browser round-trip needed. The <build> argument is located in one of two modes, and the CLI auto-detects which:

Mode	You pass…	Context comes from…	Use when
Remote	A full build URL	The URL itself (self-contained)	Someone pasted a build link from Teams, an alert, or the browser
Contextual	A bare build id	--repo → Flags → the cwd repo clone → prompt	You're investigating a build for your own service

Remote mode (by URL)

Paste the build URL exactly as it appears in the browser. It already carries the host, collection, project, and build id, so context flags are not valid in this mode (passing --host, --collection, --project, or --repo with a URL is a usage error):

saif pipeline logs "https://dev.azure.com/SAIFCorporation/Customer/_build/results?buildId=162774"

Contextual mode (by id)

A bare build id is only meaningful relative to a repo/pipeline context. Run it from inside a repository clone and the host/collection are inferred automatically from the git remote:

# Inside a clone — context inferred from the git remote
saif pipeline logs 162774

# Outside a clone — name the repo, or supply coordinates explicitly
saif pipeline logs 162774 --repo my-service
saif pipeline logs 162774 --host dev.azure.com --collection SAIFCorporation

Filtering and output

Narrow the output to the part of the build you care about. Filters are case-insensitive and combinable:

# Only the failed steps (the analog of gh's --log-failed)
saif pipeline logs 162774 --status failed

# Scope to a stage, job, or step
saif pipeline logs 162774 --stage Build --step Compile

# Last 50 lines, or raw structured output for scripting
saif pipeline logs 162774 --tail 50
saif pipeline logs 162774 --format json

saif pipeline monitor <build-id> shares the same context resolution, so a bare build id is likewise inferred from the cwd repo clone when you run it from inside one.

---

📊 Stage-Specific Issues

Build Stage

Issue	Symptom	Resolution
Restore fails	NU1101: Unable to find package	Check nuget.config, verify feed access
Build fails	CSxxxx error	Fix code issue, check package versions
Test fails	XUnit test failed	Review test output, check test dependencies

Deploy Stage

Issue	Symptom	Resolution
Slot swap fails	Slot busy	Retry or check App Service status
Config update fails	KeyVault access denied	Check managed identity permissions
Health check fails	503 Service Unavailable	Check app startup, review logs

Auth Stage

Issue	Symptom	Resolution
Okta API error	401 Unauthorized	Rotate Okta API credentials
Entra ID error	Insufficient privileges	Check service principal permissions
Role assignment fails	Principal not found	Verify user/group exists

---

🌍 Environment-Specific Issues

Development (DEV)

• More permissive, may have different variable values
• Uses non-prod Okta tenant
• Terraform workspaces suffixed with -dev

UAT

• Mirrors production configuration
• May have restricted access
• Requires approval gates

Production (PROD)

• Strict approval requirements
• Uses production Okta tenant
• Blue-green deployment slots
• Extended health check timeouts

---

🔄 Recovery Patterns

Failed Deployment Rollback

# Pipeline includes rollback logic
- task: AzureFunctionApp@1
  inputs:
    deployToSlotOrASE: true
    slotName: 'staging'
    # If health check fails, no slot swap occurs

Terraform State Recovery

# Import existing resource
terraform import azurerm_storage_account.main /subscriptions/.../storageAccounts/xxx

# Remove orphaned state
terraform state rm azurerm_storage_account.old

Retry Failed Stage

1. Go to pipeline run
2. Click on failed stage
3. Click "Rerun failed jobs"

---

📝 Pipeline Variables Reference

Built-in Variables

Variable	Description
$(Build.BuildNumber)	Pipeline build number
$(Build.SourceBranch)	Git branch
$(Build.Repository.Name)	Repository name
$(System.DefaultWorkingDirectory)	Agent working directory

Forge Variables

Variable	Description
$(ProjectId)	Forge project identifier
$(Environment)	Deployment environment
$(IsProduction)	Boolean for prod checks

---

🛡️ Prevention Strategies

1. Pin Versions

# Pin .NET version
- task: UseDotNet@2
  inputs:
    version: '10.0.x'

# Pin Node version
- task: UseNode@1
  inputs:
    version: '22.x'

2. Use Lock Files

• packages.lock.json for NuGet
• package-lock.json for npm
• .terraform.lock.hcl for Terraform

3. Validate Before Deploy

# Add validation stage
- stage: Validate
  jobs:
    - job: ValidateTerraform
      steps:
        - script: terraform validate

4. Health Checks

# Configure deployment health checks
- task: AzureWebApp@1
  inputs:
    healthCheckPath: '/health'
    healthCheckTimeout: '300'

---

📚 Related Resources

• Aspire Publish - Generate pipeline YAML from your AppHost instead of maintaining it by hand
• Pipelines Reference - What each pipeline does, its triggers, and PR path filters
• Aspire Troubleshooting - Debug Aspire startup issues
• Azure DevOps Pipeline Documentation
• Azure DevOps Services Reference
• Forge Pipeline Templates