Azure DevOps Pipelines

Azure DevOps works with Testery at three levels. You can use any one of them on its own.

1. Trigger a test run from a pipeline with the Testery CLI.
2. Publish the results back into the pipeline's Tests tab.
3. Show a Testery status check on the pull request with the Azure DevOps integration.

Before you start

1. Add your Testery API token as a secret pipeline variable named TesteryToken. Follow the Azure DevOps steps for secret variables. The token is on the Settings page under Integrations.
2. Make sure the build agent has Python 3 available. Microsoft-hosted agents do.
3. Install Testery CLI 1.20.0 or later with pip install --upgrade "testery>=1.20.0". Publishing results to the Tests tab relies on the JUnit output added in 1.20.0.

The PowerShell@2 examples below use forward slashes in paths, so they run unchanged on Windows, Linux and macOS agents. On Linux and macOS the task runs them with PowerShell Core.

Trigger a test run

Add a step that installs the CLI, uploads the build (for compiled frameworks such as .NET) and creates the test run. Update the project key, environment and paths to match your project.

- task: PowerShell@2
  displayName: Run tests in Testery
  inputs:
    targetType: inline
    script: |
      pip -q install --upgrade "testery>=1.20.0"
      testery upload-build-artifacts --token "${env:TESTERY_TOKEN}" --project "example-specflow-dotnet-core" --path "$(Build.SourcesDirectory)/Web.Specs/bin/Release/net8.0/" --build-id "$(Build.BuildNumber)" --zip-dir
      testery create-test-run --token "${env:TESTERY_TOKEN}" --git-ref "$(Build.SourceVersion)" --project "example-specflow-dotnet-core" --environment "dev" --build-id "$(Build.BuildNumber)" --wait-for-results --fail-on-failure
  env:
    TESTERY_TOKEN: $(TesteryToken)

info

--wait-for-results keeps the step running until Testery finishes, and --fail-on-failure fails the pipeline step when any test fails. Leave both off if you only want to kick the run off.

Projects whose source is in Azure Repos do not need upload-build-artifacts for interpreted frameworks such as Playwright, Cypress or PyTest. Connect the repository through the Azure DevOps integration and Testery fetches the commit itself.

Publish results to the Tests tab

testery report-test-run --output junit writes the run as JUnit XML, which Azure DevOps can publish as a native test report. The results then show in the pipeline run's Tests tab with pass, fail and skip counts, durations and failure messages.

- task: PowerShell@2
  displayName: Run tests in Testery
  inputs:
    targetType: inline
    script: |
      pip -q install --upgrade "testery>=1.20.0"
      $gitRef = if ($env:SYSTEM_PULLREQUEST_SOURCECOMMITID) { $env:SYSTEM_PULLREQUEST_SOURCECOMMITID } else { "$(Build.SourceVersion)" }
      $output = testery create-test-run --token "${env:TESTERY_TOKEN}" --git-ref "$gitRef" --project "example-playwright" --environment "dev" --build-id "$(Build.BuildNumber)"
      Write-Host $output
      $testRunId = ($output | Select-String -Pattern 'test_run_id:\s*(\d+)').Matches[0].Groups[1].Value
      testery report-test-run --token "${env:TESTERY_TOKEN}" --test-run-id $testRunId --output junit --outfile "$(Build.ArtifactStagingDirectory)/testery-junit.xml" --fail-on-failure
  env:
    TESTERY_TOKEN: $(TesteryToken)

- task: PublishTestResults@2
  displayName: Publish Testery results
  condition: always()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: testery-junit.xml
    searchFolder: $(Build.ArtifactStagingDirectory)
    testRunTitle: Testery
    failTaskOnFailedTests: false

report-test-run waits for the run to finish before it writes the file, so it does not need --wait-for-results on create-test-run. With --fail-on-failure it writes the JUnit file first and then exits non-zero if any test failed, so the step fails but the file is still there to publish. Keep condition: always() on the publish task so it still runs after the Testery step fails.

On a pull request build the script tests the pull request's source commit and falls back to $(Build.SourceVersion) on other builds. See Show a Testery status check on the pull request.

warning

--output junit needs Testery CLI 1.20.0 or later. Version 1.19.0 and earlier accept the option but quietly write SonarQube XML, which PublishTestResults@2 cannot read. Install with pip install --upgrade "testery>=1.20.0" to enforce the minimum version.

Show a Testery status check on the pull request

When the project's repository is connected through the Azure DevOps integration, Testery posts a status to the commit it tested and to any active pull request whose source commit or merge commit is that commit. The status shows pending while the run is in progress, then succeeded or failed with the pass and fail counts and a link to the run.

The status is named after the environment key: Testery (<environment-key>), for example Testery (dev). Pass --test-name to create-test-run to replace Testery, which gives <test-name> (<environment-key>). Pass --status-name to set the whole name yourself. The environment key is not appended to a --status-name value.

Use the pipeline above for pull request validation by triggering it from a build validation branch policy. On a pull request build $(Build.SourceVersion) is the merge commit Azure DevOps creates, so pass $(System.PullRequest.SourceCommitId) as the git ref to test the pull request's own commit. Fall back to $(Build.SourceVersion) because the pull request variable is only set on pull request builds:

$gitRef = if ($env:SYSTEM_PULLREQUEST_SOURCECOMMITID) { $env:SYSTEM_PULLREQUEST_SOURCECOMMITID } else { "$(Build.SourceVersion)" }
testery create-test-run --token "${env:TESTERY_TOKEN}" --git-ref "$gitRef" --project "example-playwright" --environment "dev"

Testery also matches the merge commit to its pull request, so the status still reaches the pull request if you pass $(Build.SourceVersion), but the run then tests the merged result rather than the branch. Full branch refs such as refs/heads/main (the value of $(Build.SourceBranch)) are accepted and treated as the branch name.

To block merges until Testery passes, add a Status check branch policy on the target branch and pick the Testery status (genre testery, name Testery (<environment-key>) or your --test-name / --status-name value) once it has been posted at least once.

Complete example

A pull request validation pipeline for a Playwright project whose source lives in Azure Repos:

trigger:
  - main

pool:
  vmImage: ubuntu-latest

variables:
  TESTERY_PROJECT: example-playwright
  TESTERY_ENVIRONMENT: dev

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.x'

  - script: pip -q install --upgrade "testery>=1.20.0"
    displayName: Install Testery CLI

  - script: |
      set -e
      GIT_REF="${SYSTEM_PULLREQUEST_SOURCECOMMITID:-$BUILD_SOURCEVERSION}"
      OUTPUT=$(testery create-test-run --token "$TESTERY_TOKEN" --git-ref "$GIT_REF" --project "$TESTERY_PROJECT" --environment "$TESTERY_ENVIRONMENT" --build-id "$BUILD_BUILDNUMBER")
      echo "$OUTPUT"
      TEST_RUN_ID=$(echo "$OUTPUT" | sed -n 's/.*test_run_id: *\([0-9]*\).*/\1/p')
      testery report-test-run --token "$TESTERY_TOKEN" --test-run-id "$TEST_RUN_ID" --output junit --outfile "$BUILD_ARTIFACTSTAGINGDIRECTORY/testery-junit.xml" --fail-on-failure
    displayName: Run tests in Testery
    env:
      TESTERY_TOKEN: $(TesteryToken)

  - task: PublishTestResults@2
    displayName: Publish Testery results
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: testery-junit.xml
      searchFolder: $(Build.ArtifactStagingDirectory)
      testRunTitle: Testery