Skip to main content

Azure DevOps

The Azure DevOps integration connects an Azure DevOps organization to Testery. Once connected you can:

  • Add Testery projects from repositories in any team project in the organization.
  • Trigger test runs for a branch or commit without uploading build artifacts.
  • Use testery.yml in the repository for configuration as code.
  • See a Testery status check on commits and pull requests, and block merges on it with a branch policy.

For publishing results into a pipeline's Tests tab, see Azure DevOps Pipelines.

Create a personal access token​

Testery authenticates with a personal access token (PAT) from an account that can read the repositories you want to test.

  1. In Azure DevOps, open User settings (top right) and choose Personal access tokens.
  2. Click New Token and give it a name such as Testery.
  3. Set Organization to the organization you are connecting.
  4. Set the expiration. Testery stops working when the token expires, so pick the longest period your policy allows and note the date.
  5. Under Scopes choose Custom defined and select:
    • Code: Read, and Status
    • Project and Team: Read
  6. Click Create and copy the token. Azure DevOps shows it only once.
warning

The Code (Status) scope is what lets Testery post status checks. Without it runs still work, but nothing shows on the pull request.

Connect the organization​

  1. In Testery, open Settings and the Integrations tab.
  2. On the Azure DevOps card click Connect.
  3. Enter the organization URL, for example https://dev.azure.com/your-organization, and the personal access token.
  4. Click Connect. The card shows the organization URL once the token is saved.

Azure DevOps Server (on-premises) works the same way. Enter the collection URL, for example https://azuredevops.yourcompany.com/DefaultCollection, as the organization URL.

Add a project from Azure Repos​

When you add a project and pick Azure DevOps as the provider, Testery lists the team projects in the organization and the repositories in each one. Choose the repository, the default branch and the test framework as you would for GitHub or Bitbucket.

Testery fetches the repository over git using the token, so the project does not need build artifacts for frameworks that run from source such as Playwright, Cypress, PyTest and Cucumber.js. .NET and TestNG projects still upload their compiled output with testery upload-build-artifacts.

Status checks on pull requests​

Every run for a connected project posts a status to the commit it tested. While the run is in progress the status is pending; when it ends the status is succeeded or failed with the total, pass and fail counts, and it links to the run in Testery.

The status is named after the environment key, Testery (<environment-key>), for example Testery (dev), with genre testery. To change the name:

  • testery create-test-run --test-name "UI Tests" replaces Testery, giving UI Tests (<environment-key>).
  • testery create-test-run --status-name "UI Tests" sets the whole name to UI Tests. It overrides --test-name, and the environment key is not appended.

If an active pull request's source commit or merge commit is the commit Testery tested, the same status is also posted to the pull request, so it shows in the pull request's Checks panel. That covers pipelines that test the pull request's source commit ($(System.PullRequest.SourceCommitId)) and pipelines that test the merge commit Azure DevOps builds for the pull request ($(Build.SourceVersion)).

Branch names may be given as plain names (main) or as full Azure DevOps refs (refs/heads/main, the value of $(Build.SourceBranch)). Testery treats both the same.

To make the status required:

  1. Open Project settings, then Repositories, and pick the repository.
  2. On the Policies tab choose the target branch (for example main).
  3. Under Status checks click +, pick the Testery status from the list and set it to Required.

The status appears in the list once Testery has posted it at least once, so run a test on a pull request before setting up the policy.

Pending status as soon as a pull request opens​

Testery marks a pull request pending the moment it is opened or updated, before any run exists, if you add a service hook. The hook marks both the pull request's latest commit and the pull request itself pending:

  1. Open Project settings, then Service hooks, and click +.
  2. Choose Web Hooks and click Next.
  3. Set Trigger on this type of event to Pull request created, and leave the repository and branch filters as you need them.
  4. Set the URL to https://api.testery.io/api/azure-devops-status and click Finish.
  5. Repeat for the Pull request updated event.

Without the hook, the status appears when the first run for the commit starts.

Troubleshooting​

  • The organization shows no projects. The token needs the Project and Team (Read) scope, and the organization URL must be the organization itself, not a project or repository URL.
  • Runs fail with "Could not download source code". The token has expired or lacks Code (Read). Create a new token and reconnect the integration.
  • No status on the pull request. Check the token has Code (Status), and that the commit Testery tested is the pull request's current source commit or merge commit. A run for an older commit does not post to the pull request once new commits are pushed. In pipelines, pass $(System.PullRequest.SourceCommitId) as the git ref (falling back to $(Build.SourceVersion)), as shown in Azure DevOps Pipelines.
  • The branch policy can't find the status. Look for the name Testery actually posts: Testery (<environment-key>) by default, <test-name> (<environment-key>) with --test-name, or the exact --status-name value.