# Testery Documentation Testery is a cloud-based, continuous testing platform that makes it easy to run tests quickly and reliably and push your test results to the platforms your team uses daily. If you're new to Testery, start with the [Getting Started Guide](/get-started/getting-started-guide). To setup Testery with your build server, check out our [CI/CD Integration Guide](/integrations/ci-cd-integration-guide). You can also explore our [Built-In Integrations with Testery](/integrations/built-in-integrations-with-testery) or try out the [Testery Command Line Interface (CLI)](/integrations/testery-cli-docs) or the [Testery REST API](/integrations/testery-rest-api). # Release Notes ## 2025-12-02 - v3.15.0 ### Performance Improvements * **Cypress Test Discovery Optimization**: Significantly improved the performance and reliability of Cypress test discovery for large test suites. The test finder now handles complex projects more efficiently, reducing the likelihood of timeouts during test enumeration. ### New Features * **Real-time Log Streaming**: Added real-time streaming for test run logs. You can now watch test execution logs as they happen, providing immediate visibility into test progress and failures without refreshing the page. ## 2025-11-20 - v3.14.0 ### UI Enhancements * **Dark Mode**: Added system-wide dark mode theme with keyboard toggle (press 'd' key). * **Keyboard Shortcuts**: Implemented global keyboard shortcuts for improved navigation: * `g,d` - Navigate to Dashboard * `g,t` - Navigate to Test Runs * `m` - Toggle sidebar collapse/expand * `?` - Show keyboard shortcuts help menu * **Test Runs Page Improvements**: * Moved all filters to column headers with dropdown popups for cleaner interface * Status, Date, Project/Test Suite, Branch, Pipeline Stage/Environment, and Reviewer filters now appear in column headers * Added filter icon visual indicators (active/inactive states with color coding) * Table headers now always visible, even when no test runs are found * Improved filter icon contrast for better visibility of active filters * Added tooltips to Pipeline Stage and Environment fields ### Reviewer Assignment * Enhanced reviewer assignment dropdown with immediate visual feedback * User icon changes from "user plus" to "user" when reviewer is assigned * Green icon color indicates assigned reviewer * Tooltip shows assigned reviewer's full name on hover * Check marks indicate selected reviewer in dropdown menu * All updates happen instantly without page refresh ### Navigation & Layout * Improved left navigation icon alignment and spacing * Better vertical centering of navigation icons in both collapsed and expanded states * Added top padding to header for improved logo centering * Fixed icon height consistency across all navigation items * Improved collapsed navigation with centered icons and even padding ### Bug Fixes * Fixed Chart.js imports to use auto-registration, resolving Analytics chart rendering issues * Fixed keyboard shortcut sequence handling so `g,d` and `g,t` work correctly for navigation * Fixed user icon background transparency for better appearance in both light and dark modes * Fixed card component styling in dark mode ## 2025-11-19 * Added UI elements for test selection feature allowing more flexible test filtering. * Enhanced test selection logic to support multiple selection rules simultaneously. * Fixed test selection filtering bugs when applying complex filtering scenarios. ## 2025-11-11 * Fixed test selection filtering logic when multiple selection rules are applied. ## 2025-11-05 * Improved test run analysis reliability by auto-acknowledging messages even when errors occur. * Fixed issue with failed analysis jobs to reject them instead of requeuing. ## 2025-10-17 * Added support for Tag/File OR test selection with improved flexibility. * Fixed issue with test selection when using both tags and filters. * Added support for excluded\_tags and test\_filters in testery.yml configuration. * Fixed test selection criteria defaulting behavior. * Fixed ignored tests getting incorrectly marked as failed. ## 2025-10-15 * Upgraded to Node.js 24 for improved performance and security. ## 2025-09-11 * Added support for nested sub-organizations in GitLab integrations. ## 2025-08-15 * Added GitLab support for source control integration. ## 2025-08-11 * Fixed issue preventing tests from running when not utilizing end-user's config file properly. ## 2025-07-15 * Fixed issue causing tests to fail when src\_folders parameter was null. ## 2025-06-30 * Added support for running Cucumber tests on Nightwatch framework. ## 2025-06-15 * Fixed Cypress test detection issue where spec.baseName was returning undefined. ## 2025-06-05 * Added additional environment variable support for dry runs. * Fixed test filter issues preventing proper test selection. ## 2024-12-10 * Added opt-in support for test selection rules. ## 2024-11-25 * Added Bitbucket support for test selection. ## 2024-11-22 * Fixed broken SQL query affecting test run data retrieval. ## 2024-11-21 * Fixed column name mismatch in database queries. ## 2024-11-20 * Enhanced GitHub repository source matching for test selection. ## 2024-11-19 * Added ability to parse configuration files for test selection rules. * Implemented comprehensive test selection rules framework. ## 2024-11-14 * Established framework for advanced test selection capabilities. ## 2024-10-23 * Fixed bug with insight title length validation. ## 2024-10-22 * Added ability to cancel test runs when executing test plans. ## 2024-10-21 * Fixed Playwright test triggers to work correctly with scheduled runs. ## 2024-10-08 * Added manual quarantine feature for tests allowing users to manually mark tests as quarantined. ## 2024-10-04 * Enhanced Nightwatch integration with improved Selenium standalone management. ## 2024-10-02 * Added support for sending messages via CLI to test runs. ## 2024-08-20 * Improved database performance by tuning materialized views to use partitions. ## 2024-08-19 * Enhanced GitLab integration with improved repository handling. ## 2024-08-15 * Implemented test run archiving for better data management and performance. ## 2024-08-07 * Partitioned test\_run\_test table for improved database performance. * Fixed issue with pending reruns not being properly deleted. ## 2024-08-02 * Limited test failure text to 15,000 characters to prevent database performance issues. ## 2024-07-29 * Added tracking for test plan run cancellations. ## 2023-05-15 * Fixed a bug causing test runs to errantly get marked as failed in a certain Cypress case. * Fixed a WDIO bug where tests could get marked as passed but have failed steps. This should no longer happen. ## 2023-05-11 * Added new `run-test-plan` command to the CLI with the option to override the environment. * Added `key` field to Test Plans which can also be used via the latest version of the CLI. * Fixed the bug causing an error using the `cancel-test-run` command on the CLI. * Fixed a bug causing screenshots failing to get saved when a test times out. * Fixed a bug with WDIO test runs that started with the NodeJS 16 upgrade. ## 2023-05-01 * Upgraded NodeJS from v14 to v16 on both Linux and Windows runners. * Fixed form validation error when saving Test Triggers in certain cases. ## 2023-04-20 * Fixed a bug causing Cypress screenshots to not show up in certain cases. ## 2023-04-17 * Text overflow in dropdowns should now wrap to the next line. * Fixed date filtering on the Test Runs tab. * Added date filtering to the URL, for easy bookmarking and sharing, to the Test Runs tab. * Improved sorting on various dropdown menus. * Fixed form error for Test Plans. ## 2023-04-14 * Added the ability to "bookmark" filters on the Test Runs tab. Filters will be saved in the URL which will be used to auto-set these filters again when returning to that URL. * Performance enhancements. * Better error handling. ## 2023-04-05 * Fixed an issue causing some test results to not be reported correctly in certain cases. * Fixed the search feature several dropdown menus... * Environment Dropdowns * Pipeline Stage Dropdowns * Project Dropdowns (both single selection and multiple selection) * Test Plan Dropdowns * Test Suite Dropdowns ## 2023-03-27 * Fixed Cypress issue causing some test run issues. ## 2023-03-09 * Upgraded Windows runners to Chrome v111. ## 2023-02-07 * Continued improvements for test runs getting stuck in `SUBMITTED` and `RUNNING` statuses. ## 2023-01-30 * Test runs getting stuck in a `SUBMITTED` state improvement. * Implement ability to use AWS Private Link on Windows runners. ## 2023-01-27 * Fixed logging issue causing some test names to be longer than they should be. ## 2023-01-19 * Added support for `yarn install` arguments to be added through Testery environments. ## 2023-01-12 * Updated Chrome version from 107 to 109 on Windows runners. ## 2022-12-20 * Fixed race condition in test run status transitions. ## 2022-12-14 * Fixed test trigger form validation issues. * Test run status improvements. ## 2022-12-13 * Fixed the test run reviewer assignment dropdown that wasn't showing the proper user list. ## 2022-12-06 * Test run status improvements. ## 2022-11-30 * Fixed incorrect test run counts after running tests. * Test run status improvements. ## 2022-11-21 * Microsoft Teams Integration. * Increased allotted memory on Windows runners. ## 2022-11-07 * Fixed a bug that could cause a Cypress 10 configuration file parse error in a certain case. ## 2022-11-05 * Fixed bug causing some test runs to get stuck and not complete. ## 2022-11-01 * New Feature: Two new tags/hooks have been added to Testery to allow users to run code as setup/teardown steps. Just annotate your code block with @BeforeTesteryRun or @AfterTesteryRun and Testery will run that code before/after the test run. This code will only run once before/after the run, not once on each runner. * New Feature: Users can now create teams within Testery. * Upgraded Linux runners to Chrome v107. ## 2022-10-21 * Fixed a bug that could cause CucumberJS tests to fail in some cases. ## 2022-10-10 * Fixed a bug preventing test results from showing for some Cypress 10 test runs. ## 2022-10-06 * Fixed Cypress 10 Config parse issue. ## 2022-09-26 * Added the ability for users to run scripts before a test run on each runner. Simply add a script named `testeryPreRunScript` to your `package.json` and the Testery app will run that script before the test run on each runner. * Fixed a bug causing flay tests to show in the list of failed tests in Slack notifications. ## 2022-09-16 * Fixed a bug that could cause an error when parsing a Cypress 10 config file. ## 2022-08-29 * New feature! Users running PyTest tests can now choose the Python language version in the project's settings. ## 2022-08-12 * Features * Users can now assign tests for review. * Users can now set a review status for tests. * Added support for Cypress 10 (see Project Settings in the app to select framework version). * Testery CLI will now print out it's version number on each command run. * Bugfixes/Improvements * Turned off emails to admin users when Slack channel doesn't exist. * Fixed bug that did not take the Project's Parallel Strategy into account when manually starting test runs by Test Suite. * Fixed a bug that prevented the Analytics tab from loading in certain timezones. * Fixed bug preventing screenshots and videos from working on the Cypress framework. ## 2022-07-27 * Upgraded Window and Linux runners to use Chrome v103. ## 2022-07-22 * Fixed side-scrolling issue on Test Suites tab. * Fixed bug preventing team members from showing up in Settings. * On the Test Runs tab, Suite will be on one line instead of two. ## 2022-07-11 * Fixed an issue on the Test Runs tab where selecting Latest Deploy Version may not run the proper changes in some cases. ## 2022-07-06 * Fixed UI issue on the Test Runs tab where the "Suite" field had 2 lines. * Fixed bug preventing users from seeing team members in their user Settings. ## 2022-06-29 * Fixed a bug causing some tests to fail when a large number of tests were present. ## 2022-06-28 * Implemented feature improvements... * Users can now specify an alternate location for their Cypress config file. * Implemented several fixes and improvements on the Test Runs tab. * Fixed bug causing some test runs to get stuck in a SUBMITTED state. ## 2022-06-14 * Fixed a scrolling bug in the UI in the account selection dropdown menu. ## 2022-06-07 * Fixed a bug preventing some Slack notifications from being sent. ## 2022-06-06 * Added "Triggered By" label back to the Test Runs tab. * Fixed an issue with deploy creation using previously used references. ## 2022-05-31 * Fixed NUnit runner issues. ## 2022-05-24 * Upgraded Linux runners to .NET Core 6.0. * Added email notification for failed Slack notifications. * Added better handling of Environments when they get too large. * Add Slash Commands to Slack (users may need to re-install the Testery App to start using this). ## 2022-05-02 * Fixed UI bug where the Test Run list table header was showing through the datepicker calendar. ## 2022-04-27 * Fixed UI bug when adding Environment files. ## 2022-04-20 * Increased the maximum **Test Timeout (seconds)** from 1 hour to 3 hours. ## 2022-04-10 * Fixed bugs preventing Slack notifications from being sent. ## 2022-03-29 * Fixed a couple bugs preventing Slack notifications from being sent. ## 2022-03-24 * Released new feature that allows test runs to be assigned to a user. The user will be notified via Slack (if Slack is connected). * The Test Runs tab is now mobile friendly. * Fixed a bug causing the wrong Chrome version to be used on Windows runners. ## 2022-03-22 * Fixed a bug preventing users from starting test runs. ## 2022-03-10/11 * Fixed a bug causing user defined Chrome options to be discarded ## 2022-02-23 * Added video recording of tests on Linux runners (beta) * Reworked UI when viewing a testrun's test results ## 2022-02-02 * Removed flaky tests from the failed test list in Slack messages ## 2022-01-06 * Environment Variable fixes and UI Improvements * Several fixes surrounding saving Environment Variables and Files * UI Improvement: Environments Page * New way to list Environment Variables/Files * More mobile friendly * Hover to see variable values. * UI Improvement: New/Edit Environment Page * New lightweight code/text editor when adding files. * Consolidated View * Better guarding around blank fields ## 2021-12-13 * Now supporting TypeScript for WebDriverIO tests * Increased **Description** character limit in Projects ## 2021-12-09 * Improved ES6 support in Cypress tests * Improved logging ## 2021-11-23 * Improved Slack Messages * Added interactivity features for re-running all the tests in that run or just the failed ones * Added a graph detailing passed, failed and ignored tests * Added the commit message, the committer and time of commit that belongs to the test ref * Now showing a list of failed tests * Now showing the list of test filters ## 2021-11-16 * Fixed logging issue ## 2021-02-12 * You can now filter your test runs on the Test Run tab by flaky test. * Test run analytics. See a historical view of your test runs and gain insight into how your tests are performing. Filter by time period, environment, project and build. * Flaky tests analytics. See a historical view of your flaky tests and gain insight into the stability of your tests. Filter by time period, environment, project and build. ![Test run analytics](/files/-MUu-Q-nxl_ZhROHo_ng) ## 2021-01-29 * Changed UI to show all tests (previously only show last test) in a test run when a test or test run is re-run. * Performance improvement: cached calls to reduce page load time * Added project setting for number of default runners. All tests runs for a given projects will default to this unless overidden at test run time. * Added a new alert for 'When test run completes and has flaky test'. This can be setup in Alerts * Added ability to collaborate and leave messages for your team on a test run. * Added ability to follow a test run. Once you follow, you can receive slack or email notifications when a test run moves from queued to submitted status, when a test run completes and when a new message is posted to test run. ## 2020-12-18 * Better align screenshots with specific tests. Added option in Project settings that you can turn on which will print the test name to the test output and Testery will match that against the test. * Added npm to the default image that test runners use * Performance improvement: changed UI to only load updated test results on test results page ## 2020-12-04 * Test run timeline view! Checkout the test run timeline view to see how each test executed. Easily identify any patterns that may be causing issues in your tests or environments. ( see below image) * Added filters to Schedules tab making it easier to find what you need. Filter by project or environment. * Bug fix: we fixed issue running latest version of chrome on Windows servers. * Added support for running Lighthouse ![](/files/-MQwaRNElNOyWtGypNZ9) ## 2020-11-20 * **Flaky tests**! Testery will now identify when a test is flaky by adding a tag so you can take action. Flaky tests are a nuisance and can cause the team to not trust the results. Testery now helps you make it easy to identify a test that failed once but passed the second time. (see image below) * Added ability to re-run failed tests. There is a setting on a test run now that you can turn on and when tests fail the failed tests will automatically re-run. This helps account for flaky tests and saves your team time. * Added ability to create a Testery account using email (instead of just Github and Bitbucket) * Added a new user invite. When you add a team member they will now receive an invite making the process much more smooth getting into Testery. * Display commit for project that triggered the test. If a test run is triggered by another project deploying and your test code is in a different project, you will now see both projects commit info. ![](/files/-MQwabU7KMcvr0O1rSl_) ## 2020-11-06 * Added test run execution priority. Give a pipeline stage a priority and all test runs within the stage will default to that priority. You can override this at a test run level. Now, more important environments / tests can run before others that are less urgent. ## 2020-10-23 * Added commit info for each card on the Environment Dashboard. You can now easily see what code was in the environment at the time the test ran and who committed. * Added ability to filter test results by test status. * Added a not run category in test results so you can see if a test didn't run for some reason. * create-deploy from the CLI now supports the same environment, project and commit but with different build ids. ## 2020-10-09 * Added pipeline stages. Pipeline stages represent stages in your development pipeline (dev, test, qa, prod, etc). Each environment can be added to a pipeline stage allowing you to group environments into stages. You can view your projects and test runs by pipeline stage. * We now have an **Environment Dashboard**!!! See a birds eye view of all your test runs across environments. If you have setup pipeline stages the stages will display in columns with each test run /environment combo in the column. When you click on a test run on the dashboard it will take you into the test run to see more details. * Added a link to your environment. Edit your environment, add the url to your environment and team members can easily access the environment from within test runs. * We made test\_*run\_*id available as an environment variable. * Added filters to more easily navigate the environments page when you have a lot of environments. * Added ability to change your password in Settings. * Bug fix: fixed issue when manually running a test run when using "latest deployed version". ## 2020-09-25 * Add tags for Cypress. * Added ability to run Cypress tests parallelized by test (you can't do this in Cypress Dashboard at this time!) * Added ability to give scheduled test run a name ## 2020-09-11 * Updated configuration to support webdriver.io file being stored in any location. * For Cypress.io, it's a common issue with the framework where chrome will fail to load. We added extra support for this scenario and will automatically re-run the test if chrome fails to load. * Added a "Run Now" button to Scheduled test runs. If you click the run now button the scheduled test run will run immediately. * UI Design: Enhanced how tests display within a file/feature so they are grouped better. * Updated Slack alert to send to notify for all environments and not just a single one. * Bug fix: Fixed issue where a test here or there would get run twice and count as 2 tests in results. ## 2020-08-28 * **Scheduled test runs** is live! You can now schedule a test run to run regularly at a certain day/time or when a deploy happens. (If running a test when a deploy happens you have to send us your deployment information.) * Display branch name to the Source Info box on a test run. * Added support for Bitbucket Server. * Added ability to manually upload test artifacts. A project can now be created with no artifacts associated. * Bug fix: When parallelized by file and re-running failed tests, only re-run the tests in the file that failed. * Bug fix: enhanced how screenshots are associated to a test run ## 2020-08-14 * Bug fix: Screenshots from failed test are also getting attached to next test run at times ## 2020-07-31 * Test runs page got a redesign! You can now view project name, include/exclude filters, regex, test passing % without having to click into results. * Better filters * Allow multiple test runs to be queued up that have different regex filters (given all other fields are the same) * Fixed the rerun failed tests action to account for parallelize strategy when parallelizing by file/feature. ## 2020-07-17 * Added ability to configure/override timeout for test runs. You can do now set a default timeout for a single test and the test run at the project level and override at the test run level. * Added ability to parallelize by file for the Nightwatch.js runner * For NUnit tests, allow user to specify to parallelize tests by file or test. When running tests by feature separate out test results by individual test for easier viewing and report errors appropriately. ## 2020-07-03 * Allow user to open test run results in a new window * Allow user to copy the test result name * Display test output while test is running * Do not clear regex box when the field loses focus # Getting Started Guide In most cases, getting started with Testery takes just a few minutes. If you haven't already, you'll want to start by [creating an account](/get-started/getting-started-guide/create-an-account). Then you'll [configure your project](/get-started/getting-started-guide/configure-your-project) (unless you already did this when creating an account). A few moments later and you'll be ready to [run your tests](/get-started/getting-started-guide/run-your-tests). # Create a Testery Account A free Testery account will enable you to run your smoke tests on a schedule or every commit and send notifications to Slack, OpsGenie, or wherever else you need them. **Sign-up with GitHub or BitBucket**. To create a Testery account go to and login with Github or Bitbucket. This will take you through the signup flow. ![](/files/-MK1MYBf-WJC9gNov0Rj) **Select an Organization**. Select an organization you would like to connect your Testery account to. It should be the organization that has a repository with your tests. **Enter your Organization Name**. The Testery Organization Name defaults to the Organization you selected. Feel free to change this name.![](blob:https://testery.atlassian.net/ff823b59-50a0-473d-bfb4-fbd36a2e8207#media-blob-url=true\&id=592e5019-8dcd-4c9e-a65d-c4b226ad2f74\&collection=contentId-733642782\&contextId=733642782\&mimeType=image%2Fpng\&name=Screen%20Shot%202020-06-26%20at%209.55.53%20AM.png\&size=119879\&width=2442\&height=644) ![Review/Edit Organization Name](/files/-MMIJU7IR0Dwcw_GUdDE) **Setup a Project**. For your first project, select which repository contains the tests you want to run on Testery. Select your default branch. **Update your Project Settings**. Select your testing framework. Testery currently supports WebdriverIO, CucumberJS, Cypress, Specflow, NUnit, and Nightwatch. We are actively adding more to this list so if yours isn’t listed select Other along with name of framework you are using and we will let you know the status of that framework. You may be asked a few more questions depending on which framework you select. Select whether you package json is in the root direction. If it is not, specify the directory. **Select a Plan**. Select which plan you would like. There is a free plan on Linux servers if you want to try it out. If you require more concurrency or testing minutes we would be glad to work with you on a custom plan. **Click Finish**. Click Finish and you will be taken to the Test Runs page within the Testery platform. Your test should be kicked off for you if everything was setup correctly. If you have any issues during account creation send an email to or there is a help widget you can submit a ticket through in the lower right corner. Look for the **?** ![If you ever need assistance, click the '?'](/files/-MK1LQs4ohC49u58sJO5) # Configure Your Project 1. Navigate to the [Projects](https://testery.app/testery/projects) page and click the [Add New Project](https://testery.app/testery/projects/new) button.
2. Click **Add Repository** and select the code repository that includes the tests.
3. Select **Default Branch**. This is the version of the tests that will run unless another version is specified.
4. Select your **Testing Framework**. Depending on the framework chosen, you may see different options available. For example, you may be asked where your package.json or wdio.conf files are located. *Answers to these questions are very important as this tells Testery where to find the tests to run.* 5. Click **Save** to save your project. # Run Your Tests In Testery, there are three ways to run your tests. You can perform one-off test runs (shown below), schedule your test runs, or integrate with CI/CD to run your tests. For now, let's just perform a one-off test run to make sure the project is configured correctly. 1. Click **Test Runs** or navigate to the [Test Runs](https://testery.app/testery/test-runs) page.\ \
2. Click **New Test Run**. 3. Enter your desired parameters for the test run.\ ![](/files/ybnoq6WdUgyKG0kagED2)\ \ **Environment** - This is the environment that the tests will target as the system under test. This is typically something like dev, qa, rc, or prod. See [Environments](/get-to-know-testery/set-up-your-test-environments)for more information.\ \ **Trigger Type** - Here you can select how you would like to start the tests. For now, just select **Run tests now**.\ \ **Options** - Options let you select what set of tests you'd like to run. You can run all the tests in a project, select individual suites, or filter tests by test name or tag.
4. Click **Run Test** to start the test run.
5. You will be taken back to the test runs page and should see your test run in a "SUBMITTED" state. Testery is now dynamically provisioning the infrastructure to run your tests. 6. Congratulations! You have tests running on Testery! You are now ready to [Configure Slack Alerts](/get-started/getting-started-guide/configure-slack-alerts). Everyone's test projects are a bit different and occasionally additional steps are required to configure your projects. If you need any additional assistance at this step, please email [support@testery.io](mailto:suppo) or make a request at the [Testery Service Desk](https://testery.atlassian.net/servicedesk/customer/portal/1). \
# Configure Slack Alerts Testery lets you configure various alerts and notifications to take place whenever different test run events occur. For starters, let's configure a Slack alert. With Testery Slack Alerts, you can not only see the results in Slack, but even link to the code commit that initiated the test run, rerun failed tests, or rerun the entire test run! ![](/files/aKSw6EdEP4is1MuCGd8y) 1. Navigate to Settings -> Integrations () and click **Authorize Slack Integration**. 2. Navigate to [Alerts](https://testery.app/testery/alerts) and click **Add New Alert.** 3. **Select the Event**. For example "When Test Run Completes" 4. For **Take Action** select "Send Slack Message". 5. For **Slack Channel**, specify the channel name. Be sure to include `#` in the channel name. (e.g. `#testruns`. Make sure the Slack user who authorized the integration has permission to post messages to the channel.\ ![](/files/06M08SQEAG0rud2hilGq) 6. Click **Save.** Your Slack channel will now receive notifications
For more information on the various kinds of Alerts you can configure in Testery, see Alerts. # Schedule Your Test Runs Testery is really flexible about when, where, and how you run your tests. To schedule your tests to run nightly for example you can, 1. Navigate to **Test Triggers** and click **Add New Trigger.** 2. Give your trigger a **Trigger Name** (e.g. "Nightly Run"). 3. For Trigger Type select "Run tests on trigger" and enter a valid cron expression. For example, to run the tests every night at midnight UTC you can use the following cron expression:\ \ 0 0 \* \* \*\ \ See for more information about cron. 4. Set **Project with Tests** to the project you just set up. Here you can select what tests you want to run. For now, just pick "Run all tests in project" to run all of your tests. 5. For **Environment** select the environment you set up earlier. For example "Development". 6. Then select **Specific Version** and chose **Branch** and leave **Commit** set to "latest". This will run the latest commit for the branch. As you can see, there's a lot of flexibility. See [Test Triggers](/get-to-know-testery/schedules) for more information about all the different options. # Invite Your Team Add users to your team by selecting Settings -> Team. Add users to your team by selecting **Settings** -> **Team**. Use the email address associated with the users GitHub or BitBucket account. ![](/files/-MK1OvYoDIgPaElVdDPA) # Get More From Testery You should be up-and-running now, but we're just getting started. To get more from Testery, you can: * Set up [Environments](/get-to-know-testery/set-up-your-test-environments) so that you can run tests in different environments and pass in environment variables. * Set up [Alerts](/get-to-know-testery/alerts) so that your team will get notified in Slack, OpsGenie, etc. when test runs complete or fail. * Set up [Schedules](/get-to-know-testery/schedules) so that test runs will run automatically either on a schedule or on every deployment. * Follow the [CI/CD Integration Guide](/integrations/ci-cd-integration-guide) to set up you build server to run tests on the Testery platform. * Check out the [Testery CLI Docs](/integrations/testery-cli-docs) and [Testery REST API Docs](/integrations/testery-rest-api) to build your own integrations. Be sure to let us know at what you come up with! # Dashboard The Dashboard is your window into how things are going. You will see the latest test runs for each Project and Environment grouped by Pipeline Stages. This view becomes very useful if you have some Pipeline Stages that are required to be "always green" and others that are expected to fail sometimes (e.g. production monitoring vs active dev work). ![](/files/-MK1ftiAfeNFj3dWhxDt) # Test Runs The Test Runs view shows a list of your most recent test runs. You can filter by Date, Project, Environment, or Status to find the test run you're looking for. ![](/files/-MK1WFidam8WkQKuGOMR) ## Test Run Status and Test Status There are two kinds of statuses in Testery: 1. **Test Run Status** - the status of the entire test run regardless of how many tests are in it 2. **Test Status** - the status of a single test within a test run #### Test Run Statuses | **Status** | **Description** | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SUBMITTED | Test run has been submitted to the queue to be run | | QUEUED | Waiting for another test run in the same environment to finish. We do not run 2 test runs in the same environment at once. | | RUNNING | Test run is currently running | | PASS | All tests that are not ignored passed. | | FAIL | There was at least one test in the test run that failed OR an error occurred and the test run can’t finish. If the latter happens a failure reason should display. | | CANCELED |
  1. Test run was manually canceled by someone. (user’s username will display to show who canceled the test run) OR
  2. Test run will be automatically canceled if it runs for more than 3 hours (message will be Canceled - Due to running too long) OR
  3. Test run is in queued status and a newer test run comes in, the current queued test run will be canceled (message will be Canceled - Due to newer test run)
| #### Test Statuses | **Status** | **Description** | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | ![](blob:https://testery.atlassian.net/65ddf0f8-8cd9-4259-8bd5-e7c29f387cb2#media-blob-url=true\&id=6156bb52-66b1-4a29-bb05-19a132cb3bf8\&collection=contentId-621674497\&contextId=621674497\&mimeType=image%2Fpng\&name=testPass.png\&size=5267\&width=56\&height=18)PASS | The test passed. | | ![](blob:https://testery.atlassian.net/10dbe6bf-2b76-4a74-a93a-d39b7f87f8e2#media-blob-url=true\&id=13cdf347-6650-49f9-bb91-a22d34c1c2e2\&collection=contentId-621674497\&contextId=621674497\&mimeType=image%2Fpng\&name=testFail.png\&size=4998\&width=56\&height=18)FAIL | The test failed. | | IGNORED![](blob:https://testery.atlassian.net/4f5513dd-bd4f-43d1-9605-c9defe1b3358#media-blob-url=true\&id=a8026003-57c9-4143-840f-9bea74029a9e\&collection=contentId-621674497\&contextId=621674497\&mimeType=image%2Fpng\&name=testIgnored.png\&size=5409\&width=55\&height=19) | Test was flagged as ignored and this test did NOT run. | | PENDING![](blob:https://testery.atlassian.net/49be41e6-1adf-4f95-89a1-61595a08b8ec#media-blob-url=true\&id=4c415ac0-f752-425c-9c33-2a9927e041cb\&collection=contentId-621674497\&contextId=621674497\&mimeType=image%2Fpng\&name=testPending.png\&size=5213\&width=56\&height=19) | Test is waiting to run. | | ![](blob:https://testery.atlassian.net/dd84fe69-ccdb-4793-aee2-301a5d25656c#media-blob-url=true\&id=76fc104f-5367-4e37-96cc-009890eb7948\&collection=contentId-621674497\&contextId=621674497\&mimeType=image%2Fpng\&name=testRunning.png\&size=4840\&width=55\&height=20) | Test is running. | #### Test Run Options There are 4 options for define the tests that will be executed during your test run. **Run all tests in project -** All tests detected in the project will be executed.
**Select test suite -** Tests will be determined by a predetermined test suite.
**Filter tests in project -** A one-time set of filters will be applied. These can be done with include/exclude tags or by test file filters.
**Use testery.yml -** Test selection will be controlled by the rules in the testery.yml file. [Read here for details on how to define these rules.](/get-to-know-testery/test-runs/test-selection-rules)
# Test Selection Rules The Test Selection feature enables your team to run a subset of the tests based on changes made to the source files. In order to start using this feature, your repository should have a `testery.yml` file defined. This file should include a section as described below. When running tests from the UI, be sure to select the "Use testery.yml" option.
### testery.yml syntax for defining test selection rules ``` test_selection: suites: - default: rules: - employeeRule: when: - src_changes: - "src/employees/*" then: - include_tags: - "employees" - "smoke" - itemRule: when: - src_changes: - "src/items/*" then: - include_tags: - "items" ``` Currently, Testery supports 1 suite of rules, named "default". Within the suite, you'll need a list of "rules". Each rule has a "when" and a "then". #### When Conditions When conditions are used to trigger a set of tests to be selected. * **src\_changes** - This trigger will apply a glob expression to the list of files changed in the code branch. #### Then Conditions Then conditions refine the tests that will be selected * **include\_tags**: If your testing framework supports tags, then this can be used to specify the tags that will be included in the test run * **exclude\_tags:** If your testing framework supports tags, then this can be used to specify the tags that will be excluded in the test run * **test\_filters:** A glob expression that can be applied to choose the tests to run # Projects Projects enable you run tests across multiple repositories and even testing frameworks. You can have JavaScript, Java, and even .NET test results all in the same place, regardless of what testing framework or CI/CD your individual teams are using. ![](/files/-MK1khbVVakvtq6Wv7Wb) # Test Triggers Testery gives you flexibility about when, where, and how you run your tests. With Test Triggers, you can configure your tests to run when * Run tests when another project is deployed (e.g. run your Selenium/Cypress tests when the API is deployed) * Run tests on a schedule (e.g. nightly, hourly) * Run tests when the tests are updated * Run different suites depending on what environment is deployed to Scheduling automated tests to run on a regular schedule is a common task that companies implement to ensure automated tests are run regularly and consistently. Testery allows you to create and manage your scheduled test runs right within the platform! **Scheduling an Automated Test**\ \ There are 2 ways to schedule a test run within Testery. 1. **Create a test run and make it recurring**. While creating a new test run there is a Schedule field where you can select Run Now, Run on Interval or Run Deploy. Selecting Run Now will create your test run and run it one time. Selecting either of the other 2 options will create a scheduled test run.
2. **Create a new schedule on the Schedule tab**. On the schedule tab, click the green +Add New Schedule button to add a scheduled test run. ![](/files/-MK1l_w1CtkvyJOi1qPj) ![](/files/-MK1liLk-FjcGR8tHzua) **Schedule Details**\ \ **On Deploy**. *Note: The On Deploy option will not show up until Testery has deploy info for at least one project. See* [*Testery Deployments*](https://testery.atlassian.net/wiki/x/GYAWNQ) *for more information.*\ When scheduling an automated test to run on a deploy you first select the project and environment you want your tests to run against. Next, select Run On Deploy from the Schedule dropdown. Once you do this a Deploy Project box will appear. Select the project you want to trigger your tests when it is deployed. Select the Branch, Commit and Build of the tests you are running. (*If you want to select the latest Build when scheduling a test run, Testery has to have deployment info so it is deploy-aware. See* [*Testery Deployments*](https://testery.atlassian.net/wiki/x/GYAWNQ) *for more information.*) Then, add any filters and tags to ensure you are including only those tests you want to be scheduled.\ ![](blob:https://testery.atlassian.net/1a9003e5-20a9-4b8f-ad4d-eb0c2585b5dc#media-blob-url=true\&id=c27eb35c-86bd-4e5c-9d88-2ff96cc0793d\&collection=contentId-890634241\&contextId=890634241\&mimeType=image%2Fpng\&name=fF8ViGtjsUyh205dsRhSrF78ViGwnmoVhreBfmT-dP4nBJzjYfDi9JBgoDy6etZVBwLEroDX5tMpmNqIwOjYMjW4ICbUuayAFqQseE_nGW7c3WIfSrGsxGNal3kk_XZNcBEQc_8w\&size=72295\&width=1254\&height=866) ![](/files/-MK1lop3sdGlZ7p-Wde7) **On Interval**. When scheduling an automated test to run on an interval you first select the project and environment you want your tests to run against. Next, select Run On Interval from the Schedule dropdown. Once you do this a Cron Expression box will appear. Enter the day/time you would like your test run to execute. Testery will validate the cron expression using the [crontab](https://en.wikipedia.org/wiki/Cron) format. Select the Branch, Commit and Build of the tests you are running. (*If you want to select the latest Build when scheduling a test run, Testery has to have deployment info so it is deploy-aware. See Testery Deployments for more information.*) Then, add any filters and tags to ensure you are including only those tests you want to be scheduled. The structure of the Cron Expression should look like the line below. There are five fields(\*) specifying the time to execute the command. \* \* \* \* \* Where the asterisks(\*) represent the following time fields: * minute (0-59) (*in UTC timezone*) * hour (0-23) (*in UTC timezone*) * day of the month (1-31) * month of the year (1-12) * day of the week (0-6, Sunday to Saturday) You can leave the asterisk which means you want all values for the time field. You can also specify a specific number or even multiple numbers (separate multiple numbers with a comma). Here are some examples of how you might specify the cron expression in Testery: * Run a test daily at midnight ( 0 0 \* \* \* ) * Run a test each week on Wednesday at 5:00 AM ( 0 5 \* \* 3 ) * Run a test each day of week at 6 AM and 11 AM (0 6,11 \* \* \* ) * Run a test every hour of every day ( \* \*/1 \* \* \* ) Now, just save the test run and voila!!! You now have scheduled your automated tests to run on a schedule. **Managing schedules**\ \ From the Schedules tab in Testery is where you will manage your schedules. You can create new schedules, edit existing schedules, or delete a schedule. You also have the ability to enable and disable schedules.
![](/files/-MK1m4Kn4_pESU6oUJ8v) # Environments Testery is an **environment-aware** testing platform. This means Testery supports running the same tests or different subsets of tests against different environments. This is useful for situations where you want to run different sets of tests against different environments. For example, you may want to run your important tests on every commit to your development environment, full regression against your QA environment, and smoke tests against your production environment. ### Add a New Test Environment 1. Go to **Environments** and click **Add New Environment**. 2. Enter the **Name, Key**, **Environment URL, Pipeline Stage and Maximum Parallel Test Runs** for your environment. If you don't have any Pipeline Stages configured, you can leave that blank for now. 3. Add your **Variables**. ### Maximum Parallel Test Runs This value sets the maximum number of test runs that can use this environment at once. If multiple test runs are kicked off against this environment, they will be queued until the environment is clear for more tests as test runs complete. Enter `0` if you do not want to limit this environment. ### Variables **Variables** are key value pairs that get passed into the tests as standard environment variables. They can be added or modified when creating the environment or even when starting a test run from the API or CLIE. Be sure to check the **Encrypted** check box for any parameters that should be stored securely. ### Audit Trail Testery provides a 90 day audit trail of changes to the variables on both Environments and Pipelines. This will contain the time, user, and changed variables. Any encrypted variables will not be shown in the audit log.

Example of the Audit Log for an Environment

#### To view the audit log for an Environment 1. Go to **Environments** and click on the environment you want the log for. 2. Click the **View Audit Log** button at the top of the edit screen

When editing an environment, you have the option to view the Audit Log using the button shown.

#### To view the audit log for an Pipeline Stage 1. Go to **Environments** and click on **Pipeline Stages** tab 2. Open the three dot menu to the right of the Stage and click **View Audit Log**

Click the "View Audit Log" button on the Environment to see your Audit Log

# Alerts Alerts enable you to notify your team (or anyone else you want) when test runs pass, fail, or complete with any status. You can send messages to various places like Slack, Microsoft Teams, OpsGenie, or even webhooks.
#### Specifying Criteria in Alerts Testery uses the [SpEL expression language](https://docs.spring.io/spring-framework/docs/3.0.x/reference/expressions.html#expressions-language-ref) syntax for defining criteria in alerts. For example, you can use `environment.key=='foo'` to match a specific environment or `passCount < 100` or `failCount > 100` The following objects/parameters are available for use in expressions: ### Test Run
ParameterData TypeNullable
environment#environmentfalse
projectIdLongfalse
endedBooleanfalse
priorityInttrue
passCountIntfalse
failCountIntfalse
ignoredCountIntfalse
totalCountIntfalse
timeoutCountIntfalse
totalTestingTimeIntfalse
maxRunnersIntfalse
hasFlakyTestBooleanfalse
retryFailedTestsBooleanfalse
gitRefStringtrue
branchStringtrue
environmentIdLongfalse
testSuiteIdLongtrue
statusSetByUserBooleanfalse
includeTagsList<String>true
excludedTagsList<String>false
runnerConfigurationIdLongtrue
testFiltersList<String>false
defaultToParallelizeByFileBooleanfalse
testTimeoutSecondsIntfalse
timeoutMinutesIntfalse
recordVideoBooleantrue
### Environment
ParameterData TypeNullable
nameStringfalse
keyStringfalse
archivedBooleanfalse
pipelineStageIdLongtrue
urlStringtrue
# Settings ## Notification Settings #### **Setting Slack Member ID** You can receive slack notifications for test run status changes (when a test run gets submitted and completes) and when a team member leaves a message on a test run. You can choose to receive a notification to your email, a Slack DM or both. Once these notifications are turned on, all manual test runs will default to that setting. You have the option to turn them off/on at the time you manually start a test run. In Testery, go to Settings --> Profile tab. You will see the following notification settings. ![Testery Notification Settings](/files/XBcxbFUMWjdSHIm0Zbmr) Turn on email or slack notifications or both. If you turn on Slack notifications you will need to connect your Slack user to your Testery User. To do this... 1. Ensure your Slack Workspace is authorized. In the Testery App, go to Settings > Integrations. If you see, "Authorize Slack Integration," click that and complete that process. * If you see "Remove Slack Integration," then you are already authorized and can move forward. * This only needs to be done once in your Testery account. 2. In Slack, type the command `/testery connect`. Follow the prompts to complete the process and your users will then be connected. * Each Testery user that wants/needs Slack notifications must also use the `connect` Slash Command above. {% hint style="info" %} If Slash commands aren't working, you may need re-install your Slack App... 1. Go to Settings > Integrations 2. Click the "Remove Slack Integration" button. Complete the process. 3. Click the "Authorize Slack Integration" button. Complete the process. {% endhint %} {% hint style="info" %} These settings only apply to test runs kicked off manually. If there is a particular test run that is scheduled that you would like to follow, you would have to go into the test run once its activated and follow the test run. {% endhint %} # Test Plans Test plans in Testery allow you to create a plan where you can lay out the stages of testing, specify what tests or test suites will be in each stage, what environments the test should run in, etc. ## Create a Test Plan Click on Test Plans and click Add New Test Plan button. Give your test plan a name and objective so others on your team understand what the goal of the test plan. ![](/files/-McyWJ-1lfJ-I_mPMu4r) ## Test Plan Stages {% hint style="info" %} Defining stages allows you to control the order groups of tests will execute in and how they execute (parallel or sequential). Control the flow if certain tests pass or fail. Test plan stages gives users a ***test execution super power*** :superhero::zap: {% endhint %} On the Stages tab within a test plan create the Add New Stage button. Enter the following fields: * Stage Name - Give this group of tests a logical name. (Required) * Description - Describe what this group of tests will be testing. (Optional) * Execution Type - Parallel or Sequential. This tells the runner to run the tests within this stage in parallel or sequentially. (Required) * Always Run This Stage - Tells the runner whether this stage should always run no matter if tests within a previous stage pass or fail. For instance, if you have a cleanup tests stage you may want to set that stage to always run to ensure your data is cleaned up. (Required) ![Test Plan Stage](/files/-MeBBus8XRvr82oayDgK) Once stages are added they can be ordered by dragging and dropping using the menu to the left of the name. ![](/files/-MeBXptRvQiUB58LD_e0) ### Add Test Suites To Stage Once stages have been added, you can add test suites to each stage. The number of suites you can add is unlimited. You can create a new test suite or select an existing one to add to your stage. Select Add Test Suite button in your stage to add a test suite. {% hint style="info" %} We will be adding the ability to add tests without having a test suite. This will be in a future release. {% endhint %} Test suites, once added, can be ordered by dragging and dropping within a stage. This ordering only matters if the stage has a test execution of sequential or if you have more test suites within your stage than number of concurrent runners in your account. Then specify the following: * Environment - Select the environment you want the test suite to run in. You can select different environments for different test suites if needed. * You can also use Dynamic Environment Expressions to determine the environment at runtime. By selecting the "Use Dynamic Environment" toggle, you have the option to enter an expression using variables that are supplied via the CLI. For example, if you provide the environment `${my_variable_name}-qa` in the test plan, when you call `run-test-plan` from the Testery CLI, you will need to provide an argument `--variable="my_variable_name=environment1"` In this scenario, the test plan will run against the environment named `environment1-qa`. The environment will not be created for you, you are responsible for creating that environment before the test run, either using the testery app or CLI. * Project - Select the project that contains the existing test suite or the project where you want to create the test suite (depending on if you are selecting an existing suite or creating a new one) * Version - Select the version of test code you want to run in your test suite. You can choose Latest Deployed Version (if you have [sent deployment info to Testery](https://docs.testery.io/get-to-know-testery/deployments)) or Specific Version of your test code specifying the branch and commit. * Test Suite - If you have an existing test suite you want to add to the stage select Select Test Suite and choose your test suite from the dropdown. If you want to create a new test suite select Create Test Suite and select your options for the newly created test suite. ![](/files/-MeBoV6ZtwbbrQcDs5An) Once the test suites are added, you can order them by dragging and dropping. ![](/files/-MeBqeFs2JNnSOuL9ljd) {% hint style="success" %} Add test suites from different testing frameworks within the same stage. This supports running tests across teams that may use different tools/languages. {% endhint %} ## Run Test Plan Once your test plan is created with the appropriate stages and test suites...it's time to run your test plan! You can run your test plan from the test plan list selecting the Run Test Plan option from the action menu or click the Run Test Plan button on the Stages tab. Once a test plan is started you will be taken to the Runs tab of the test plan so you can see the progress as it completes. ![test plan run results](/files/-MeBtUpN4AQFey9g7oCM) ### Results Test plan run results will show: * Total number of tests that passed and failed at the test plan level * Duration of test plan * Number of tests that passed and failed in each test suite * If a suite was skipped (if no tests were run in suite) * If a test suite had an error (along with error message) * Duration of each test suite (duration of each test suite will not total duration of test plan due to parallelization) * Information: Environment, Branch, Git Ref of test code that was run # Test Selection Test Selection allows you to intelligently run only the tests that are relevant to your code changes. By defining rules in a `testery.yml` file, you can automatically select which tests to run based on which source files have changed. ## Overview When you push code changes to your repository, Testery can analyze which files were modified and automatically determine which tests should run. This helps you: * **Save time** by running only relevant tests instead of your entire test suite * **Get faster feedback** on changes that affect specific features * **Reduce costs** by minimizing unnecessary test execution * **Maintain confidence** by ensuring related tests always run when code changes ## How It Works Test Selection uses rules defined in a `testery.yml` configuration file at the root of your repository. Each rule specifies: 1. **When** to apply the rule (which source files trigger it) 2. **Then** what tests to run (using tags or filters) Testery compares your commit's changed files against these rules and selectively runs the appropriate tests. ## Configuration ### Basic Structure Create a `testery.yml` file in your repository root with the following structure: ```yaml test_selection: suites: - default: rules: - ruleName: when: - src_changes: - pattern1 - pattern2 then: - include_tags: - tag1 - tag2 ``` ### Example Configuration Here's a real-world example showing how to configure test selection rules: ```yaml test_selection: suites: - default: rules: - authenticationRule: when: - src_changes: - src/auth/**/* - src/login/**/* then: - include_tags: - authentication - login - paymentRule: when: - src_changes: - src/payment/**/* - src/checkout/**/* then: - include_tags: - payment - checkout - databaseRule: when: - src_changes: - src/repositories/**/* - src/models/**/* then: - test_filters: - Tests.Integration.Database.* ``` ## Rule Components ### When Conditions The `when` section defines which file changes should trigger the rule. Currently supported: #### src\_changes Specifies file path patterns to watch for changes. Supports two pattern types: **Glob Patterns** (default): ```yaml when: - src_changes: - src/**/*.js # All JavaScript files in src - components/**/*.tsx # All TypeScript React files - config/*.json # JSON files in config directory ``` **Regex Patterns**: Prefix with `regex:` to use regular expressions: ```yaml when: - src_changes: - regex: .*Grid\.[a-z0-9]+ # Files containing "Grid" with any extension - regex: ^src/.*Test\.cs$ # C# test files in src directory ``` ### Then Actions The `then` section defines which tests to run when the rule matches. You can use: #### include\_tags Run tests with specific tags: ```yaml then: - include_tags: - smoke - critical ``` #### exclude\_tags Exclude tests with specific tags: ```yaml then: - exclude_tags: - slow - flaky ``` #### test\_filters Run tests matching specific name patterns: ```yaml then: - test_filters: - MyApp.Tests.Unit.* - MyApp.Tests.Integration.Auth.* ``` ## Multiple Rules You can define multiple rules, and Testery will combine the results. If your changes match multiple rules, all matching tests will be selected (union behavior). ```yaml test_selection: suites: - default: rules: - frontendRule: when: - src_changes: - frontend/**/* then: - include_tags: - ui - e2e - backendRule: when: - src_changes: - backend/**/* then: - include_tags: - api - integration - sharedRule: when: - src_changes: - shared/**/* then: - include_tags: - ui - api ``` If changes are made to both `frontend/` and `shared/` files, tests tagged with `ui`, `e2e`, and `api` will all run. ## Best Practices ### 1. Start with Broad Rules Begin with high-level rules covering major components: ```yaml rules: - uiRule: when: - src_changes: - src/components/**/* then: - include_tags: - ui ``` ### 2. Add Specific Rules for Critical Paths Create targeted rules for high-risk areas: ```yaml rules: - paymentProcessingRule: when: - src_changes: - src/payment/processor.ts then: - include_tags: - payment - critical ``` ### 3. Use Regex for Complex Patterns When glob patterns aren't sufficient, use regex: ```yaml rules: - generatedFilesRule: when: - src_changes: - regex: .*\.generated\.(ts|js)$ then: - include_tags: - generated-code ``` ### 4. Combine Tags and Filters Use both tags and filters for fine-grained control: ```yaml rules: - databaseMigrationRule: when: - src_changes: - migrations/**/* then: - include_tags: - database - test_filters: - Tests.Migrations.* ``` ### 5. Consider Dependencies Include tests for code that depends on changed files: ```yaml rules: - coreLibraryRule: when: - src_changes: - lib/core/**/* then: - include_tags: - core - integration # Run integration tests too ``` ## Enabling Test Selection To enable test selection for a test run: 1. Add a `testery.yml` file to your repository with test selection rules 2. Ensure your tests are tagged appropriately 3. Enable the "Apply Test Selection Rules" option when creating or configuring your test run in Testery When test selection is enabled and rules are configured, Testery will: 1. Detect which files changed in your commit 2. Evaluate your test selection rules 3. Identify which rules match the changed files 4. Run only the tests specified by the matching rules ## Common Patterns ### Running All Tests for Specific Files ```yaml rules: - configChangeRule: when: - src_changes: - config/**/* - *.config.js then: - include_tags: - all ``` ### Framework-Specific Patterns **Pytest:** ```yaml rules: - pytestRule: when: - src_changes: - src/**/*.py then: - test_filters: - tests/unit/test_*.py ``` **NUnit:** ```yaml rules: - nunitRule: when: - src_changes: - src/**/*.cs then: - test_filters: - MyApp.Tests.Unit.* ``` **Cypress:** ```yaml rules: - cypressRule: when: - src_changes: - src/components/**/* then: - include_tags: - '@component-tests' ``` ## Troubleshooting ### No Tests Running If no tests run when you expect them to: 1. Check that your `testery.yml` is in the repository root 2. Verify file patterns match your actual file paths 3. Ensure tags in rules match tags on your tests 4. Check that "Apply Test Selection Rules" is enabled for your test run ### Too Many Tests Running If more tests run than expected: 1. Review your patterns for overly broad matches 2. Use more specific glob patterns or regex 3. Consider using `exclude_tags` to filter out tests 4. Check that multiple rules aren't combining unexpectedly ### Pattern Not Matching If patterns don't seem to match your files: 1. Verify file paths are relative to repository root 2. Test glob patterns (they're case-sensitive) 3. For regex patterns, ensure the `regex:` prefix is present 4. Check that special characters are properly escaped ## See Also * [Tags](/get-to-know-testery/tags) - Learn about tagging your tests * [Test Runs](https://github.com/testery/testery-docs/blob/master/get-to-know-testery/test-runs.md) - Creating and configuring test runs * [Projects](/get-to-know-testery/projects) - Configuring your Testery projects # Test Stacks Managing your dependencies with Test Stacks Every Testery Test Run has a **Test Stack.** A Test Stack is a collection of operating systems and softwares to enable your Test Framework to run. A full list of supported Test Stacks is available in the Testery app You never need to specify a Test Stack - if you don't provide one, a default stack will be selected for you. However, you have the ability to control the versions of Node.js and Chrome that your tests will run against by providing a testery.yml file in your source repository. For example, a Cypress run might be assigned, by default, to the `node-chrome.120-java.17-nodejs.18` test stack. This uses Node.js 18 and Chrome 120. If you wanted to test your code against Chrome 121, you would provide the following in your testery.yml file. ``` test_stack: configuration_key: node-chrome.121-java.17-nodejs.18 ``` One benefit of this approach is allowing you to test new versions of dependencies on a branch. Testery will respect the testery.yml file in your branch, allowing you to fix issues on your branch, while using the default Test Stack on your `main` branch. # Python How to choose which version of Python to use on your runners ### Changing your Python Version Using Test Stacks is easy, you just need to provide a `testery.yml` file in your source repository. Add the following content... ``` test_stack: configuration_key: python-chrome.115-java.17-nodejs.18-python.3.8 ``` # Node.js 16 End of Life Handling the Upgrade to Node 18 On September 11, 2023, Node.js will reach its end of life date. For details, see On September 11, 2023 at 12pm Central Time, Testery will begin running all linux Node.js workloads against Node.js 18. We encourage all Testery customers to migrate before this date, however, we will maintain Node 16 support until September 30, 2023 ### Switching to Node.js 18 before September 11, 2023 Migrating your code to the new Test Stack is easy, you just need to provide a testery.yml file in your source repository. Add the following content ``` test_stack: configuration_key: node-chrome.114-java.17-nodejs.18 ``` ### Remaining on Node.js 18 after September 11, 2023 You may use a legacy runner until September 30, 2023 by providing a testery.yml file in your source repository. Add the following content ``` test_stack: configuration_key: node-chrome.114-java.17-nodejs.16 ``` # Default Chrome Version Change Handling the Upgrade to Chrome 120 On April 19, 2024 at 12pm Central Time, Testery will change the default Chrome version for all Linux Runners to Chrome version 120 Chrome 114, 115, 117, and 119 will remain available on the testery platform through the end of April for you to use if needed, by specifying a non-default test stack. ### Switching to Chrome 120 before April 19, 2024 Migrating your code to the new Test Stack is easy, you just need to provide a testery.yml file in your source repository. Add the following content (example for node), check this page to find your exact configuration. ``` test_stack: configuration_key: node-chrome.120-java.17-nodejs.18 ``` ### Remaining on Chrome 114 after April 19, 2024 You may use a legacy runner until April 30, 2024 by providing a testery.yml file in your source repository. Add the following content (example for node) check this page to find your exact configuration. ``` test_stack: configuration_key: node-chrome.114-java.17-nodejs.16 ``` # Python 3.8 End of Life Handling the Upgrade to new python On October 7th, 2024 python 3.8 reached its end of life date. For details, see On October 31st, 2024 at 12pm Central Time, Testery decommission test stack images running with python 3.8 We encourage all Testery customers to migrate before this date, however, we will maintain Python 3.8 support until October 31, 2024 ## Switching to Python 3.9 before October 31, 2024 Migrating your code to the new Test Stack is easy, you just need to provide a **testery.yml** file in your source repository. Add the following content: ``` test_stack: configuration_key: python-chrome.123-java.17-nodejs.20-python.3.9 ``` ## Defaulting to new version without a testery.yml If you choose to not migrate beforehand, the test runs will automatically pick up the new default test stack **python-chrome.123-java.17-nodejs.20-python.3.9** after October 31st. This may result in test run failures if you have not checked compatibility beforehand. # Deployments A Testery Deployment is a representation of each deployment that happens in any of your environments. When a build system tells Testery when a deployment happens, it gives Testery users more insight into knowing what is deployed where, in what environment, what version of code, and what tests have been run against a deployment. It also allows users that do not normally have much control over the build process to make decisions and easily change what tests get run, in what environments and when they should run.\
**Benefits of sending deployment info to Testery** * Allows someone like who normally has zero control over when something gets deployed to control what tests get run when something gets deployed. This is many times the case for QA teams. * Enables the ability to automatically run automated tests when a project is deployed to any environment. * Ties deployment and environment data to automated testing data that enables some powerful insights into what is being tested and where. **Sending deployment info to Testery**\ The easiest way to send deployment info to Testery is to make a cURL request from your build system to Testery after every deployment. Here is an example: ``` testery create-deploy --token --project --environment --commit --branch ``` **Scheduling a test run using deployment info**\ Testery will recognize if you have sent deployment information about your deploys and offer additional functionality when scheduling test runs. The Schedule option to Run on Deploy is available once at least one project has deployment info. Each project that sets up a schedule using Run on Deploy will need deployment info to work. The images below show a new option for Latest Deployed Version when creating a schedule to run on deploy or run on interval. When this option is selected it will run the latest version of tests in the specified environment. This is possible because now Testery is aware of the deployments. ![](/files/-MK1mj44_L1HQj6gzvl_) ![](/files/-MK1mkwStIvOuzcQrF9B) **What if code and tests are in different projects?**\ If your code and test code are in different projects, you can kick off a test run from your test code project when the code project gets deployed. # System Variables Testery sets System Variables that get passed to the tests as standard environment variables. They may be accessed using the standard method of accessing environment variables in your testing framework. These values may be useful to you for logging and debugging purposes. If you're looking for how to set variables in Testery that can vary by environment and will get passed in to your test runs, see[ Environments - Variables](/get-to-know-testery/set-up-your-test-environments#variables). * `IS_TESTERY` - Set to TRUE if the tests are running in the Testery platform. Useful for writing tests that perform differently when running in the platform. * `TESTERY_CHROME_ARGS` - Arguments that should be passed to Chrome if you are manually launching it in your tests. * `TESTERY_TEST_RUN_KEY` - A unique key string given to each test run. * `TESTERY_TEST_RUN_ID` - A unique numeric id given to each test run. * `TESTERY_ENVIRONMENT_URL` - The Environment URL set on the Environment. See Setting Up Environments. # Tags Tags are useful for classifying different types of tests. # Screenshots Testery supports taking screenshots and having those screenshots attached to your test results. This works by placing screenshots in a `screenshots` folder in your current directory using the framework of your choice. Testery highly recommends adding screenshots to your tests as this can often help you troubleshoot test failures that occur due to pages not loading quickly, elements changing, sites being down, etc. {% tabs %} {% tab title="SpecFlow" %} ```csharp [AfterScenario("screenshot")] public static void takeScreenshot() { Screenshot s = ((ITakesScreenshot)((IWebDriver)FeatureContext.Current["driver"])).GetScreenshot(); string fileName = ScenarioContext.Current.ScenarioInfo.Title.Replace(" ","") + DateTime.Now.ToString("_MMddyyyyHHmmss") + ".png"; try { string screenshotDir = ScenarioContext.Current.ScenarioContainer.Resolve().WorkDirectory.ToString() + @"\screenshots"; if (!Directory.Exists(screenshotDir)) { Directory.CreateDirectory(screenshotDir); } s.SaveAsFile(screenshotDir + @"\" + fileName, ScreenshotImageFormat.Png); TestContext.Out.WriteLine($"-> info: added screenshot ({fileName}) to dir ({screenshotDir})");; } catch(Exception e) { TestContext.Out.WriteLine($"-> error: ({e})"); } } ``` {% endtab %} {% tab title="webdriver.io" %} ```javascript let fs = require('fs'); if (!fs.existsSync("./screenshots")){ console.log("Screenshots dir being created."); fs.mkdirSync("./screenshots"); } browser.saveScreenshot("./screenshots/" + Math.random() + ".png"); ``` {% endtab %} {% endtabs %} In the SpecFlow example above, the after scenario hook will be executed right after a test either passes or fails if when the @takeScreenshot tag is included on a feature file. The ("screenshot") can be removed from the \[AfterScenario] annotation and that will run the hook without having to specify the @screenshot tag. # Uploading Test Artifacts to Testery There are several reasons to upload artifacts to Testery: * You are uploading test assemblies because you are using a framework which requires compiled tests (e.g. SpecFlow, TestNG). * You are uploading test assemblies because you are unable to provide GitHub / BitBucket access. * You are uploading additional information about test runs that you'd like associated with the test run (e.g. code coverage reports). [Here](https://docs.testery.io/testery-integration/testery-cli-docs#upload-build-artifacts) are more details on how to upload artifacts using the Testery CLI # Setting Test Execution Priority Testery enables you to set a Test Execution Priority to optimize how your test runs are queued up. When there are multiple test runs in a queued state, the test runs with the lowest numbered priority will be executed first. If you would like multiple test runs to execute at the same time, make sure that you have also [set the number of parallel tests](/get-to-know-testery/setting-the-number-of-parallel-tests) to a number lower than your account limit. #### Setting Default Priority for a Pipeline Stage To set the priority for different Pipeline Stages, navigate to `Environments` and click `Pipeline Stages`.
![](/files/-MMa2RcTc73zliP3ss7t) Here you can select a Test Execution Priority for the Pipeline Stage. When two test runs are queued up, the test run with the lowest numbered Test Execution Priority will be executed first. #### Setting Priority for an Ad Hoc Test Run Test Execution Priority may also be set for an individual ad hoc test run. To do this, expand `Show Advanced Options` and set the Priority for the test run. ![](/files/-MMa3hDkaRN7En870-SW) #### Setting Priority for a Scheduled Test Run Test Execution Priority can be set for [Schedules](/get-to-know-testery/schedules) as well. To do this, expand `Show Advanced Options` when creating or editing a schedule. ![](/files/-MMa4Uu2v7Vph3FP3Usn) # Setting the Number of Parallel Tests Testery enables you to set the number of parallel tests to optimize how your test runs are queued up or throttle test runs so they don't overwhelm the system-under-test. #### Setting Number of Parallel Tests for a Scheduled Test Run When editing the [scheduled test run](/get-to-know-testery/schedules), select `Show Advanced Options` and specify the Number of Parallel tests. Be sure to set this number to less than the total available for your account if you'd like to be able to have multiple test runs running at the same time. ![](/files/-MMa6P4WVUAE7XcLDE32) #### Setting the Number of Parallel Tests for an Ad Hoc Test Run You can also set the number of parallel tests when running an ad hoc test run. To do this, when creating the test run, expand `Show Advanced Options` and specify the desired number of parallel tests. ![](/files/-MMa6m_KTs-i1dfkfWpO) # Built-In Integrations with Testery Testery has built-in integrations with the following platforms: * Jira * GitHub * BitBucket * Bitbucket Server * Microsoft Teams * Slack * OpsGenie We're always adding new ones, so if you don't see what you're looking for, contact or roll your own using [Testery CLI Docs](/integrations/testery-cli-docs) or [Testery REST API Docs](/integrations/testery-rest-api). # Jira Testery's built-in Jira integration allows user to create stories for errant tests and bugs found while testing. Stories can be tied to individual or groups of tests. ## {% embed url="" %} ## Setup 1. Navigate to the Integrations tab in Settings and click the "Authorize Jira Integration" button.\ 2. Review the requested permissions and click the "Accept" button when ready. You will be redirected back to the Testery app and your Jira Integration setup is complete.\ ![](/files/RCBgB8pmDkjSiGhwQtSA) ## How to Create a Jira Story There are 3 ways to do this... ### From Test Results 1. When viewing a test result, choose any "Review Status".\ ![](/files/DyyRXJc1NO8b1P2Dxnu0)![](/files/0ixirN0HiVlyf9PdkHZ9) 2. You will be prompted to create a Jira Issue, click "Yes".\ ![](/files/fnMFaU1cGBtCGIm5xXvK) ### From a Test Run Analysis * Under the Analysis tab when viewing a test run, click the "Create Jira Issue" button.\ ![](/files/BOAW9G17n4DHv3TuQQtm) * On a Test Run Analysis Slack message, click the "Create Jira Issue" button.\ ![](/files/5fUjdde5VENTaXtDgITm) # Slack ## Invite Testery App to a Private Slack Channel In order for Testery to send alerts to a channel, the Testery bot must be invited. There are several ways to do this in Slack, but the easiest is to `@-mention` the Testery app.\ \ For example, you can send something like this to the channel,
When you send a message to `@Testery` and the app isn't invited to the channel yet, you'll get a notice to invite the app's bot to the channel. ![](/files/J4Jydi8NrFtafsYi1sK8) Click **Invite Them**, and now Testery will be able to send alerts to that channel. # CI/CD Integration Guides {% hint style="info" %} Don't see your CI/CD on this list? We can integrate with nearly any CI/CD that's out there. Email and let us know what you'd like to see. {% endhint %} # Azure Devops Pipelines #### Trigger a Test Run from Azure Devops Pipelines 1. Create the Azure Pipelines project the way you normally would. 2. Add the Testery API Token as a secret variable in your project (follow these steps: ). The Testery API token can be found on the Settings tab on the Integrations sub-tab. 3. Add the following task to your `azure-pipelines.yml`. Make sure you update the command arguments to match your project. ```yaml - task: PowerShell@2 inputs: displayName: Run Tests in Testery targetType: inline script: | pip -q install testery --upgrade testery upload-build-artifacts --token "${env:TESTERY_TOKEN}" --project "example-specflow-dotnet-core" --path "$(Build.SourcesDirectory)\Web.Specs\bin\Release\netcoreapp3.0\" --build-id "$(Build.BuildNumber)" --zip-dir testery create-test-run --token "${env:TESTERY_TOKEN}" --git-ref "$(Build.SourceVersion)" --project "example-specflow-dotnet-core" --environment "dev" --build-id "$(Build.BuildNumber)" env: TESTERY_TOKEN: $(TesteryToken) ``` # Set Up CircleCI to Run Testery Tests How-to trigger a Testery test run from a CircleCI build. Testery integrates with CircleCI via the [Testery CLI](/integrations/testery-cli-docs). You'll need to define a GitHub Actions workflow. This is done by creating a yml file in the `.github/workflows` folder in your repository. **Step 1. Retrieve your Testery API Token.** To retrieve your token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. Keep this token secure as it allows access to Testery on your behalf. **Step 2. Store your Testery API Token as a** [**secret environment variable in CircleCI**](https://circleci.com/docs/2.0/env-vars/#section=projects)**.** **Step 3.** **Add steps to your CircleCI job to install the** [**Testery CLI**](/integrations/testery-cli-docs) **and create the test run.** Here is a sample `.circleci/config.yml` ```yaml version: 2 # use CircleCI 2.0 jobs: # A basic unit of work in a run build: # runs not using Workflows must have a `build` job as entry point working_directory: ~/circleci-demo-python-django docker: # run the steps with Docker - image: circleci/python:3.6.4 steps: # steps that comprise the `build` job - checkout # check out source code to working directory - run: sudo chown -R circleci:circleci /usr/local/bin - run: sudo chown -R circleci:circleci /usr/local/lib/python3.6/site-packages - restore_cache: # Read about caching dependencies: https://circleci.com/docs/2.0/caching/ key: deps9-{{ .Branch }}-{{ checksum "Pipfile.lock" }} - run: command: | pip install testery --upgrade testery create-test-run --token "${TESTERY_TOKEN}" --git-ref "$CIRCLE_SHA1" --build-id "$CIRCLE_BUILD_NUM" --environment "" --project "" --wait-for-results ``` # Jenkins Configure Jenkins in minutes to run your end-to-end tests concurrently via Testery Cloud Testery integrates with Jenkins via the [Testery CLI](/integrations/testery-cli-docs) or [Testery REST API](/integrations/testery-rest-api). We strongly recommend using the CLI as it is usually faster to get running and provides better integration functionality. The CLI requires Python 3.6 or greater. **Step 1. Retrieve your Testery API Token.** To retrieve your token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. Keep this token secure as it allows access to Testery on your behalf. If you don't already have a Testery account, you can sign up for a free account in a few minutes [here](https://testery.io/signup). **Step 2. Store your Testery API Token in Jenkins as a secret.** For more information on the various options for doing this, see You can now access this token as follows in your `Jenkinsfile`: ```groovy pipeline { // ... environment { TESTERY_API_TOKEN = credentials('testery-api-token') } // ... } ``` **Step 3.** **Add steps to your Jenkins pipeline to install the** [**Testery CLI**](/integrations/testery-cli-docs) **and tell Testery when there are deployments ready to be tested.** Here is a sample `Jenkinsfile`: ```groovy pipeline { agent any stages { // ... stage('Testery Tests') { steps { script { echo "Running E2E tests..." def commit = sh(returnStdout: true, script: 'git rev-parse HEAD').trim() sh 'pip install testery --upgrade' sh 'testery create-deploy --token ${TESTERY_API_TOKEN} --project --environment --build-id ${BUILD_ID} --commit ${commit}' } } } // ... } } ``` **Step 4. Configure Testery to trigger a test run whenever there are** [**deployments**](/get-to-know-testery/deployments)**.** Go to **Schedules**, click **Add New Schedule** and then specify that you want the tests to **Run on Deploy**. The Run on Deploy option will only be available after you have completed the previous steps. ![](/files/-MVXIye7qAFRjoBENSdZ) **Step 5. You're all set!** Tests will now run whenever your Jenkins pipeline is run. If you get stuck and would like any assistance, please don't hesitate to reach out to . # GitHub Actions Testery integrates with GitHub Actions via the [Testery CLI](/integrations/testery-cli-docs). You'll need to define a GitHub Actions workflow. This is done by creating a yml file in the `.github/workflows` folder in your repository. **Step 1. Retrieve your Testery API Token.** To retrieve your token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. Keep this token secure as it allows access to Testery on your behalf. **Step 2. Store your Testery API Token as a** [**GitHub Actions secret**](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets)**.** This is referenced in your GitHub Actions workflow as follows: ```yaml jobs: build: ... env: TESTERY_TOKEN: ${{ secrets.TesteryTokenProd }} ... ``` **Step 3.** **Add steps to your GitHub Actions workflow to install the** [**Testery CLI**](/integrations/testery-cli-docs) **and create the test run.** Here is a sample `.github/workflows/testery.yml` ```yaml name: Testery Tests on: [push] jobs: build: runs-on: ubuntu-latest env: TESTERY_TOKEN: ${{ secrets.TesteryTokenProd }} steps: - uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v1 with: python-version: "3.x" - name: Install dependencies run: | python -m pip install --upgrade pip pip install testery --upgrade - name: Deploy Application run: | ... logic to deploy your application ... - name: Run Testery Tests run: | # Create test run. testery create-test-run --api-url https://api.testery.io/api --token "$TESTERY_TOKEN" --include-tags "smoke" \ --variable "TEST_URL=${TEST_URL}" \ --git-ref "$GITHUB_SHA" --project "example-webdriverio" --environment "testery-dev" \ --wait-for-results ``` # Octopus Deploy #### Step 1. Upload Test Artifacts to Testery #### Step 2. **Add the** Testery - Create Test Run step to your process. * Select **Deployments → Process → Add Step.** * Search for "testery". Install and add the **Testery - Create Test Run** step.
**Step 3. Configure the create test run step.** * Set **Testery Token** to your Testery API Token. To retrieve your Testery token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. Keep this token secure as it allows access to Testery on your behalf. ![](/files/-MK5gKPZRhMDojZLaxJL) * Set **Testery Project Name** to match the name of your [Project](/get-to-know-testery/projects) in Testery. * Set **Testery Environment** to match the name of your [Environment](/get-to-know-testery/set-up-your-test-environments) in Testery. It is common when integrating with Octopus Deploy to have the name of the Testery environment match either your Octopus Deploy environments or tenant names. In these cases you may want to use an Octopus system variable such as `#{Octopus.Environment.Name}`. * Optionally set **Testery Include Tags** to run a specific set of [Tags](/get-to-know-testery/tags). This is useful if you only want to run smoke tests instead of full regression, etc. * Set **Build ID** to a unique number. This could come from your build system or from Octopus Deploy. When [uploading test artifacts to Testery](/get-to-know-testery/uploading-test-artifacts-to-testery). * Set **Git Reference** to the commit hash of the repository that includes the version of the tests you want to run. {% hint style="info" %} You may want to get the Git Reference from a previous step's package build information. To do so, set the value to\ `#{each buildInfo in Octopus.Deployment.PackageBuildInformation}#{buildInfo.VcsCommitNumber}#{/each}`\ \ For more information about publishing build information to Octopus see {% endhint %} * Ignore **Testery API URL** as this is for testing purposes only. * You may want to set conditions so that this step only runs in certain environments / situations. # TugboatQA Using Tugboat and Testery is one of the coolest, easiest, and fastest ways to run your E2E Cypress tests on every PR. Follow these four steps and in about 10 minutes, you'll have Cypress tests running **Step 1. Update the init section for your service in .tugboat/config.yml to install Python3 (if not already present).** ``` services: your-service-name: commands: init: apt-get install -y python3 apt-get install -y python3-pip ``` **Step 2. Update the build section for your service in .tugboat/config.yml to run the tests.** ``` services: your-service-name: build: | # Install the Testery CLI pip3 install testery --upgrade # Register the preview environment with Testery. testery update-environment --create-if-not-exists --token "$TESTERY_TOKEN" --key "${TUGBOAT_PREVIEW}" --name "${TUGBOAT_PREVIEW}" --variable "TUGBOAT_DEFAULT_SERVICE_URL=${TUGBOAT_DEFAULT_SERVICE_URL}" # Start a test run. testery create-test-run --token "$TESTERY_TOKEN" --git-ref "$TUGBOAT_PREVIEW_SHA" --project "testery" --environment "${TUGBOAT_PREVIEW}" ``` **Step 3. Create a free Testery account and get your Testery token.** Click [here](https://testery.io/signup?plan=free) to sign up for a free account. Be sure to select Cypress as your test runner. Get your Testery token by clicking on `Settings` -> `Integrations` -> `Show API Token`. Copy the token that is displayed.![](https://blog.testery.io/content/images/2020/06/image.png) **Step 4. Save your Testery token in the Tugboat project settings.** In Tugboat, go to your Repository Settings and add your Testery token as an Environment Variable. Be sure to use the same Environment Variable name that was used in Step 2 (we called it `TESTERY_TOKEN`).![](https://blog.testery.io/content/images/2020/06/image-1.png) **Step 5. You're All Set!** You should now be able to run your Cypress tests on every PR. If you have any questions or need further assistance, please contact [support@testery.io](mailto://support@testery.io). You'll now see test results directly on the pull request.\ ![](https://blog.testery.io/content/images/2020/06/image-4.png) Clicking `Details` will show you the results. ![](https://blog.testery.io/content/images/2020/06/image-2.png) # Testery CLI The Testery CLI is available as a pip package here: . To install, run ```bash pip install --upgrade testery ``` {% hint style="info" %} If pip install errors you might need to run pip3 install depending on which versions of python you have on your machine. {% endhint %} You can see the command line help by running, ```bash > testery --help Usage: testery [OPTIONS] COMMAND [ARGS]... Testery CLI Kick off test runs from your CI/CD platform and run them on Testery's next-generation, cloud-based testing grid. Options: --help Show this message and exit. Commands: add-file Adds a file to a test run. cancel-test-run Cancels a test run. create-deploy Creates a deploy for a project and... create-environment Creates an environment where tests can be... create-test-run Submits a Git-based test run to the Testery... delete-environment Deletes an environment. list-environments Returns a list of environments. load-users monitor-test-run monitor-test-runs report-test-run Outputs individual test results for the... update-environment Updates an environment where tests can be... upload-build-artifacts Uploads a file or directory of build... verify-token Verifies your username and authentication... ``` #### Create a Deployment Creating a deployment is useful for telling Testery when you've pushed new code into your test environment. What tests you want to run can be configured with [Schedules](/get-to-know-testery/schedules). ``` > testery create-deploy --help Usage: testery create-deploy [OPTIONS] Creates a deploy for a project and environment. Options: --token TEXT Your Testery API token. [required] --project TEXT The project key of the repo being deployed. [required] --git-provider TEXT The Git provider used for the repository that is being deployed. Should be GitHub or BitBucket --git-owner TEXT The organization owner in Git for the repository that is being deployed. --git-repo TEXT The repository name that is being deployed. --build-id TEXT The build the atifact should be associated with. --environment TEXT Which environment you would like to run your tests against. [required] --commit TEXT The Git commit that was deployed. --branch TEXT The Git branch the deploy came from. --help Show this message and exit. ``` #### Create an Environment ``` Usage: testery create-environment [OPTIONS] Creates an environment where tests can be run. Options: --token TEXT Your Testery API token. [required] --name TEXT The display name for the environment. [required] --key TEXT An identifier for the environment. This is used when creating test runs to indicate where tests should run. [required] --pipeline-stage TEXT The name of a pipeline stage to associate this environment to. --variable TEXT A variable to add to the environment. Specified as "KEY=VALUE". To encrypt value, pass in "secure:KEY=VALUE", Multiple variables can be provided. --maximum-parallel-test-runs INTEGER The maximum number of test runs that can run in parallel on this environment. --help Show this message and exit. ``` #### Create a Test Run ``` Usage: testery create-test-run [OPTIONS] Submits a Git-based test run to the Testery platform. Options: --token TEXT Your Testery API token. [required] --git-ref TEXT The git commit hash of the build being tested. --git-branch TEXT The git branch whose latest commit you want to run. --test-name TEXT The name you want to use on the Git status. --wait-for-results If set, the command will poll until the test run is complete. --project TEXT Legacy option. Use --project-key instead. --project-key TEXT The project key. --test-suite TEXT Name of Test Suite to use for this test run. --environment TEXT Legacy option. Use --environment-key instead. --environment-key TEXT Which environment you would like to run your tests against. --include-tags TEXT List of tags that should be run. --exclude-tags TEXT List of tags that should excluded from the test run. --copies INTEGER The number of copies of the tests to submit. --build-id TEXT A unique identifier that identifies this build in your system. --output TEXT The format for outputting results [json,pretty,teamcity] --fail-on-failure When set, the testery command will return exit code 1 if there are test failures. --include-all-tags When set, overrides the testery.yml and runs all available tags. --parallelize-by-file Pass this flag if you want the test run to parallelize by file/feature. --parallelize-by-test Pass this flag if you want the test run to parallelize by test/scenario. --timeout-minutes INTEGER The maximum number of minutes this test run can take before it is killed automatically. --test-timeout-seconds INTEGER The maximum number of seconds a test can take before it is killed automatically. --runner-count INTEGER Specify number of parallel runners to use in for this testrun. --variable TEXT A variable to add to the enviroment. Specified as "KEY=VALUE". To encrypt value, pass in "secure:KEY=VALUE", Multiple variables can be provided. --test-filter-regex TEXT A regular expression to be used for filtering tests. --help Show this message and exit. ``` #### Upload Build Artifacts If your project is using a compiled language (like .Net, Java) or if you aren't attaching a code repository to your project, then you'll have to upload your build artifacts to Testery before you can run a test. To do that you can use the **upload-build-artifacts** command of the Testery CLI. ``` > testery upload-build-artifacts --help Usage: testery upload-build-artifacts [OPTIONS] Uploads a file or directory of build artifacts and associates them with the specified build-id Options: --token TEXT Your Testery API token. [required] --project TEXT Legacy option. Use --project-key instead. --project-key TEXT The project key. --branch TEXT The Git branch the build came from. --build-id TEXT The build the artifact should be associated with. [required] --path TEXT The path to the file or directory you want to upload. [required] --zip-dir Creates a zip file of the directory contents before uploading. --help Show this message and exit. ``` #### Load Users Use the **load-users** command to bulk-invite users into your Testery account. Each user is added with the `USER` role. This is useful when onboarding a team or organization. ``` > testery load-users --help Usage: testery load-users [OPTIONS] Options: --token TEXT Your Testery API token. Falls back to ~/.testery/credentials or $TESTERY_API_TOKEN. --profile TEXT Profile name in ~/.testery/credentials to use for auth. --user-file TEXT List of email addresses to load as user accounts. --help Show this message and exit. ``` The `--user-file` should point to a plain text file containing one email address per line: ``` alice@example.com bob@example.com carol@example.com ``` For example: ```bash testery load-users --user-file users.txt ``` # Testery REST API The Testery API is a REST API. ### Authenticate with the Testery REST API The Testery API is a REST API that uses token-based authentication. To retrieve your token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. Keep this token secure as it allows access to Testery on your behalf. When making calls to the Testery API, pass your token in the authorization header. For example, to retrieve your list of recent test runs via curl, ```bash curl -H 'Accept: application/json' -H 'Content-Type:application/json' \ -H 'Authorization: Bearer ' \ "https://api.testery.io/api/test-runs?offset=0&limit=10" ``` Once you've confirmed you can authenticate and make a request, check out all the things you can do with [API End Points](/integrations/testery-rest-api/testery-rest-api). # Testery REST API Resources A description of the available Testery REST API end points. #### Test Run Details ## Get Single Test Run `GET` `https://api.testery.io/test-runs/:id` Returns a single test run by id. #### Path Parameters | Name | Type | Description | | ---- | ------ | ------------------------------------------ | | id | string | ID of the cake to get, for free of course. | #### Headers | Name | Type | Description | | ------------- | ------ | ------------------------------------------------------------------ | | Authorization | string | Set to "Bearer \". See \[\[Finding Your API Token]]. | {% tabs %} {% tab title="200 Test Run successfully retrieved" %} ```javascript { "id": 55676, "name": "...", "testName": null, "key": "ABCDABCD75349414053EB9961013126945472A93", "startTime": "2020-05-04 23:16:33+0000", "endTime": "2020-05-04 23:22:59+0000", "createTime": "2020-05-04 23:15:10+0000", "accountId": 123, "projectId": 123, "status": "PASS", "copies": 1, "passCount": 30, "failCount": 0, "ignoredCount": 0, "codeCoverageName": null, "codeCoverageVersion": null, "codeCoveragePercentage": null, "codeCoverageTotalStatements": null, "codeCoverageCoveredStatements": null, "totalTestingTime": 1036000, "isSchedule": null, "totalCount": 30, "gitRef": "abcd4e87d8d928a21a393b0def6ab0c0a21d35a2", "buildKey": "23", "environment": "dev", "statusInfo": null, "includeTags": [], "complete": true } ``` {% endtab %} {% tab title="404 Test Run not found" %} ``` NO CONTENT ``` {% endtab %} {% endtabs %} **List of Test Runs** ## Get List of Test Runs `GET` `https://api.testery.io/test-runs?offset=##&limit=##` Returns the list of test runs. Be sure to specify an offset and limit as these are required parameters. #### Query Parameters | Name | Type | Description | | ---------------------------------------- | ---- | -------------------------------------------------------------------------------------------------------------------------- | | offset\* | int | The starting index for which test run to return. This index starts with 0 and should increment by the limit for each page. | | limit\* | int | The number of test runs to return in each call. | #### Headers | Name | Type | Description | | ------------- | ------ | ------------------------------------------------------------------ | | Authorization | string | Set to "Bearer \". See \[\[Finding Your API Token]]. | {% tabs %} {% tab title="200 Test Run successfully retrieved" %} ```javascript { "id": 55676, "name": "...", "testName": null, "key": "ABCDABCD75349414053EB9961013126945472A93", "startTime": "2020-05-04 23:16:33+0000", "endTime": "2020-05-04 23:22:59+0000", "createTime": "2020-05-04 23:15:10+0000", "accountId": 123, "projectId": 123, "status": "PASS", "copies": 1, "passCount": 30, "failCount": 0, "ignoredCount": 0, "codeCoverageName": null, "codeCoverageVersion": null, "codeCoveragePercentage": null, "codeCoverageTotalStatements": null, "codeCoverageCoveredStatements": null, "totalTestingTime": 1036000, "isSchedule": null, "totalCount": 30, "gitRef": "abcd4e87d8d928a21a393b0def6ab0c0a21d35a2", "buildKey": "23", "environment": "dev", "statusInfo": null, "includeTags": [], "complete": true } ``` {% endtab %} {% tab title="400: Bad Request " %} ```javascript { // Response } ``` {% endtab %} {% endtabs %} # Microsoft Teams # Testery MCP Server The Testery MCP Server allows AI assistants like Claude to interact directly with your Testery account. Using the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), you can manage projects, trigger test runs, review results, and more — all through natural language conversation. ## Prerequisites * [Claude Code](https://docs.anthropic.com/en/docs/claude-code) installed * A [Testery account](https://testery.com/signup?plan=2) ## Setup ### Step 1: Add the MCP Server Run the following command to add the Testery MCP server to Claude Code: ```bash claude mcp add testery --transport http "https://mcp.testery.io/mcp" ``` ### Step 2: Authenticate In your Claude Code session, type: ``` /mcp ``` This will open an authentication flow where you can log in with your Testery account. Once authenticated, Claude will have access to your Testery data. ## What You Can Do Once connected, you can ask Claude to interact with Testery using natural language. For example: * "List all my Testery projects" * "Create a new test run for project X" * "Show me the test results for the latest test run" * "List environments for project Z" * "Get details about test suite A" The MCP server supports the following Testery resources: | Resource | Capabilities | | ----------------- | --------------------------------- | | **Projects** | List, get, create, update, delete | | **Test Runs** | List, get, create, delete | | **Test Results** | List, get | | **Environments** | List, get, create, update, delete | | **Test Suites** | List, get, create, update, delete | | **Users** | List, get | | **Organizations** | List, get | ## Troubleshooting **Authentication Issues** * Run `/mcp` again to re-authenticate * Make sure you are logging in with the correct Testery account **Permission Denied** * Check your organization and project access rights in Testery # Claude Plugin The Testery Claude Plugin turns [Claude Code](https://docs.anthropic.com/en/docs/claude-code) into a hands-on Testery teammate. It adds a full set of slash commands and skills that let Claude drive the Testery platform — create test runs, monitor them, manage environments and schedules, upload artifacts — and scaffold or author [playwright-bdd](https://github.com/vitalets/playwright-bdd) tests, all from natural-language conversation. {% hint style="info" %} Looking for read-only, conversational access to your Testery data instead? See the [Testery MCP Server](/integrations/testery-mcp-server). The plugin and the MCP server complement each other — the plugin runs the Testery CLI for write operations, and the MCP server is great for inspection. {% endhint %} ## Prerequisites * [Claude Code](https://docs.anthropic.com/en/docs/claude-code) with plugin support installed * A [Testery account](https://testery.com/signup?plan=2) * The [Testery CLI](/integrations/testery-cli-docs) on your `PATH` (the `/testery-onboard` command will install it for you if it is missing): ```bash pip install testery ``` ## Install From inside a Claude Code session, add the Testery marketplace and install the plugin: ``` /plugin marketplace add testery/testery-skills /plugin install testery@testery ``` Claude Code clones the repo, registers the marketplace, and installs the plugin. Every `/testery-*` and `/bdd-*` command and skill becomes available immediately. ### Install from a local clone If you have already cloned the [plugin repository](https://github.com/testery/testery-skills): ``` /plugin marketplace add /absolute/path/to/testery-claude-plugin /plugin install testery@testery ``` ### Update or uninstall ``` /plugin update testery@testery /plugin uninstall testery@testery /plugin marketplace remove testery ``` ## Quickstart After installing the plugin, run these two commands to authenticate and wire up a project: ``` /testery-onboard # sign up or log in on testery.io, persist your API key /testery-init # scaffold playwright-bdd + register the project on Testery ``` `/testery-onboard` opens the Testery signup/login page, walks you through generating an API key, then saves it to `~/.testery/credentials` and your shell profile so it persists across sessions. `/testery-init` scaffolds a playwright-bdd project, runs a local smoke test, registers the project on Testery, and optionally fires your first cloud run. Prefer to set the token yourself? Export it before starting Claude Code: ```bash export TESTERY_TOKEN= # bash / zsh ``` ```powershell $env:TESTERY_TOKEN = '' # PowerShell ``` ## What you can do Once installed, just talk to Claude in natural language — it picks the right skill automatically. For example: * "Run my tests on Testery against the staging environment and watch them" * "What test runs are currently active?" * "Show me the results of the last run" * "Register a new `qa` environment" * "Schedule the smoke suite to run nightly at 2am" * "Add a scenario that checks the login page rejects a bad password" ### Slash commands You can also invoke any capability explicitly with a slash command. #### Testery platform | Command | What it does | | ---------------------------------------- | ------------------------------------------------------------------ | | `/testery-onboard` | **Start here.** Sign up / log in, capture your API key, persist it | | `/testery-init` | Scaffold playwright-bdd in this project and wire it to Testery | | `/testery-create-test-run` | Submit a Git-based test run | | `/testery-monitor-test-run` | Follow a run to completion | | `/testery-cancel-test-run` | Cancel a running test run | | `/testery-list-active-test-runs` | Show in-flight runs | | `/testery-report-test-run` | Output per-test results | | `/testery-upload-artifacts` | Upload a local file/dir as a build | | `/testery-add-file` | Attach a file to a test run | | `/testery-register-environment` | Create a new environment | | `/testery-update-environment` | Update an existing environment | | `/testery-deregister-environment` | Delete an environment | | `/testery-list-environments` | List environments | | `/testery-upload-environment-file` | Upload a file to an environment | | `/testery-create-schedule` | Create cron / on-deploy / follow schedules | | `/testery-delete-schedule` | Remove a schedule | | `/testery-create-deploy` | Notify Testery of a deploy | | `/testery-create-alert` | Set up an alert | | `/testery-run-test-plan` | Execute a saved test plan | | `/testery-load-users` | Bulk-load users | | `/testery-verify-token` | Auth health check | | `/testery-run-playwright-bdd-local` | Run playwright-bdd tests on this machine | | `/testery-run-playwright-bdd-on-testery` | Run playwright-bdd tests on Testery (local build or remote Git) | #### Playwright-BDD authoring | Command | What it does | | --------------------- | -------------------------------------------------------------------- | | `/bdd-add-scenario` | Add a scenario, implement step definitions, drive the red→green loop | | `/bdd-test` | Run the playwright-bdd suite (or a single scenario by name) | | `/bdd-implement-code` | Implement app code to make a failing scenario pass | | `/bdd-view-report` | Open the Playwright HTML report in a browser | ## How it works Each slash command delegates to a matching skill, so the documented behavior lives in one place. The `testery-*` skills wrap the [Testery CLI](/integrations/testery-cli-docs) for write operations — creating runs, uploading artifacts, managing schedules, environments, and deploys. The `testery-playwright-bdd-*` skills help you author and run [playwright-bdd](https://github.com/vitalets/playwright-bdd) tests and are modeled on a project shaped like the Testery example web app: `tests/features/`, `tests/steps/`, a `playwright.config.ts` that calls `defineBddConfig`, and a `test:e2e` script that runs `bddgen && playwright test`. ## Troubleshooting **`testery` command not found** * Install the CLI with `pip install testery`, or re-run `/testery-onboard`. **Authentication failures** * Run `/testery-verify-token` to confirm your token is valid. * Re-run `/testery-onboard` to regenerate and persist a new API key. **Commands not appearing** * Confirm the plugin is installed with `/plugin`, then reinstall with `/plugin install testery@testery`. ## Learn more * Plugin source: [github.com/testery/testery-skills](https://github.com/testery/testery-skills) * [Testery CLI documentation](/integrations/testery-cli-docs) * [Testery MCP Server](/integrations/testery-mcp-server) # Supported Testing Frameworks {% hint style="info" %} Your framework not on this list? Email and let us know. We'll put it on the list and let you know when it's ready. {% endhint %} Testery currently supports recent versions of: * Cucumber * Cypress * Nightwatch.js * NUnit * Playwright * PyTest * SpecFlow * TestNG * WebdriverIO # Cypress Testery has a [Cypress.io](http://cypress.io/) test runner so that Cypress tests can be run on the Testery platform. Cypress is a next generation front end testing tool built for the modern web. Here is an [overview](https://docs.cypress.io/guides/overview/why-cypress.html#In-a-nutshell) of Cypress. [Here](https://docs.cypress.io/guides/overview/why-cypress.html#Features) is a list of things Cypress claims it can do that no other testing framework can. Read about [what makes Cypress unique](https://docs.cypress.io/guides/overview/key-differences.html#Architecture). When running Cypress tests in Testery, there are a few things to take note of: * Testery supports screenshots and video recordings * Testery supports Chrome browser only * Due to limits within the Cypress framework, we can only parallelize tests runs at the file level. ### How to Take Screenshots & Videos with Cypress In order to take high-resolution videos and screenshots with Cypress on Testery, you will need to configure some browser behavior in your `cypress.config.js` file. Add the following code: ```javascript module.exports = defineConfig({ e2e: { // Other settings can go here viewportHeight: 1080, viewportWidth: 1920, setupNodeEvents(on, config) { on('before:browser:launch', (browser = {}, launchOptions) => { if (browser.name === 'chrome') { launchOptions.args.push(`--window-size=1920,1080`) // force screen to be non-retina and just use our given resolution launchOptions.args.push('--force-device-scale-factor=1') } return launchOptions }); } }); ``` Videos will be automatically recorded, but screenshots need to be triggered using the `cy.screenshot()` directive in your test code. Cypress' screenshot functionality provides a number of additional options you can explore [here](https://docs.cypress.io/api/commands/screenshot). # Updating Cypress Tests to Output in JUnit XML Format In this article, we'll walk through the steps to update your Cypress tests to output in JUnit XML format and how to upload them to Testery for automated analysis. ### Prerequisites Before we get started, ensure you have the following prerequisites in place: 1. **Cypress Installed**: You should have Cypress installed in your project. If it's not already installed, you can do so by running `npm install cypress --save-dev`. 2. **Node.js**: Make sure you have Node.js installed on your machine. 3. **A Cypress Test Suite**: You should have a Cypress test suite already set up and running. 4. **Testery CLI**. You should have the Testery CLI installed. If you don't have it already, run `pip install testery`. ### Step 1: Install Cypress JUnit Reporter To generate JUnit XML reports, we'll need to install the `cypress-junit-reporter` package. This package provides a custom reporter for Cypress that converts test results into JUnit XML format. Open your project's terminal and run the following command to install the package: ```bash npm install cypress-junit-reporter --save-dev ``` ### Step 2: Update Your Cypress Configuration Now that you have the JUnit reporter installed, you need to update your Cypress configuration to use it. 1. Open your Cypress configuration file, usually named `cypress.json`. 2. Add a `"reporter"` option to specify the reporter you want to use. Set it to `"junit"`: ```json { "reporter": "junit" } ``` 3. Optionally, you can also configure the reporter with additional options. For example, you can specify the output folder for the JUnit XML files: ```json { "reporter": "junit", "reporterOptions": { "mochaFile": "cypress/reports/junit/test-results.xml" } } ``` This configuration will save the JUnit XML reports in a `cypress/reports/junit` folder with filenames that include a hash to ensure uniqueness. ### Step 3: Run Your Cypress Tests Now that your Cypress configuration is updated, you can run your Cypress tests as usual. Cypress will automatically use the JUnit reporter to generate XML reports. To run your tests, use the following command: ```bash npx cypress run ``` After running the tests, you should see JUnit XML files generated in the specified output folder. ### Step 4: Upload Your Test Results to Testery The final step is to upload the generated JUnit XML files into your CI/CD pipeline. The specific steps for doing this will depend on your CI/CD platform. Here's a general outline of what you'll need to do: ```bash testery upload-results \ --environment develop \ --project name \ --file cypress/reports/junit/test-results.xml ``` # Playwright Instructions for getting your Playwright tests to run on Testery [Playwright](https://playwright.dev/) enables reliable end-to-end testing for modern web apps. Here's more information on [Getting Started](https://playwright.dev/docs/intro) with Playwright. Your project requires simple modifications to your playwright config file.
**Parallel** Testery will handle the level of parallelism for the test files, so for best result set this option to **false**. If you would like to keep this **true** for local runs and **false** for Testery, you can add an environment variable check for `IS_TESTERY` and set to `false` when present. ``` fullyParallel: false ``` ``` fullyParallel: process.env.IS_TESTERY ? false : true ``` **Retries** Testery also will handle any retries, so it's best to leave them at 0 when running on the cloud. ``` retries: 0 ``` **Viewport** Add the **viewport** option in your `use` array for each project under the `projects` array and set to `1920x1080`. This will allow better sized video and screenshots to be displayed in Testery. ```json use: { ...devices['Desktop Chrome'], channel: 'chrome', viewport: { width: 1920, height: 1080 }, } ``` #### Screenshots If you want your captured screenshots to display in Testery, you'll need adjust the `outputDir` option to a predefined path: `./screenshots` ``` outputDir: './screenshots' ``` Also make sure your `use` block to contains right value for screenshots to be taken. For example: ``` screenshot: 'only-on-failure' ``` For better screenshot names, it's recommended to use a fixture instead. ```typescript screenshots: [async ({ page }, use, testInfo) => { await use() await page.screenshot({ path: `./screenshots/${testInfo.title}.png` }) }, { auto: true }], ``` **Suggested Testery config example** ```typescript export default defineConfig({ testDir, outputDir: './screenshots', fullyParallel: false, // Testery will handle the parallelism retries: 0, // Testery will handle the retries workers: 4, // Leave this to local running preference. Testery will run one at a time per worker (File/Feature) reporter: [['list']], // Leave this to local running preference. Testery will change this to json report when running on cloud use: { trace: 'retain-on-failure', screenshot: 'only-on-failure' }, timeout: 60_000, // Make sure this is less than test timeout in Testery to get more detailed timeout error in the log projects: [ // recommend using the playwright project field in test runs to separate different project to different runs { name: 'chromium', use: { ...devices['Desktop Chrome'], viewport: { width: 1920, height: 1080 } } } ] }) ``` # Update Playwright Tests to Output in JUnit XML Format for Import Into Testery In this article, we'll guide you through the process of updating your Playwright tests to output results in JUnit XML format, allowing you to seamlessly integrate them into your CI/CD pipeline. Prere ### Prerequisites Before you begin, make sure you have the following prerequisites in place: 1. **Playwright Installed**: Ensure that you have Playwright installed in your project. You can install it using npm: ```bash npm install playwright --save-dev ``` 2. **Node.js**: Make sure you have Node.js installed on your machine. 3. **A Playwright Test Suite**: You should have a Playwright test suite already set up and running. 4. **Testery CLI**: You will need the Testery CLI to upload your JUnit XML files to Testery. If you don't have it installed, you can do so using pip: ```bash pip install testery ``` ### Step 1: Install Playwright JUnit Reporter To generate JUnit XML reports for your Playwright tests, you'll need to install the `playwright-junit-reporter` package. This package provides a custom reporter for Playwright that converts test results into JUnit XML format. Open your project's terminal and run the following command to install the package: ```bash npm install playwright-junit-reporter --save-dev ``` ### Step 2: Update Your Playwright Configuration Now that you have the JUnit reporter installed, you need to update your Playwright configuration to use it. 1. Open your Playwright configuration file, which is usually named `playwright.config.js`. 2. Add a `"reporter"` option to specify the reporter you want to use. Set it to `"junit"`: ```javascript module.exports = { reporter: 'junit', // Other configuration options... }; ``` 3. Optionally, you can configure the reporter with additional options. For instance, you can specify the output folder for the JUnit XML files: ```javascript module.exports = { reporter: 'junit', reporterOptions: { outputFile: 'playwright/reports/junit/test-results.xml', }, // Other configuration options... }; ``` This configuration will save the JUnit XML reports in a `playwright/reports/junit` folder with a filename of `test-results.xml`. ### Step 3: Run Your Playwright Tests With your Playwright configuration updated, you can now run your tests as usual. Playwright will automatically use the JUnit reporter to generate XML reports. To run your tests, use the following command: ```bash npx playwright test ``` After running the tests, you should see JUnit XML files generated in the specified output folder. ### Step 4: Upload Your Test Results to Testery The final step is to upload the generated JUnit XML files to Testery for further analysis. The exact steps for this may vary depending on your CI/CD platform. Here's a general outline of what you'll need to do: ```bash testery upload-results \ --environment develop \ --project name \ --file playwright/reports/junit/test-results.xml ``` Replace `environment` with your specific environment, `project` with your project name, and adjust the file path accordingly. This step ensures that your Playwright test results are accessible and can be used for monitoring and debugging within Testery. ### Conclusion By updating your Playwright tests to output results in JUnit XML format, you streamline the integration of your tests into your CI/CD pipeline. This standardized format enables you to easily monitor test results, track failures, and ensure the reliability of your web applications across different environments. # PyTest Testery supports the [PyTest](https://docs.pytest.org) testing framework. The `pytest` framework makes it easy to write small, readable tests, and can scale to support complex functional testing for applications and libraries. In order to eliminate package conflicts, Testery does not install the `pytest` nor `pytest-selenium` packages by default. If your tests require these packages, make sure to include them in your `requirements.txt`. # Update Pytest to Generate JUnit XML for Import into Testery ### Step 1: Install the pytest-xdist Plugin To generate JUnit XML reports for your Pytest tests, you'll need to install the `pytest-xdist` plugin. This plugin extends Pytest's functionality and provides the JUnit XML report generation feature. Open your project's terminal and run the following command to install the plugin: ```bash pip install pytest-xdist ``` ### Step 2: Update Your Pytest Configuration Now that you have the `pytest-xdist` plugin installed, you need to update your Pytest configuration to use it. 1. Create a Pytest configuration file, if you don't already have one. The configuration file should be named `pytest.ini` or `pyproject.toml` (for newer Pytest versions). 2. Add the following configuration to enable the JUnit XML report generation: For `pytest.ini`: ```ini [pytest] junit_family = xunit2 junit_suite_name = Your_Test_Suite_Name junit_logging = all junit_log_passing_tests = True junit_log_truncate = True ``` Replace `Your_Test_Suite_Name` with an appropriate name for your test suite. For `pyproject.toml`: ```ini [tool.pytest.ini_options] junit_family = 'xunit2' junit_suite_name = 'Your_Test_Suite_Name' junit_logging = 'all' junit_log_passing_tests = true junit_log_truncate = true ``` Again, replace `Your_Test_Suite_Name` as needed. ### Step 3: Run Your Pytest Tests With your Pytest configuration updated, you can now run your tests as usual. Pytest will automatically use the JUnit XML reporter to generate XML reports. To run your tests, use the following command: ```bash pytest ``` After running the tests, you should see JUnit XML files generated in your project directory. ### Step 4: Upload Your Test Results to Testery The final step is to upload the generated JUnit XML files to Testery for further analysis. You can use the Testery CLI to achieve this. Ensure you have the Testery CLI installed: ```bash pip install testery ``` Once installed, you can use the following command to upload your test results to Testery: ```bash testery upload-results \ --environment develop \ --project Your_Project_Name \ --file junit.xml ``` Replace `Your_Project_Name` with your specific project name and adjust the `--file` option to point to the location of your JUnit XML files generated by Pytest. This step will make your Pytest test results accessible within Testery, allowing you to monitor and analyze them efficiently. # TestNG ## Getting started There's a few things you should know when getting started running your tests using the TestNG framework on Testery. Since Java is a compiled language, Testery cannot pull your tests directly from your repository. You will need to upload a JAR file to Testery. ### **If using Gradle..**. To see an example project using our Gradle plugin with TestNG you can look at our [example-testng ](https://github.com/testery/example-testng)repository. To use the plugin in your project you'll first have to add the Testery Gradle plugin to your `build.gradle` file: ``` plugins { id 'io.testery' version '1.3' } ``` Then configure the plugin file in your `build.gradle` file**:** ``` testery { apiToken = System.getenv("TESTERY_API_TOKEN") // required projectKey = "my-testery-project-key" // required buildId = System.getenv("BUILD_ID") // required commitHash = System.getenv("GIT_COMMIT") // optional branch = System.getenv("GIT_BRANCH") // optional } ``` Depending on your CI and environment, you might have to pull your `apiToken` `buildId` `commitHash` and `branch` from different variables. {% hint style="info" %} You'll most likely have to configure an environment variable with your API token. The API token can be found in Testery on the Settings --> Integrations tab. {% endhint %} Once that is configured you can run this command: `./gradlew uploadBuildToTestery` ### **If using Maven...** #### Create a fat JAR file Testery will need a JAR file named `testery.jar` that contains all your test code and any dependencies that the tests depend on. It may be easiest to package up all source code, test code and all dependencies in one fat JAR file. Add the following to your POM.xml file: ``` org.apache.maven.plugins maven-assembly-plugin 3.1.1 jar-with-dependencies src/main/assembly/assembly.xml make-assembly package single ``` You will also need to create this assembly.xml file (replacing the path in your POM.xml file to the correct location in your project): ``` fat-tests jar false / true true test ${project.build.directory}/test-classes / **/*.class true ``` Now run this command: `mvn clean compile test-compile assembly:single` Rename your JAR file to testery.jar before uploading to Testery. #### Upload your testery.jar file to Testery At this point you should have a fat JAR file named testery.jar that contains all your test code and required dependencies. Before uploading your JAR file to Testery make sure you have a project created in Testery to upload the JAR file to. To upload your JAR using Testery CLI run this: `testery upload-build-artifacts --token {api-token} --project-key {project-key} --build-id {build-id} --path {directory_path}/testery.jar` {% hint style="info" %} The API token can be found in Testery on the Settings --> Integrations tab. {% endhint %} For more details on uploading artifacts using the Testery CLI see [upload your JAR file to Testery using the Testery CLI](https://docs.testery.io/testery-integration/testery-cli-docs#upload-build-artifacts). ### Run your tests in Testery Now that you've uploaded your JAR to Testery make sure your project settings are correct. Especially testing framework field (should be TestNG) and the Java package field. Once that those set you can run your tests. Go to the Test Runs tab and select New Test Run. At a minimum, select your project and build along with any other options you want and click Run Test ## Running TestNG tests on Testery There are a few things to note when running your tests on Testery. * Ignored tests will now show up in Testery * Testery does not currently support TestNG suites (although this is coming soon) * Testery handles parallelization of tests so any parallel parameters in your tests will be ignored # Updating TestNG Tests to Output in JUnit XML Format In this article, we'll guide you through the process of updating your TestNG tests to output results in JUnit XML format and then uploading these results to Testery for further analysis and monitoring ### Prerequisites Before you begin, ensure you have the following prerequisites in place: 1. **TestNG Installed**: You should have TestNG installed in your Java project. You can do this by adding the TestNG dependency to your project's build file, such as Maven or Gradle. 2. **Java**: Make sure you have Java installed on your machine. 3. **A TestNG Test Suite**: You should have a TestNG test suite already set up and running. 4. **Testery CLI**: You will need the Testery CLI to upload your JUnit XML files to Testery. If you don't have it installed, you can do so using pip: ```bash pip install testery ``` ### Step 1: Add TestNG JUnit XML Listener To generate JUnit XML reports for your TestNG tests, you'll need to add the `org.testng.reporters.JUnitReportReporter` listener to your TestNG suite. This listener automatically generates JUnit XML reports during test execution. Open your TestNG suite XML file and add the following listener configuration: ```xml xmlCopy code ``` Replace `Your_Test_Suite_Name` with an appropriate name for your test suite and update the class names as needed. ### Step 2: Run Your TestNG Tests With the TestNG suite configured to use the JUnit XML listener, you can run your tests as usual. TestNG will automatically generate JUnit XML reports during test execution. You can run your TestNG suite using your preferred build tool or directly from the command line. For example, using Maven: ```bash mvn clean test ``` After running the tests, you should see JUnit XML files generated in the default output folder. ### Step 3: Upload Your Test Results to Testery The final step is to upload the generated JUnit XML files to Testery for further analysis. You can use the Testery CLI to achieve this. Ensure you have the Testery CLI installed: ```bash pip install testery ``` Once installed, you can use the following command to upload your test results to Testery: ```bash testery upload-test-run \ --environment-key develop \ --project-key Your_Project_Name \ --path target/surefire-reports/junitreports/*.xml ``` Replace `Your_Project_Name` with your specific project name and adjust the `--path` option to point to the location of your JUnit XML files generated by TestNG. This step will make your TestNG test results accessible within Testery, allowing you to monitor and analyze them efficiently. # Enable or Disable Automatic Rerunning of Tests Testery can be configured to automatically retry tests when there are test failures. When the setting is enabled, tests will automatically rerun. If tests fail initially but subsequently pass, Testery will flag them as "flaky". Please keep in mind that automatically rerunning test failures can be considered a bad practice. If the system under test is not producing reliable results, rerunning failed tests can mask this issue (e.g. you have two servers behind a load balancer and one of them is unresponsive, resulting in 50% of traffic getting rejected). That said, it can also be a lot of work to get your tests to be reliable, especially when running at high levels of parallelism. So we provide you the option and promise not to judge you. To change this setting, navigate to **Projects** and click ![](/files/uCsTgo72iPlPD1Qg9pda)next to the Project you want to change the setting for. Select **Edit Settings.** You can then modify the setting to automatically retry tests. ![](/files/NWBAJVKHed6C3ZBXvNre) # Run Cypress Tests without Connecting Repository A step-by-step guide to running Cypress tests on the Testery platform when integration with GitHub / Bitbucket isn't an option. For some organizations, integrating with GitHub or Bitbucket is not an option. While you'll be missing out on some pretty cool functionality, don't fret. You can still benefit from the other great features Testery has to offer. **Create a Project but don't select a repository.** Select `Projects` and then `Add New Project`. Ignore `+ Add Repostory` since in this example we are not integrating with a repository. Be sure to select Cypress for `Testing Framework`. Give it a `Project Key` as a unique identifier for this project. The Project Key will be used to identify artifacts later. ![](/files/-MRLeVKbDE59fT5jTs8Y) **Configure your build server to manually upload the test artifacts.** Instead of having Testery automatically fetch the artifacts from the repository, we will have the build server upload them. * `token` - Your API token. To retrieve your token, click **Settings → Integrations → Show API Token** and copy the token that is displayed. * `project-key` - The key you specified above when creating the project. * `build-id` - The build id from your CI/CD, usually found in an environment variable. This will be used to pull the correct artifacts when starting a test run. * `path` - The folder containing the tests. This path should include the entire cypress folder. You will need to upload everything your root cypress project folder except the node modules. You *can* upload the node modules but Testery will delete them as we resolve your node modules for you. It can also take your upload a LOT longer with the node modules in it. ``` testery upload-build-artifacts --token --project-key --build-id --path --zip-dir ``` **Create a test run.** Now you can test it out by creating a test run the way you normally would. You can do this manually, from the CLI, or from the API. To create a test run manually, select **Test Runs** -> **New Test Run**. Be sure to select the project defined above for `Project With Tests` and select the corresponding `Build` you created in the previous step. ![](/files/-MRLgxlzxb6V14AIN5zI) # How to Store Sensitive Data Like Username and Password For a Cypress Test Let's say you want to test the login for an application. Here's how you'd go about storing username and password in Testery. 1. Navigate to Environments Tab 2. Add a new environment (if you don't have one) 3. Add a variable for your sensitive data like password 4. Enter name of variable (i.e. password) as CYPRESS\_password and its value. For Cypress, you'll be able to access the value using Cypress.env('password') in your code (omitting the CYPRESS\_ prefix) 5. Click on "encrypt" checkbox (screenshot below) 6. Save ![](/files/-MRfmlGqARhLEcuODRuy) # Connect to a Private npm Repository Typically javascript/typescript based tests will include their dependencies in the `package.json`. If these libraries are hosted in publicly available repositories, Testery will just download them and install them prior to running your tests. But sometimes, you may want to have a privately hosted module (e.g. one that contains wrappers for your business objects). To connect to a private npm repository, you will want to include the connection info in an `.npmrc` file or simply by adding your npm token as an environment variable. Click Environments -> Add New Environment (or edit existing). Under Files click Add File. Call the file `npmrc` and add the following content, ``` //registry.npmjs.org/:_authToken={{INSERT YOUR TOKEN HERE}} ``` The contents of this file will be encrypted on Testery. When you run tests, the `npmrc` file will be created prior to running `npm install` and give the test runner access to your private npm repository. # Running Scripts Before the Tests Sometimes in your javascript/typescript projects you would like to run additional steps before the test runner runs. This could be used for setting up dependencies, transpiling code, or various other things to help your tests run more smoothly. In order to support this, Testery provides the ability to define a section of your package.json that will run prior to the tests. To do this, locate the `scripts` section of your `package.json` and add a string parameter for `testeryPreRunScript`. Say you wanted to install the `vim` system package on all of your runners. Your script would look something like this: `"testeryPreRunScript": "apt-get install -y vim".` ``` { "scripts": "testeryPreRunScript":"{{YOUR SCRIPTS GO HERE}}", ... } ``` # Upload Test Run for Analysis If you'd like to upload a test run for reporting purposes, to generate an AI-driven analysis, or to kick off alerts. You can do so either from the UI or from the CLI. You will want to have a copy of your test run output in JUnit XML format. To generate this for your framework, see our various Framework Specific Guidance. * [Updating TestNG Tests to Output in JUnit XML Format](/framework-specific-guidance/testng/updating-testng-tests-to-output-in-junit-xml-format) * [Updating Cypress Tests to Output in JUnit XML Format](/framework-specific-guidance/cypress/updating-cypress-tests-to-output-in-junit-xml-format) ## Upload Test Run (UI Option) 1. Navigate to [https://testery.app/testery/test-runs](https://testery.app/tendersnyder/test-runs). 2. Click `Upload Test Run`. 3. Select your Project and Environment and then click Choose File to upload your JUnit XML.
## Upload a Test Run (CLI Option) Ensure you have the Testery CLI installed: ```bash pip install testery ``` Once installed, you can use the following command to upload your test results to Testery: ```bash testery upload-test-run\ --environment-key develop \ --project-key Your_Project_Name \ --path target/surefire-reports/junitreports/junit.xml ``` Replace `Your_Project_Name` with your specific project name and adjust the `--path`option to point to the location of your JUnit XML file generated by TestNG. If your junit.xml file is large, you should consider compressing it using .gz format # Troubleshooting Steps ## What To Do When Tests Pass Locally But Fail In Testery * **Ensure you're pointing at the same system-under-test when running locally and try again.** Sometimes a test will pass locally because when running locally it's pointed at a web server on a local host or a development environment, and then when running remotely it's pointed at a QA environment. Environment differences can cause test failures (and could even be bugs)! * **Make sure your timeouts are set appropriately**. All tests that interact with websites have timeouts (even Cypress). When Testery runs a test, you are getting a clean test runner that doesn't have the same items in its browser cache, the same network latency, or likely the same CPU and memory profile as your beefed-up developer workstation. While test run times are generally significantly better than running locally, this is due to parallelization, as individual tests can take longer to run. ## What to do when the client’s time zone is causing errors in your test If you have been developing and running test code locally and then start using a runner that is in a different time zone, you may come across unexpected errors. Here are a few things to try to address these issues. * **Ensure tests are time-independent**. Most modern code languages have time/date libraries to convert between time zones easily. It is advisable to standardize the time retrieved from both the application's interface and the runner by converting it to a common time zone. ``` from datetime import datetime import pytz # Get the current UTC time (Testery) utc_now = datetime.now(pytz.UTC) # Convert to Pacific Time pacific_timezone = pytz.timezone('America/Los_Angeles') pacific_time = utc_now.astimezone(pacific_timezone) # Format the time as a string formatted_time = pacific_time.strftime("%Y-%m-%d %H:%M:%S %Z") print(f"Current Pacific Time: {formatted_time}") ``` * **Modify the browser’s time zone**. If your application is limited to operations in certain time zones, changing the browser’s time zone on the test runner may yield beneficial results. You can use browser options when initializing the driver to achieve this. ```python driver = webdriver.Chrome() tz_params = {'timezoneId': 'America/Los_Angeles'} driver.execute_cdp_cmd('Emulation.setTimezoneOverride', tz_params) ``` # Fix Common Selenium Exceptions ### Common Selenium WebDriver Exceptions Selenium WebDriver is a powerful tool for automating web tests, but it's essential to be aware of the common exceptions that can occur during test execution. Understanding these exceptions and knowing how to handle them is crucial for building robust and reliable test scripts. Here's a summary of the most common Selenium WebDriver exceptions: #### 1. NoSuchElementException **Description:** This exception occurs when WebDriver cannot locate an HTML element using the provided selector. It typically happens due to incorrect selectors, timing issues, or changes in the web page's structure. **Handling:** Use explicit waits, verify element existence, and update selectors as needed to address this exception. #### 2. TimeoutException **Description:** TimeoutException happens when an operation or condition specified in your test script cannot be fulfilled within the specified time limit. Common causes include waiting for elements to load or become interactable. **Handling:** Adjust timeout values, use explicit waits with proper conditions, and implement retry mechanisms. #### 3. ElementNotVisibleException **Description:** This exception occurs when you attempt to interact with an element on a web page that is present in the DOM but not visible or interactable to the user. **Handling:** Ensure the element is supposed to be visible, use explicit waits for visibility, and simulate user actions if needed. #### 4. StaleElementReferenceException **Description:** StaleElementReferenceException happens when an element that was previously located and stored in a variable becomes "stale" or detached from the DOM, often due to page refresh or changes. **Handling:** Re-locate the element, use explicit waits, and implement try-catch blocks to manage this exception. #### 5. WebDriverException **Description:** WebDriverException is a general exception that occurs for various reasons, including browser crashes, incorrect WebDriver configuration, network issues, or unsupported actions. **Handling:** Ensure proper WebDriver setup, use error-handling techniques, and monitor system resources. #### 6. UnhandledAlertException **Description:** This exception arises when Selenium encounters an unexpected JavaScript alert, confirmation dialog, or prompt dialog that it doesn't know how to handle. **Handling:** Implement code to handle alerts and pop-ups explicitly when they occur. #### 7. NoSuchWindowException **Description:** NoSuchWindowException happens when attempting to interact with a browser window or tab that no longer exists or cannot be found, often due to timing or incorrect window handles. **Handling:** Keep track of window handles, use explicit waits for window operations, and handle window switch exceptions gracefully. #### 8. InvalidSelectorException **Description:** This exception occurs when you provide an invalid or malformed selector while attempting to locate an element on a web page. **Handling:** Double-check selector syntax, use browser developer tools, and validate selectors to avoid this exception. #### 9. InvalidSelectorException **Description:** The InvalidSelectorException is raised when you provide an invalid or malformed selector (e.g., CSS selector or XPath) while trying to locate an element on a web page. **Handling:** Verify selector syntax, use developer tools to inspect elements, and implement proper error handling. #### 10. InvalidSelectorException **Description:** The InvalidSelectorException is raised when you provide an invalid or malformed selector (e.g., CSS selector or XPath) while trying to locate an element on a web page. **Handling:** Double-check selector syntax, use browser developer tools, and validate selectors to avoid this exception. Understanding and addressing these common exceptions will help you create more stable and reliable Selenium WebDriver automation scripts. Proper error handling, synchronization, and selector validation are key to successful automation testing. # Fix a ChromeDriver Version Exception A ChromeDriver version exception or error occurs when there is a mismatch between the version of the ChromeDriver executable and the version of the Google Chrome browser that you are using. ChromeDriver is a separate executable that facilitates communication between Selenium WebDriver and the Chrome browser. Here are some common reasons why a ChromeDriver version exception might happen: 1. **Incompatible Versions:** The most common reason for this exception is using an incompatible version of ChromeDriver for the installed version of Google Chrome. ChromeDriver is tightly coupled with the Chrome browser, and different versions of Chrome require specific versions of ChromeDriver for compatibility. 2. **Outdated ChromeDriver:** If you've updated your Chrome browser to a newer version, you may also need to update ChromeDriver to a compatible version. Failing to do so can result in a version mismatch and an exception. 3. **Multiple Chrome Versions:** Sometimes, users have multiple versions of the Chrome browser installed on their systems. If the WebDriver attempts to use the wrong version of ChromeDriver for the currently active Chrome instance, it can lead to this exception. 4. **Incorrect WebDriver Initialization:** If you're initializing the WebDriver with an incorrect path to the ChromeDriver executable, it can cause version issues. Ensure that you're specifying the correct path when setting up the WebDriver. To resolve a ChromeDriver version exception: 1. **Update ChromeDriver:** Download and use the latest compatible version of ChromeDriver that matches your installed version of Google Chrome. You can find the latest ChromeDriver releases on the official Chromium website or on the Selenium WebDriver GitHub repository. 2. **Check Chrome Version:** Verify the version of Google Chrome you have installed and ensure it matches the ChromeDriver version you are using. You can find your Chrome version by navigating to "chrome://settings/help" in the browser. 3. **Single Chrome Installation:** If you have multiple Chrome versions installed, try to ensure that your WebDriver uses the appropriate version by specifying the correct path to the Chrome executable. 4. **Automation Tools:** Consider using automation tools like WebDriverManager or WebDriverManager-Py, which can automate the process of downloading and managing the appropriate ChromeDriver version based on your browser installation. By keeping Chrome and ChromeDriver versions in sync and ensuring that you are using a compatible combination, you can prevent ChromeDriver version exceptions and ensure the smooth operation of your Selenium automation tests. # Fix a NoSuchElement Exception A `NoSuchElementException` in Selenium occurs when the WebDriver (Selenium's core component responsible for interacting with web pages) is unable to locate an HTML element on a web page using the provided selector. Several reasons can lead to this exception: 1. **Incorrect Selector:** The most common reason is that the selector used to locate the element is incorrect or does not match any element on the page. This can happen if there's a typographical error in the selector, or if the element's attributes have changed since the test script was last updated. 2. **Timing Issues:** Web pages often have dynamic content that loads asynchronously using JavaScript or AJAX. If Selenium tries to locate an element before it has appeared in the DOM, a `NoSuchElementException` can occur. Properly implementing waits using `WebDriverWait` can help mitigate this issue. 3. **Element Not Present:** Sometimes, the element might not exist on the page at all, either due to a page layout change, conditional rendering, or a mistake in the test script's logic. 4. **Frame or iFrame Context:** If the element you are trying to access is within an HTML `