Release Notes

2023-05-15

• Fixed a bug causing test runs to errantly get marked as failed in a certain Cypress case.

• Fixed a WDIO bug where tests could get marked as passed but have failed steps. This should no longer happen.

2023-05-11

• Added new run-test-plan command to the CLI with the option to override the environment.

• Added key field to Test Plans which can also be used via the latest version of the CLI.

• Fixed the bug causing an error using the cancel-test-run command on the CLI.

• Fixed a bug causing screenshots failing to get saved when a test times out.

• Fixed a bug with WDIO test runs that started with the NodeJS 16 upgrade.

2023-05-01

• Upgraded NodeJS from v14 to v16 on both Linux and Windows runners.

• Fixed form validation error when saving Test Triggers in certain cases.

2023-04-20

• Fixed a bug causing Cypress screenshots to not show up in certain cases.

2023-04-17

• Text overflow in dropdowns should now wrap to the next line.

• Fixed date filtering on the Test Runs tab.

• Added date filtering to the URL, for easy bookmarking and sharing, to the Test Runs tab.

• Improved sorting on various dropdown menus.

• Fixed form error for Test Plans.

2023-04-14

• Added the ability to "bookmark" filters on the Test Runs tab. Filters will be saved in the URL which will be used to auto-set these filters again when returning to that URL.

• Performance enhancements.

• Better error handling.

2023-04-05

• Fixed an issue causing some test results to not be reported correctly in certain cases.

• Fixed the search feature several dropdown menus...

  • Environment Dropdowns

  • Pipeline Stage Dropdowns

  • Project Dropdowns (both single selection and multiple selection)

  • Test Plan Dropdowns

  • Test Suite Dropdowns

2023-03-27

• Fixed Cypress issue causing some test run issues.

2023-03-09

• Upgraded Windows runners to Chrome v111.

2023-02-07

• Continued improvements for test runs getting stuck in SUBMITTED and RUNNING statuses.

2023-01-30

• Test runs getting stuck in a SUBMITTED state improvement.

• Implement ability to use AWS Private Link on Windows runners.

2023-01-27

• Fixed logging issue causing some test names to be longer than they should be.

2023-01-19

• Added support for yarn install arguments to be added through Testery environments.

2023-01-12

• Updated Chrome version from 107 to 109 on Windows runners.

2022-12-20

• Fixed race condition in test run status transitions.

2022-12-14

• Fixed test trigger form validation issues.

• Test run status improvements.

2022-12-13

• Fixed the test run reviewer assignment dropdown that wasn't showing the proper user list.

2022-12-06

• Test run status improvements.

2022-11-30

• Fixed incorrect test run counts after running tests.

• Test run status improvements.

2022-11-21

• Microsoft Teams Integration.

• Increased allotted memory on Windows runners.

2022-11-07

• Fixed a bug that could cause a Cypress 10 configuration file parse error in a certain case.

2022-11-05

• Fixed bug causing some test runs to get stuck and not complete.

2022-11-01

• New Feature: Two new tags/hooks have been added to Testery to allow users to run code as setup/teardown steps. Just annotate your code block with @BeforeTesteryRun or @AfterTesteryRun and Testery will run that code before/after the test run. This code will only run once before/after the run, not once on each runner.

• New Feature: Users can now create teams within Testery.

• Upgraded Linux runners to Chrome v107.

2022-10-21

• Fixed a bug that could cause CucumberJS tests to fail in some cases.

2022-10-10

• Fixed a bug preventing test results from showing for some Cypress 10 test runs.

2022-10-06

• Fixed Cypress 10 Config parse issue.

2022-09-26

• Added the ability for users to run scripts before a test run on each runner. Simply add a script named testeryPreRunScript to your package.json and the Testery app will run that script before the test run on each runner.

• Fixed a bug causing flay tests to show in the list of failed tests in Slack notifications.

2022-09-16

• Fixed a bug that could cause an error when parsing a Cypress 10 config file.

2022-08-29

• New feature! Users running PyTest tests can now choose the Python language version in the project's settings.

2022-08-12

• Features

  • Users can now assign tests for review.

  • Users can now set a review status for tests.

  • Added support for Cypress 10 (see Project Settings in the app to select framework version).

  • Testery CLI will now print out it's version number on each command run.

• Bugfixes/Improvements

  • Turned off emails to admin users when Slack channel doesn't exist.

  • Fixed bug that did not take the Project's Parallel Strategy into account when manually starting test runs by Test Suite.

  • Fixed a bug that prevented the Analytics tab from loading in certain timezones.

  • Fixed bug preventing screenshots and videos from working on the Cypress framework.

2022-07-27

• Upgraded Window and Linux runners to use Chrome v103.

2022-07-22

• Fixed side-scrolling issue on Test Suites tab.

• Fixed bug preventing team members from showing up in Settings.

• On the Test Runs tab, Suite will be on one line instead of two.

2022-07-11

• Fixed an issue on the Test Runs tab where selecting Latest Deploy Version may not run the proper changes in some cases.

2022-07-06

• Fixed UI issue on the Test Runs tab where the "Suite" field had 2 lines.

• Fixed bug preventing users from seeing team members in their user Settings.

2022-06-29

• Fixed a bug causing some tests to fail when a large number of tests were present.

2022-06-28

• Implemented feature improvements...

  • Users can now specify an alternate location for their Cypress config file.

  • Implemented several fixes and improvements on the Test Runs tab.

• Fixed bug causing some test runs to get stuck in a SUBMITTED state.

2022-06-14

• Fixed a scrolling bug in the UI in the account selection dropdown menu.

2022-06-07

• Fixed a bug preventing some Slack notifications from being sent.

2022-06-06

• Added "Triggered By" label back to the Test Runs tab.

• Fixed an issue with deploy creation using previously used references.

2022-05-31

• Fixed NUnit runner issues.

2022-05-24

• Upgraded Linux runners to .NET Core 6.0.

• Added email notification for failed Slack notifications.

• Added better handling of Environments when they get too large.

• Add Slash Commands to Slack (users may need to re-install the Testery App to start using this).

2022-05-02

• Fixed UI bug where the Test Run list table header was showing through the datepicker calendar.

2022-04-27

• Fixed UI bug when adding Environment files.

2022-04-20

• Increased the maximum Test Timeout (seconds) from 1 hour to 3 hours.

2022-04-10

• Fixed bugs preventing Slack notifications from being sent.

2022-03-29

• Fixed a couple bugs preventing Slack notifications from being sent.

2022-03-24

• Released new feature that allows test runs to be assigned to a user. The user will be notified via Slack (if Slack is connected).

• The Test Runs tab is now mobile friendly.

• Fixed a bug causing the wrong Chrome version to be used on Windows runners.

2022-03-22

• Fixed a bug preventing users from starting test runs.

2022-03-10/11

• Fixed a bug causing user defined Chrome options to be discarded

2022-02-23

• Added video recording of tests on Linux runners (beta)

• Reworked UI when viewing a testrun's test results

2022-02-02

• Removed flaky tests from the failed test list in Slack messages

2022-01-06

• Environment Variable fixes and UI Improvements

  • Several fixes surrounding saving Environment Variables and Files

  • UI Improvement: Environments Page

    • New way to list Environment Variables/Files

    • More mobile friendly

    • Hover to see variable values.

  • UI Improvement: New/Edit Environment Page

    • New lightweight code/text editor when adding files.

    • Consolidated View

    • Better guarding around blank fields

2021-12-13

• Now supporting TypeScript for WebDriverIO tests

• Increased Description character limit in Projects

2021-12-09

• Improved ES6 support in Cypress tests

• Improved logging

2021-11-23

• Improved Slack Messages

  • Added interactivity features for re-running all the tests in that run or just the failed ones

  • Added a graph detailing passed, failed and ignored tests

  • Added the commit message, the committer and time of commit that belongs to the test ref

  • Now showing a list of failed tests

  • Now showing the list of test filters

2021-11-16

• Fixed logging issue

2021-02-12

• You can now filter your test runs on the Test Run tab by flaky test.

• Test run analytics. See a historical view of your test runs and gain insight into how your tests are performing. Filter by time period, environment, project and build.

• Flaky tests analytics. See a historical view of your flaky tests and gain insight into the stability of your tests. Filter by time period, environment, project and build.

Test run analytics

2021-01-29

• Changed UI to show all tests (previously only show last test) in a test run when a test or test run is re-run.

• Performance improvement: cached calls to reduce page load time

• Added project setting for number of default runners. All tests runs for a given projects will default to this unless overidden at test run time.

• Added a new alert for 'When test run completes and has flaky test'. This can be setup in Alerts

• Added ability to collaborate and leave messages for your team on a test run.

• Added ability to follow a test run. Once you follow, you can receive slack or email notifications when a test run moves from queued to submitted status, when a test run completes and when a new message is posted to test run.

2020-12-18

• Better align screenshots with specific tests. Added option in Project settings that you can turn on which will print the test name to the test output and Testery will match that against the test.

• Added npm to the default image that test runners use

• Performance improvement: changed UI to only load updated test results on test results page

2020-12-04

• Test run timeline view! Checkout the test run timeline view to see how each test executed. Easily identify any patterns that may be causing issues in your tests or environments. ( see below image)

• Added filters to Schedules tab making it easier to find what you need. Filter by project or environment.

• Bug fix: we fixed issue running latest version of chrome on Windows servers.

• Added support for running Lighthouse

2020-11-20

• Flaky tests! Testery will now identify when a test is flaky by adding a tag so you can take action. Flaky tests are a nuisance and can cause the team to not trust the results. Testery now helps you make it easy to identify a test that failed once but passed the second time. (see image below)

• Added ability to re-run failed tests. There is a setting on a test run now that you can turn on and when tests fail the failed tests will automatically re-run. This helps account for flaky tests and saves your team time.

• Added ability to create a Testery account using email (instead of just Github and Bitbucket)

• Added a new user invite. When you add a team member they will now receive an invite making the process much more smooth getting into Testery.

• Display commit for project that triggered the test. If a test run is triggered by another project deploying and your test code is in a different project, you will now see both projects commit info.

2020-11-06

• Added test run execution priority. Give a pipeline stage a priority and all test runs within the stage will default to that priority. You can override this at a test run level. Now, more important environments / tests can run before others that are less urgent.

2020-10-23

• Added commit info for each card on the Environment Dashboard. You can now easily see what code was in the environment at the time the test ran and who committed.

• Added ability to filter test results by test status.

• Added a not run category in test results so you can see if a test didn't run for some reason.

• create-deploy from the CLI now supports the same environment, project and commit but with different build ids.

2020-10-09

• Added pipeline stages. Pipeline stages represent stages in your development pipeline (dev, test, qa, prod, etc). Each environment can be added to a pipeline stage allowing you to group environments into stages. You can view your projects and test runs by pipeline stage.

• We now have an Environment Dashboard!!! See a birds eye view of all your test runs across environments. If you have setup pipeline stages the stages will display in columns with each test run /environment combo in the column. When you click on a test run on the dashboard it will take you into the test run to see more details.

• Added a link to your environment. Edit your environment, add the url to your environment and team members can easily access the environment from within test runs.

• We made test_run_id available as an environment variable.

• Added filters to more easily navigate the environments page when you have a lot of environments.

• Added ability to change your password in Settings.

• Bug fix: fixed issue when manually running a test run when using "latest deployed version".

2020-09-25

• Add tags for Cypress.

• Added ability to run Cypress tests parallelized by test (you can't do this in Cypress Dashboard at this time!)

• Added ability to give scheduled test run a name

2020-09-11

• Updated configuration to support webdriver.io file being stored in any location.

• For Cypress.io, it's a common issue with the framework where chrome will fail to load. We added extra support for this scenario and will automatically re-run the test if chrome fails to load.

• Added a "Run Now" button to Scheduled test runs. If you click the run now button the scheduled test run will run immediately.

• UI Design: Enhanced how tests display within a file/feature so they are grouped better.

• Updated Slack alert to send to notify for all environments and not just a single one.

• Bug fix: Fixed issue where a test here or there would get run twice and count as 2 tests in results.

2020-08-28

• Scheduled test runs is live! You can now schedule a test run to run regularly at a certain day/time or when a deploy happens. (If running a test when a deploy happens you have to send us your deployment information.)

• Display branch name to the Source Info box on a test run.

• Added support for Bitbucket Server.

• Added ability to manually upload test artifacts. A project can now be created with no artifacts associated.

• Bug fix: When parallelized by file and re-running failed tests, only re-run the tests in the file that failed.

• Bug fix: enhanced how screenshots are associated to a test run

2020-08-14

• Bug fix: Screenshots from failed test are also getting attached to next test run at times

2020-07-31

• Test runs page got a redesign! You can now view project name, include/exclude filters, regex, test passing % without having to click into results.

• Better filters

• Allow multiple test runs to be queued up that have different regex filters (given all other fields are the same)

• Fixed the rerun failed tests action to account for parallelize strategy when parallelizing by file/feature.

2020-07-17

• Added ability to configure/override timeout for test runs. You can do now set a default timeout for a single test and the test run at the project level and override at the test run level.

• Added ability to parallelize by file for the Nightwatch.js runner

• For NUnit tests, allow user to specify to parallelize tests by file or test. When running tests by feature separate out test results by individual test for easier viewing and report errors appropriately.

2020-07-03

• Allow user to open test run results in a new window

• Allow user to copy the test result name

• Display test output while test is running

• Do not clear regex box when the field loses focus