Test Plans
Test plans in Testery allow you to create a plan where you can lay out the stages of testing, specify what tests or test suites will be in each stage, what environments the test should run in, etc.
Create a Test Plan
Click on Test Plans and click Add New Test Plan button. Give your test plan a name and objective so others on your team understand what the goal of the test plan.
Test Plan Stages
Defining stages allows you to control the order groups of tests will execute in and how they execute (parallel or sequential). Control the flow if certain tests pass or fail. Test plan stages gives users a test execution super power 🦸⚡
On the Stages tab within a test plan create the Add New Stage button. Enter the following fields:
Stage Name - Give this group of tests a logical name. (Required)
Description - Describe what this group of tests will be testing. (Optional)
Execution Type - Parallel or Sequential. This tells the runner to run the tests within this stage in parallel or sequentially. (Required)
Always Run This Stage - Tells the runner whether this stage should always run no matter if tests within a previous stage pass or fail. For instance, if you have a cleanup tests stage you may want to set that stage to always run to ensure your data is cleaned up. (Required)
Once stages are added they can be ordered by dragging and dropping using the menu to the left of the name.
Add Test Suites To Stage
Once stages have been added, you can add test suites to each stage. The number of suites you can add is unlimited. You can create a new test suite or select an existing one to add to your stage. Select Add Test Suite button in your stage to add a test suite.
We will be adding the ability to add tests without having a test suite. This will be in a future release.
Test suites, once added, can be ordered by dragging and dropping within a stage. This ordering only matters if the stage has a test execution of sequential or if you have more test suites within your stage than number of concurrent runners in your account. Then specify the following:
Environment - Select the environment you want the test suite to run in. You can select different environments for different test suites if needed.
You can also use Dynamic Environment Expressions to determine the environment at runtime. By selecting the "Use Dynamic Environment" toggle, you have the option to enter an expression using variables that are supplied via the CLI. For example, if you provide the environment
${my_variable_name}-qain the test plan, when you callrun-test-planfrom the Testery CLI, you will need to provide an argument--variable="my_variable_name=environment1"In this scenario, the test plan will run against the environment namedenvironment1-qa. The environment will not be created for you, you are responsible for creating that environment before the test run, either using the testery app or CLI.
Project - Select the project that contains the existing test suite or the project where you want to create the test suite (depending on if you are selecting an existing suite or creating a new one)
Version - Select the version of test code you want to run in your test suite. You can choose Latest Deployed Version (if you have sent deployment info to Testery) or Specific Version of your test code specifying the branch and commit.
Test Suite - If you have an existing test suite you want to add to the stage select Select Test Suite and choose your test suite from the dropdown. If you want to create a new test suite select Create Test Suite and select your options for the newly created test suite.
Once the test suites are added, you can order them by dragging and dropping.
Add test suites from different testing frameworks within the same stage. This supports running tests across teams that may use different tools/languages.
Run Test Plan
Once your test plan is created with the appropriate stages and test suites...it's time to run your test plan! You can run your test plan from the test plan list selecting the Run Test Plan option from the action menu or click the Run Test Plan button on the Stages tab. Once a test plan is started you will be taken to the Runs tab of the test plan so you can see the progress as it completes.
Results
Test plan run results will show:
Total number of tests that passed and failed at the test plan level
Duration of test plan
Number of tests that passed and failed in each test suite
If a suite was skipped (if no tests were run in suite)
If a test suite had an error (along with error message)
Duration of each test suite (duration of each test suite will not total duration of test plan due to parallelization)
Information: Environment, Branch, Git Ref of test code that was run
Last updated
