Testery CLI
The Testery CLI is available as a pip package here: https://pypi.org/project/testery/.
To install, run
pip install --upgrade testery
If pip install errors you might need to run pip3 install depending on which versions of python you have on your machine.
You can see the command line help by running,
> testery --help
Usage: testery [OPTIONS] COMMAND [ARGS]...
Testery CLI
Kick off test runs from your CI/CD platform and run them on Testery's
next-generation, cloud-based testing grid.
Options:
--help Show this message and exit.
Commands:
add-file Adds a file to a test run.
cancel-test-run Cancels a test run.
create-deploy Creates a deploy for a project and...
create-environment Creates an environment where tests can be...
create-test-run Submits a Git-based test run to the Testery...
delete-environment Deletes an environment.
list-environments Returns a list of environments.
load-users
monitor-test-run
monitor-test-runs
report-test-run Outputs individual test results for the...
update-environment Updates an environment where tests can be...
upload-build-artifacts Uploads a file or directory of build...
verify-token Verifies your username and authentication...
Create a Deployment
Creating a deployment is useful for telling Testery when you've pushed new code into your test environment. What tests you want to run can be configured with Schedules.
> testery create-deploy --help
Usage: testery create-deploy [OPTIONS]
Creates a deploy for a project and environment.
Options:
--token TEXT Your Testery API token. [required]
--project TEXT The project key of the repo being deployed. [required]
--git-provider TEXT The Git provider used for the repository that is being
deployed. Should be GitHub or BitBucket
--git-owner TEXT The organization owner in Git for the repository that
is being deployed.
--git-repo TEXT The repository name that is being deployed.
--build-id TEXT The build the atifact should be associated with.
--environment TEXT Which environment you would like to run your tests
against. [required]
--commit TEXT The Git commit that was deployed.
--branch TEXT The Git branch the deploy came from.
--help Show this message and exit.
Create an Environment
Usage: testery create-environment [OPTIONS]
Creates an environment where tests can be run.
Options:
--token TEXT Your Testery API token. [required]
--name TEXT The display name for the environment.
[required]
--key TEXT An identifier for the environment. This is
used when creating test runs to indicate
where tests should run. [required]
--pipeline-stage TEXT The name of a pipeline stage to associate
this environment to.
--variable TEXT A variable to add to the environment.
Specified as "KEY=VALUE". To encrypt value,
pass in "secure:KEY=VALUE", Multiple
variables can be provided.
--maximum-parallel-test-runs INTEGER
The maximum number of test runs that can run
in parallel on this environment.
--help Show this message and exit.
Create a Test Run
Usage: testery create-test-run [OPTIONS]
Submits a Git-based test run to the Testery platform.
Options:
--token TEXT Your Testery API token. [required]
--git-ref TEXT The git commit hash of the build being
tested.
--git-branch TEXT The git branch whose latest commit you want
to run.
--test-name TEXT The name you want to use on the Git status.
--wait-for-results If set, the command will poll until the test
run is complete.
--project TEXT Legacy option. Use --project-key instead.
--project-key TEXT The project key.
--test-suite TEXT Name of Test Suite to use for this test run.
--environment TEXT Legacy option. Use --environment-key
instead.
--environment-key TEXT Which environment you would like to run your
tests against.
--include-tags TEXT List of tags that should be run.
--exclude-tags TEXT List of tags that should excluded from the
test run.
--copies INTEGER The number of copies of the tests to submit.
--build-id TEXT A unique identifier that identifies this
build in your system.
--output TEXT The format for outputting results
[json,pretty,teamcity]
--fail-on-failure When set, the testery command will return
exit code 1 if there are test failures.
--include-all-tags When set, overrides the testery.yml and runs
all available tags.
--parallelize-by-file Pass this flag if you want the test run to
parallelize by file/feature.
--parallelize-by-test Pass this flag if you want the test run to
parallelize by test/scenario.
--timeout-minutes INTEGER The maximum number of minutes this test run
can take before it is killed automatically.
--test-timeout-seconds INTEGER The maximum number of seconds a test can
take before it is killed automatically.
--runner-count INTEGER Specify number of parallel runners to use in
for this testrun.
--variable TEXT A variable to add to the enviroment.
Specified as "KEY=VALUE". To encrypt value,
pass in "secure:KEY=VALUE", Multiple
variables can be provided.
--test-filter-regex TEXT A regular expression to be used for
filtering tests.
--help Show this message and exit.
Upload Build Artifacts
If your project is using a compiled language (like .Net, Java) or if you aren't attaching a code repository to your project, then you'll have to upload your build artifacts to Testery before you can run a test. To do that you can use the upload-build-artifacts command of the Testery CLI.
> testery upload-build-artifacts --help
Usage: testery upload-build-artifacts [OPTIONS]
Uploads a file or directory of build artifacts and associates them with
the specified build-id
Options:
--token TEXT Your Testery API token. [required]
--project TEXT Legacy option. Use --project-key instead.
--project-key TEXT The project key.
--branch TEXT The Git branch the build came from.
--build-id TEXT The build the artifact should be associated with.
[required]
--path TEXT The path to the file or directory you want to upload.
[required]
--zip-dir Creates a zip file of the directory contents before
uploading.
--help Show this message and exit.
Report Test Run Results
The report-test-run command waits for a test run to finish and writes its individual test results to a file your CI system can publish as a test report. Use --output junit for JUnit XML (Azure DevOps PublishTestResults@2, Jenkins, GitLab) or the default sonarcube for the SonarQube generic test execution format.
> testery report-test-run --help
Usage: testery report-test-run [OPTIONS]
Outputs individual test results for the entire run to the specified
destination in the specified format.
Options:
--token TEXT Your Testery API token. Falls back to
~/.testery/credentials or $TESTERY_API_TOKEN.
--profile TEXT Profile name in ~/.testery/credentials to use
for auth.
--test-run-id TEXT The ID for the test run you would like to
monitor and wait for. [required]
--fail-on-failure When set, the testery command will return exit
code 1 if there are test failures.
--output [sonarcube|junit] The format for outputting results. "sonarcube"
writes the SonarQube generic test execution
format; "junit" writes JUnit XML that CI systems
such as Azure DevOps (PublishTestResults@2),
Jenkins and GitLab can publish as a test report.
--outfile TEXT The filename to write the results to.
--help Show this message and exit.
--output junit requires Testery CLI 1.20.0 or later. Version 1.19.0 and earlier accept the option but quietly write SonarQube XML instead. Install with pip install --upgrade "testery>=1.20.0".
With --fail-on-failure, the results file is written before the command exits with code 1, so a CI step that publishes it afterwards (with a condition such as always()) still has the file.
Example:
testery report-test-run --token $TESTERY_TOKEN --test-run-id 12345 --output junit --outfile testery-junit.xml --fail-on-failure
The JUnit file has one <testsuite> per test file, a <testcase> per test with its duration in seconds, and <failure>, <skipped> or <error> elements for tests that did not pass. See Azure DevOps Pipelines for a pipeline that publishes it.
Load Users
Use the load-users command to bulk-invite users into your Testery account. Each user is added with the USER role. This is useful when onboarding a team or organization.
> testery load-users --help
Usage: testery load-users [OPTIONS]
Options:
--token TEXT Your Testery API token. Falls back to
~/.testery/credentials or $TESTERY_API_TOKEN.
--profile TEXT Profile name in ~/.testery/credentials to use for auth.
--user-file TEXT List of email addresses to load as user accounts.
--help Show this message and exit.
The --user-file should point to a plain text file containing one email address per line:
alice@example.com
bob@example.com
carol@example.com
For example:
testery load-users --user-file users.txt