Testery CLI

The Testery CLI is available as a pip package here: https://pypi.org/project/testery/.

To install, run

pip install --upgrade testery

If pip install errors you might need to run pip3 install depending on which versions of python you have on your machine.

You can see the command line help by running,

> testery --help
Usage: testery [OPTIONS] COMMAND [ARGS]...

  Testery CLI

  Kick off test runs from your CI/CD platform and run them on Testery's
  next-generation, cloud-based testing grid.

Options:
  --help  Show this message and exit.

Commands:
  add-file                Adds a file to a test run.
  cancel-test-run         Cancels a test run.
  create-deploy           Creates a deploy for a project and...
  create-environment      Creates an environment where tests can be...
  create-test-run         Submits a Git-based test run to the Testery...
  delete-environment      Deletes an environment.
  list-environments       Returns a list of environments.
  load-users
  monitor-test-run
  monitor-test-runs
  report-test-run         Outputs individual test results for the...
  update-environment      Updates an environment where tests can be...
  upload-build-artifacts  Uploads a file or directory of build...
  verify-token            Verifies your username and authentication...

Create a Deployment

Creating a deployment is useful for telling Testery when you've pushed new code into your test environment. What tests you want to run can be configured with Schedules.

> testery create-deploy --help
Usage: testery create-deploy [OPTIONS]

  Creates a deploy for a project and environment.

Options:
  --token TEXT         Your Testery API token.  [required]
  --project TEXT       The project key of the repo being deployed.  [required]
  --git-provider TEXT  The Git provider used for the repository that is being
                       deployed. Should be GitHub or BitBucket
  --git-owner TEXT     The organization owner in Git for the repository that
                       is being deployed.
  --git-repo TEXT      The repository name that is being deployed.
  --build-id TEXT      The build the atifact should be associated with.
  --environment TEXT   Which environment you would like to run your tests
                       against.  [required]
  --commit TEXT        The Git commit that was deployed.
  --branch TEXT        The Git branch the deploy came from.
  --help               Show this message and exit.

Create an Environment

Usage: testery create-environment [OPTIONS]

  Creates an environment where tests can be run.

Options:
  --token TEXT                    Your Testery API token.  [required]
  --name TEXT                     The display name for the environment.
                                  [required]
  --key TEXT                      An identifier for the environment. This is
                                  used when creating test runs to indicate
                                  where tests should run.  [required]
  --pipeline-stage TEXT           The name of a pipeline stage to associate
                                  this environment to.
  --variable TEXT                 A variable to add to the environment.
                                  Specified as "KEY=VALUE". To encrypt value,
                                  pass in "secure:KEY=VALUE", Multiple
                                  variables can be provided.
  --maximum-parallel-test-runs INTEGER
                                  The maximum number of test runs that can run
                                  in parallel on this environment.
  --help                          Show this message and exit.

Create a Test Run

Usage: testery create-test-run [OPTIONS]

  Submits a Git-based test run to the Testery platform.

Options:
  --token TEXT                    Your Testery API token.  [required]
  --git-ref TEXT                  The git commit hash of the build being
                                  tested.

  --git-branch TEXT               The git branch whose latest commit you want
                                  to run.

  --test-name TEXT                The name you want to use on the Git status.
  --wait-for-results              If set, the command will poll until the test
                                  run is complete.

  --project TEXT                  Legacy option. Use --project-key instead.
  --project-key TEXT              The project key.
  --test-suite TEXT               Name of Test Suite to use for this test run.
  --environment TEXT              Legacy option. Use --environment-key
                                  instead.

  --environment-key TEXT          Which environment you would like to run your
                                  tests against.

  --include-tags TEXT             List of tags that should be run.
  --exclude-tags TEXT             List of tags that should excluded from the
                                  test run.

  --copies INTEGER                The number of copies of the tests to submit.
  --build-id TEXT                 A unique identifier that identifies this
                                  build in your system.

  --output TEXT                   The format for outputting results
                                  [json,pretty,teamcity]

  --fail-on-failure               When set, the testery command will return
                                  exit code 1 if there are test failures.

  --include-all-tags              When set, overrides the testery.yml and runs
                                  all available tags.

  --parallelize-by-file           Pass this flag if you want the test run to
                                  parallelize by file/feature.

  --parallelize-by-test           Pass this flag if you want the test run to
                                  parallelize by test/scenario.

  --timeout-minutes INTEGER       The maximum number of minutes this test run
                                  can take before it is killed automatically.

  --test-timeout-seconds INTEGER  The maximum number of seconds a test can
                                  take before it is killed automatically.

  --runner-count INTEGER          Specify number of parallel runners to use in
                                  for this testrun.

  --variable TEXT                 A variable to add to the enviroment.
                                  Specified as "KEY=VALUE". To encrypt value,
                                  pass in "secure:KEY=VALUE", Multiple
                                  variables can be provided.

  --test-filter-regex TEXT        A regular expression to be used for
                                  filtering tests.

  --help                          Show this message and exit.

Upload Build Artifacts

If your project is using a compiled language (like .Net, Java) or if you aren't attaching a code repository to your project, then you'll have to upload your build artifacts to Testery before you can run a test. To do that you can use the upload-build-artifacts command of the Testery CLI.

> testery upload-build-artifacts --help
Usage: testery upload-build-artifacts [OPTIONS]

  Uploads a file or directory of build artifacts and associates them with
  the specified build-id

Options:
  --token TEXT        Your Testery API token.  [required]
  --project TEXT      Legacy option. Use --project-key instead.
  --project-key TEXT  The project key.
  --branch TEXT       The Git branch the build came from.
  --build-id TEXT     The build the artifact should be associated with.
                      [required]

  --path TEXT         The path to the file or directory you want to upload.
                      [required]

  --zip-dir           Creates a zip file of the directory contents before
                      uploading.

  --help              Show this message and exit.