Run tests in your pipeline

One test, any pipeline

gtr-cli, the command-line runner, runs a test exactly like the Run Test button: same checks, same report, same history.It adds what a pipeline needs: an exit code, result files your CI tool understands, and every piece of evidence of a failure in one folder.

jobs:  gui-tests:    runs-on: [self-hosted, windows]    steps:      - run: gtr-cli run tests --format github --junit results.xml

Prepare the build machine

A GUI test replays on a real screen, so the machine that runs it needs one.

Create the key on your account page, under Machine keys, then run this once on the agent, as the Windows account the pipeline uses:

gtr-cli login --key GTS-XXXX-XXXX-XXXX-XXXX

gtr-cli whoami says whether the agent is signed in and until when.Without a sign-in, a run does not start: it prints a line starting with COULD NOT RUN and ends with exit code 3.

Build agents: machine keys

Your first command

In a terminal on the build machine, run every test of a folder:

gtr-cli run tests --countdown 0 --junit results\junit.xml --artifacts results\artifacts

--countdown 0 starts at once (nobody is at the screen), --junit writes the results file your CI tool reads, and --artifacts gathers videos, reports and failure pictures into one folder.

gtr-cli check tests

What your pipeline gets back

The runs also land in each test's results folder, so the History of GUI Test Runner-Recorder shows pipeline runs beside your own.

Ready-made examples

jobs:
  gui-tests:
    runs-on: [self-hosted, windows]
    steps:
      - uses: actions/checkout@v4
      - run: gtr-cli run tests --countdown 0 --format github --junit results\junit.xml --summary results\summary.md --artifacts results\artifacts
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gui-test-results
          path: results

Build a command without typing it

In either app, Command Line & CI/CD (in GUI Test Studio's File menu, in GUI Test Runner-Recorder's Test menu) builds the command for you: tick the options you want, copy the result, or copy the ready-made snippet for your CI tool with your real paths.Its Reference tab explains every option.

Enlarge

Options you will use

gtr-cli help

Report failures to Jira

A failed test can become a Jira issue by itself: the steps to reproduce, the expected and found pictures with what changed outlined, the bug report as a PDF and the evidence.The same defect seen again, on another build, browser or device, adds a comment to its open issue instead of a second one.

gtr-cli run tests --countdown 0 --jira on-failure --jira-project SCRUM --jira-labels ui-regression

The connection comes from the environment, so the token never shows in a command line or a build log: JIRA_BASE_URL (your Jira site), JIRA_USER_EMAIL and JIRA_API_TOKEN.In your CI tool, keep the token as a secret or password parameter.On a computer where GUI Test Runner-Recorder is connected to Jira, its stored connection is used instead.

The whole flow, with a real issue and a real build: Connect Jira to the Confidence Report.

Web apps on a build machine

A test recorded on a web app opens its browser by itself, in full-screen kiosk mode, and closes it at the end.On a build machine that asks for the screen size and display scaling of the recording, a signed-in and unlocked desktop, and a run that ends by its own time limit.Open Web App says what happens otherwise and what to do.

The same notes are in the tool itself:

gtr-cli help web-apps

Every browser and device, in the pipeline

A web test made for several browsers and devices runs them all with --branch all, each with its own entry in the JUnit file and its own history.Phones, tablets and the other branches without a window run side by side, so they need no screen of their own.gtr-cli info lists the name of each, and gtr-cli generate makes the ones still missing.See Every browser and device from one recording.

gtr-cli generate tests --branch missing
gtr-cli run tests --branch all --countdown 0 --junit results\junit.xml

Share tests without their recording

A test file holds its recording, which you need for editing but not for running.File, Export as Lightweight Runner Test (.GTR) in GUI Test Studio writes the same test without it: a much smaller file for a build machine or a colleague who only runs tests. gtr-cli and GUI Test Runner-Recorder run .GTR files like any test.

When to use a runner test

Use it wherever a test only runs: a build machine, a tester's computer, a set of tests that has to travel or be kept with your code.A run of a runner test gives the same result and the same report as its test.

Keep the test itself

A runner test cannot be edited: GUI Test Studio does not open it.Keep the .GTS as the original, edit it there, and export again after a change.GUI Test Runner-Recorder can export one too, from its Test menu or the export button on the test's page.

A web test with several browsers and devices

The export asks what to include.Every Browser and Device puts every generated branch in the one file, ready for --branch all.Only the recording, or only the branch you show, makes a runner test of that one alone.A runner test can only be run: new browsers and devices are made from the .GTS.

Keeping tests with your code

Tests are ordinary files: keep them in your repository next to the code they test, and every branch carries the tests that fit it.