Run your end-to-end tests on every build: a TeamCity example that works with any CI/CD tool

Why run end-to-end tests on every build

An end-to-end test drives your application the way a user does: it opens it, clicks and types through a whole workflow, and verifies what the screen shows at the end.It is the test that catches what unit tests cannot: a broken button, a form that no longer submits, a page that lost its content.

An end-to-end test that runs only when someone remembers to run it catches those problems late.Run it on every change instead, and a regression is caught minutes after it was pushed, while the change is still fresh in the developer's mind.

GUI Test Studio comes with a command-line runner for exactly that: gtr-cli.It is installed with the apps.It runs the end-to-end GUI tests you recorded and edited, without opening a window, and it answers the way a pipeline expects: an exit code, a JUnit file, a log your CI tool understands.

What you need

Step 1: Prepare the build machine

An end-to-end test replays clicks and typing on a real screen, so the build agent needs a desktop to work on.

1. 1

   Install GUI Test Studio and sign in once

   The installer puts gtr-cli on the PATH.Sign in to the apps once on that machine, with the account the agent runs as.

2. 2

   Run the agent in a signed-in Windows session

   Start the TeamCity agent from a signed-in user session, not as a Windows service: a service has no screen to replay a test on.Keep the screen size and scaling of the recording.See Prepare the build machine.

3. 3

   Or run web tests headless

   A test of a web app can run without a browser window at all, with --headless on.It then needs no free screen, and several browsers and devices run side by side.

Step 2: Build the command

You do not have to learn the options.In either app, open the Command Line & CI/CD window, tick what you want, and copy the command, or the ready-made snippet for your CI tool with your real paths.

Enlarge

This is the command our example pipeline runs:

gtr-cli run tests\Checkout\Checkout.GTS --branch all --countdown 0 --format teamcity ^
  --junit ci-out\junit.xml --json ci-out\result.json ^
  --artifacts ci-out\artifacts --bug-report pdf

Every option is explained in Run tests in your pipeline and by gtr-cli help on any computer where GUI Test Studio is installed.

Step 3: Add the build step in TeamCity

Create a build configuration for your repository, add one build step and give it the command.A PowerShell or Command Line step both work.In our example the step runs a small script from the repository that starts the shop, then calls gtr-cli.

Enlarge

Add a VCS trigger so every push on every branch starts a build.That is the whole setup: TeamCity needs no plugin, because gtr-cli speaks its log language.

Step 4: Read the results in TeamCity

Push a change and watch.Each browser and device is a test of its own, so you see at a glance where it failed.

Enlarge

The build log

gtr-cli writes one line per step and says plainly what failed: which check, on which browser and device, and why.

Enlarge

The failed build

The build's overview names the failed tests and their reason, without opening the log.

Enlarge

The tests

The Tests tab lists every test of the build.Because each result is reported as a test, TeamCity also keeps its history, its duration and how often it failed.

Enlarge

The evidence

The artifacts hold everything the run saw: a folder per browser and device with the run's video, its report and its bug report, and for every failure the expected and found pictures with what changed outlined in red.Open the same run in GUI Test Runner-Recorder for the full Confidence Report.

Enlarge

Every browser and device in one build

A web test can be made once and run on several browsers and devices: Chrome, Edge and Firefox, desktops, phones and tablets.In the pipeline, --branch all runs them all in one build, and the ones without a window run side by side.Read Every browser and device from one recording.

Not using TeamCity?The same command anywhere

gtr-cli is not tied to any tool.It is a program that runs tests and returns an exit code, so anything that can run a command can run your end-to-end tests: a CI server, a scheduled task, a release script.Functional tests, regression tests, smoke tests, acceptance tests: whatever you call the tests that drive your application through its interface, they run the same way.

- name: End-to-end tests
  run: gtr-cli run tests --countdown 0 --format github --junit results\junit.xml --artifacts results\artifacts

Where to go next