Automated Game Testing Without an SDK: Unity, Unreal, Godot

What is automated game testing?

Automated game testing means letting software play parts of your game and check what happens, so every new build is verified without a person repeating the same steps by hand.It covers the repeatable part of game QA: start the build, go through the menus, load a level, pause, finish the level, check the score.

Game test automation is harder than testing a website or a business app.A game draws everything itself.The player's input changes what happens next.Two runs are rarely identical, frame for frame.

This guide explains why the usual test tools see nothing inside a game, what game teams use instead, and how a visual, black-box approach tests a Unity, Unreal, Godot or custom engine game without adding anything to its build.

Why test tools see nothing inside a game

Desktop and web test tools find elements through the description an application gives of itself.A web page has its document, its elements and their ids.A Windows program describes its buttons, fields and lists to the system: that is the accessibility tree screen readers use.Selenium and Playwright find elements in the web page's structure.WinAppDriver and Appium find them in what an app tells the system about its controls, such as an automation id.

A game gives none of that.The engine opens one window and draws every frame on one surface, usually with the graphics card.The main menu, the inventory, the HUD and the dialog boxes are pixels, not controls.Ask Windows what is inside the window, and the answer is a single pane with no name.

If you are choosing an end-to-end tool for your other applications, see the comparison study.

How game teams automate testing today

Most studios rely on people first.Testers and playtesters go through each build, follow test plans and write bug reports.Playtesting stays essential: only a person can tell whether a jump feels right or a boss fight is fair.Yet a good part of a tester's day goes to checking, again, that what worked yesterday still works today.

To automate that part, video game QA automation usually instruments the game: the team adds code or a plugin to the build, so a test can reach the engine's objects from inside.

Unity test automation

The Unity Test Framework lets a team write tests in C#, with NUnit, in Edit Mode and Play Mode.The tests run in the editor or on target platforms such as Standalone, Android and iOS.

Unreal Engine automated testing

Unreal Engine has an automation test framework built in C++, with unit, feature, smoke, content stress and screenshot comparison tests.Gauntlet, Unreal's automation framework, runs sessions of a project that perform tests and validate the results.

Godot testing

Godot teams often add a testing add-on.GUT is a unit testing framework for the Godot Engine.GdUnit4 tests GDScript, C# and scenes in the Godot editor, and its scene runner can simulate input.

Tools that plug into the game

AltTester offers test automation for Unity and Unreal Engine.Its SDK is a plugin that instruments the game to expose the objects of its hierarchy.Tests are written in C#, Python, Java or Robot Framework.

GameDriver works with games built with Unity, Unreal Engine and Godot.You drop its agent into the project, and tests find objects with HierarchyPath, a query language close to XPath.

Airtest is an open source framework for games and apps that locates elements by image recognition, from Python scripts.Its companion, Poco, reads the game's object hierarchy once the Poco SDK is installed in the game.

These tools are strong when a test needs the game's inner state: a property, a position, a value no one sees.They also come with costs that are easy to underestimate.

What a visual, black-box approach adds

A black-box test sees the game only from the outside, the way a player does: the picture on the screen and the controls in their hands.It needs no access to the code, so it works the same way on every engine.

The two approaches are not rivals.An instrumented test is the right tool to set up a save game or read a value nobody sees.A black-box test is the right tool to check that what reaches the player is right.

What to automate in a game, and what to leave to people

Be honest about what a replay does well.It presses the same keys at the same moments.A game whose outcome depends on physics, randomness or exact timing does not always answer the same way, so a recorded test of a hard platforming section can fail for no real reason.

How do you test a game's HUD, menus and end screens?

A check looks at one part of the screen and says whether it is right.Here is where checks go on an example 2D game, drawn for this article.

Enlarge

Count the lives with a Visual Count Assert: exactly three hearts.Read a fixed label, or a value the level always reaches, with a Text Assert.Check a value that changes with a Dynamic Text Assert: the timer must look like a time, whatever time it shows.

Enlarge

On a menu, a Text Assert reads the title.A Visual Assert compares the selected item with its picture, so a highlight drawn in the wrong colour fails.The button hints at the bottom are a picture too, and the wrong icon for a controller is a real bug.

Enlarge

At the end of a level, a Text Assert waits for the words "LEVEL COMPLETE".A Visual Count Assert counts the stars.A Dynamic Text Assert checks that the final score is a number, and a Visual Assert finds the Next Level button.

Each check waits up to its timeout for what it looks for, so a menu that fades in is not a failure.A Visual Absent Assert waits until a loading screen is gone, so the test goes on the moment the level is ready.

Every kind of check is explained in Visual checks and Text checks.The game is found by sight, because it has no accessibility information: how GUI Test Studio finds the elements explains why that makes a test robust.

Keyboard, mouse and gamepad input

A game test is only as good as its input.GUI Test Studio records and replays the keyboard, the mouse, and Xbox-style controllers, the ones Windows sees as XInput controllers, up to four at once.

Every button press and stick move is recorded with its timing.On the timeline, a press and its release become one Pad Press block, and a run of stick moves one Stick Move block.While you edit, the preview draws the controller and what it holds at each moment.

When the test runs, GUI Test Runner-Recorder drives a virtual controller, so the game receives the presses from a controller.That needs a driver installed once on the computer that runs the tests.The app says so before the run and opens the driver's download page.

Each recorded press also gets a check of the whole screen, taken just before it.In a scene that moves a lot, set those checks to Relaxed, or replace one with a check of the part that matters.

For a game that aims or looks around with the mouse, turn on Record continuous mouse movement in GUI Test Runner-Recorder's Preferences, under Recording Defaults, Advanced.Every movement is then recorded and replayed, button held or not.All the events are described in Events.

Game test automation with GUI Test Studio, step by step

GUI Test Studio automates end-to-end tests on Windows: desktop applications, web applications, phones and games.Here is how a first game test comes together.

1. 1

   Install it and choose your profile

   Install GUI Test Studio on the Windows computer that runs your game.The first time it starts, it asks what you work on: choose Video game developer.The controller is then recorded and replayed from the start, and the recommended settings suit screens that move.

2. 2

   Record your game

   In GUI Test Runner-Recorder, click Record and choose An app.Click Browse... to pick your game's program, and tick the box that starts the app at the beginning of the test and closes it at the end.Every run then starts from a fresh game.

   To watch one part of the screen only, such as the game's view inside an editor, choose an area of the screen instead.

3. 3

   Play the path you want to check

   Go through the menus, start the level, do what the test is about, then press the Stop key.The recording adds a check before each click, and checks for what appears on the screen.

4. 4

   Add the checks that matter

   Open the test in GUI Test Studio.Drag a Visual Count Assert, a Text Assert or a Dynamic Text Assert from the Assertions palette to the moment you want, and draw where it looks.

5. 5

   Run it and read the report

   Click Run Test.The Confidence Report shows every check with the picture expected and the picture found, and says in plain words what failed.

Games are one of the cases on our use cases page, beside web, desktop and mobile apps.

Make game tests repeatable

A game test passes every time when the game starts the same way and the checks look for the right things.These habits help.

The details are in Web, phones, virtual machines and games.

Regression testing for games on every build

Regression testing for games means running the same checks on every new build, so a broken menu or a missing HUD element is found the day it appears. gtr-cli, the command-line runner, runs a test exactly like the Run Test button.It returns an exit code your CI tool understands, writes JUnit results, and gathers every failure's pictures in one folder.

gtr-cli run tests --countdown 0 --junit results\junit.xml --artifacts results\artifacts

The build agent needs an open, unlocked Windows desktop session, because the game draws on a real screen and the test plays it there.

A complete pipeline example, with TeamCity: run your end-to-end tests on every build.

Questions

Unity, Unreal Engine, Godot, GameMaker, Construct, Defold, Phaser, MonoGame, RPG Maker, AltTester, GameDriver, Airtest, Selenium, Playwright, Appium, WinAppDriver, Xbox, Windows and TeamCity are trademarks of their respective owners.