Repeatable AI Testing: Use AI to Create the Test, Not to Improvise Every Run
Repeatable AI testing: AI creates the test, then the same steps replay at every run. Why a test must not improvise, and where AI still helps.
Repeatable AI testing means AI helps create and change a test, while the test itself replays the same steps and gives the same answer at every run. AI works where judgment helps, and never improvises the run.
A build turns red on Monday and green on Tuesday, and nothing changed in the app. If an AI chose the steps of each run, you cannot tell whether the app broke or the AI took another road. A red build that may mean nothing is soon ignored.
What is repeatable testing?
Definition
A repeatable test gives the same result every time it runs on the same app.
- The same steps happen in the same order.
- Each check is decided by the same rule.
So when the result changes, the app changed.
That property is what makes a test useful on every build.A red result is a fact about your product, not about the test.It is the whole point of regression testing.
Why can an AI give two different answers to the same question?
An AI model does not look an answer up. It composes one each time it is asked, and it can compose it differently. Ask an assistant the same question twice and you often get two wordings, sometimes two conclusions.
That is a strength when you want a draft, an idea or an explanation. It is a weakness when the answer is a verdict that must hold for three hundred builds.
A test must give the same answer twice. Otherwise a red build means nothing.
Two ways to use AI in test automation
AI can sit in two places in a test.
AI drives every run
An AI reads your instructions and drives the app again at every run, choosing each click as it goes.
AI helps, the test runs itself
Or AI helps where a person would, when the test is created, changed or reviewed, and the run itself is a fixed recording.
An AI drives every run
Run 1
Run 2
Run 3
Did the app break, or did the AI take another road?
A saved test replays
Run 1
Run 2
Run 300
A red run means the app changed.
| Question | An AI drives every run | A saved test replays |
|---|---|---|
| Are the steps the same at every run? | Not guaranteed: each run is a new reading of the instructions. | Yes: the same steps in the same order. |
| Can you reproduce a failure? | Not always: the next run may take another path. | Yes: run the test again, or watch the run's video. |
| What does a run cost? | AI use at every run of every test. | The build machine's time. |
| How fast is a run? | The AI thinks before each step. | As fast as the recorded workflow. |
| What does a red build mean? | The app changed, or the AI chose differently. | The app changed. |
The two ways have names, set side by side in agentic testing vs AI-assisted testing.
Where AI helps, and where a test must not improvise
The question is not whether to use AI in testing. It is where. Some parts of testing gain from judgment, and others must stay fixed.
| Part of testing | What AI does there | Why |
|---|---|---|
| Creating the test | It helps: a sentence becomes steps and checks, tried on the real screen. | It is the work a person would do, done faster. |
| Changing the test | It helps: it makes the change you ask for, once you confirm. | The decision stays yours. |
| Running the steps | Nothing: the saved test replays. | The same steps must happen at every run. |
| Deciding each check | Nothing: the same comparison every time. | The same screen must get the same verdict. |
| A check that failed | An optional second look: a final verdict on that check, and an explanation. | A difference sometimes needs reading, as a person would read it. |
| Reading the results | It helps: your assistant reads a run and explains it. | It saves you time and changes nothing in the test. |
AI test automation you can rely on: how GUI Test Studio does it
GUI Test Studio, which automates end-to-end tests of any app without code, is built on the second way. You describe the test in a prompt to Claude, to ChatGPT with Codex, or to a local AI model we provide or you bring. You can also record it yourself, with no AI at any point.
Your prompt
Create an order for Atlas Textiles and verify the total.
AI works once
It performs the workflow on your screen and tries each check there.
A saved test
Order.GTS
Its steps, its pictures, its recording.
Run 1
- 1Open Orders
- 2New order
- 3Type the client
- 4Save
- 5Check the total
Run 2
- 1Open Orders
- 2New order
- 3Type the client
- 4Save
- 5Check the total
Run 300
- 1Open Orders
- 2New order
- 3Type the client
- 4Save
- 5Check the total
You describe it
One sentence
AI records it
Once, on your screen
A saved test
A file you own
Every run
The same steps
Three things make the run repeatable.
- The AI works once: it performs the workflow on your real screen and tries each check there before it keeps it.
- What it leaves is a saved test, a file with its steps, its pictures and its recording.
- When that test runs, it replays the same steps in the same order, and no AI drives the app.
When the app changes, the same assistant changes the saved test in place: another expected text, a longer wait, a step removed. It shows the change and asks you to confirm before it writes. Between two changes, the test is the same file, and so is its run.
Create a test from a prompt shows each window, and AI test generator: from a prompt to a test that runs tells how to write a prompt that makes a good test.
Each check is decided the same way every time
A run is repeatable only if its verdicts are. Every visual check answers two fixed questions: was the element found on the screen, and does it look as expected? The comparison is perceptual, like a human eye. It notices what a person would notice, such as a wrong digit, a missing icon or a changed word, and ignores what nobody sees.
Three more things keep a run steady.
- Every check waits for its element and passes the moment it is there, so a slow screen does not change the result.
- A value that changes by nature, such as a date or an order number, is checked by its shape.
- Each check says what a failure does: stop the test, or continue.
Timing is the first cause of tests that pass and fail at random: see fixed sleeps in tests: wait for the screen instead and flaky tests and false positives.
When a check fails: AI takes a second look, and its answer is kept
Some differences need judgment. Is a slightly bolder title a bug? AI Smart Verification is the optional AI side of the GTS Engine + AI. Only when a check fails, it takes a second look at that check, gives the final verdict on it and explains the difference in plain words.
The checks of one run
Sign-in title
Passed: no AI looks at it
Client name
Passed: no AI looks at it
Order total
Passed: no AI looks at it
Confirm order button
Failed: the only one AI looks at
Order number
Passed: no AI looks at it
AI Smart Verification, if you switched it on
A second look at the failed check
Final verdict: failed
The Confirm order button has changed from active to disabled.
Four rules keep that judgment in its place.
- The AI never drives the app, and it never looks at a check that passed.
- Its verdict concerns one failed check, and the report shows its reasons beside the two pictures.
- Its answer is kept with the run, so opening the report again asks nothing again.
- With AI off, every check passes or fails on its own comparison.
So the run is the same every time, and the one place where a model judges is small, visible and yours to switch off.
Guide the AI in plain words: AI Instructions
Some differences are expected, and only you know which. A check can carry AI Instructions: a few words in your own language that the AI follows when it judges that check.
| You write | The AI |
|---|---|
| Ignore changes to the font size. | Leaves a larger or smaller text out of the differences. |
| The date in the corner changes every day. | Does not count the date as a difference. |
| The Save button must stay green. | Fails the check whenever the button is another colour. |
The report shows the instructions a verdict was given under, beside the AI's answer, so a judgment is never a mystery.The AI Smart Verification guide has every setting.
You choose the AI, or none
Which AI takes that second look is your choice, and you can change your mind.
A cloud AI, with your own key
A model of OpenAI or Anthropic answers through your own account, and only the two small pictures of a failed check travel to it.
A local AI model
A model we provide or your own runs on your computer, and nothing leaves it.
No AI at all
With no AI at all, you record the tests yourself, and every check passes or fails on its own comparison.
Tests, recordings and reports are files on your computers in every case.What that control is worth to a team is the subject of AI sovereignty in software testing.
A test that does not change by itself is also the independent check you need when AI writes the code: see how to test AI-generated code.And a test that repairs itself between runs raises the same question, as self-healing tests shows.
What to know before you start
| Limit | What it means |
|---|---|
| A test checks what it was told to check | A repeatable test does not explore. It repeats one workflow and its checks, so each workflow that matters needs its test. |
| The same start gives the same answer | Put the app back where the recording started before a run: the same page, the same data, the same open windows. |
| The AI's second look is a judgment | On a borderline difference, a model's reading can vary. It is asked about failed checks only, its reasons are shown, and it can stay off. |
| A run uses the real screen | A run moves the real mouse and types on the real keyboard, unless a web test runs without a browser window. |
Key takeaways
- 1A test must give the same answer twice, or a red build means nothing.
- 2An AI that drives every run improvises: another path, a cost per run, a failure you cannot reproduce.
- 3In GUI Test Studio, AI creates the test and changes it when asked. The saved test replays the same steps, with no AI driving the app.
- 4Each check is decided the same way every time. Only a failed check can get an AI's second look, and its answer is kept with the run.
- 5You choose the AI: a cloud AI with your own key, a local AI model, ours or your own, or none.
Questions
What is repeatable testing?
Repeatable testing means a test gives the same result every time it runs on the same app. The same steps happen in the same order, and each check is decided by the same rule. When the result changes, the app changed.
Is AI testing repeatable?
It depends on where the AI sits. An AI that drives the app at every run reads its instructions anew each time and can take another path. A test that AI created once and that replays by itself is repeatable: the same steps and the same checks at every run.
Why do AI-driven tests give different results from one run to the next?
Because an AI model composes its answer each time it is asked, and it can compose it differently. If the model chooses each click during the run, two runs on the same app can follow two paths. A failure seen once may not come back when you try to reproduce it.
Can I use AI in test automation and still get repeatable results?
Yes. Use AI to create the test and to change it when you ask, and let the saved test run by itself. In GUI Test Studio, a test made from a prompt replays the same steps at every run, with no AI driving the app.
Does GUI Test Studio use AI during a test run?
No AI drives the app during a run: the saved test replays its steps. If you switch AI Smart Verification on, it looks only at a check that failed, gives the final verdict on that check and explains the difference. It is optional.
What does AI Smart Verification do when a check fails?
It takes a second look at the failed check and gives its final verdict. It passes the check when nothing major changed, fails it when something did, and says what changed in plain words. Its answer is kept with the run.
Can I run my tests with no AI at all?
Yes. Record the test yourself and leave AI Smart Verification off. Every step replays and every check passes or fails on its own comparison, with no model involved at any point.
How do I tell the AI what to ignore in a check?
Give the check AI Instructions, a few words in your own language such as "ignore changes to the font size". The AI follows them when it judges that check, and the report shows the instructions beside its answer.
Claude, ChatGPT, Codex, OpenAI, Anthropic and Windows are trademarks of their respective owners.


