AI Test Generator: From a Prompt to an End-to-End Test That Runs
What an AI test generator really leaves you: test code, an AI at every run, or a test that replays by itself.How to turn a prompt into an end-to-end test you can trust.
10 min readBy the GUI Test Studio team
What is an AI test generator?
Definition
An AI test generator turns a request in plain words into an automated test.
- You write "sign in and check that the dashboard shows my name", and the AI does the rest.
- It replaces hours of writing test scripts with a sentence.
That promise is real, and it is changing how teams test.But two generators can take the same sentence and leave you with very different things.The question to ask is not whether a tool can generate a test.It is what you are left with the day after.
Three kinds of AI test generators
AI test generators work in one of three ways.
It writes test code
The AI writes test code from your request, and from then on the code is yours to run, read and repair.
It tests at every run
An AI agent performs the test itself every time it runs, looking at the screen and deciding what to do.
It records a test
The AI performs the workflow once on the real application, and what it did becomes a recorded test that replays by itself.
Where the AI works: at creation, or at every run
Generates test code
AI writes codeCode runsCode runsCode runsYou maintain the code
Tests with AI at every run
AI exploresAI decidesAI decidesAI decidesDifferent answers, a cost every run
Records a test, replays it
AI recordsTest replaysTest replaysTest replaysSame answer every run, nothing to maintain
| Question | Generated code | AI at every run | Recorded test |
|---|---|---|---|
| What do you get? | A test script in a programming language. | A description the AI follows each time. | A test you can open, read and edit. |
| Who maintains it? | A developer, like any other code. | Nobody, until it decides differently. | Anyone on the team, without code. |
| Same answer on every run? | Yes, once it works. | Not guaranteed. | Yes. |
| Cost of each run | The build machine's time. | AI calls, every run. | The build machine's time. |
What goes wrong with AI-generated tests
Teams that tried AI test generation run into the same four problems.
- Checks are invented without being tried: a test expects a message the application never shows, or checks something that would pass whatever happens.
- Generated code is long and nobody read it, so the first broken selector after a redesign lands on a developer who has never seen the test.
- An AI that decides at every run can take another path on Tuesday than on Monday, so a red result may be the AI and not your application.
- And every run costs AI calls, which adds up fast once a suite runs on every build.
A test that might answer differently tomorrow is not a regression test.It is an opinion.
This matters even more when the code under test was also written by AI.An independent test that a person owns is the safety net, as how to test AI-generated code explains.
What a good AI test generator does
Five properties separate a test you can rely on from a demo.
- Every check is tried on the real screen before it is kept, and refused if it could not pass.
- The result is a test a person can read and change without code.
- The test runs the same way every time, with no AI needed at run time.
- Nothing is changed or overwritten without your approval.
- And you choose the AI: a cloud assistant, or a local model on your own computer.
From a prompt to a test with GUI Test Studio
GUI Test Studio lets an AI assistant create end-to-end tests of web apps, Windows desktop apps and Android phones, and it was built around those five properties.
- 1
Open Create a Test From a Prompt
In GUI Test Runner-Recorder, click Create Test From Prompt, beside Record.Type what the test should do, or start from one of the examples: a whole app, one workflow, several browsers and devices, a Figma design.
- 2
Send it to your AI assistant
Claude Desktop, Claude Code, ChatGPT with Codex and Copilot Chat in VS Code each have a card with one button: Connect the first time, Open after that.No key and no setting: the connector is installed with GUI Test Studio.
- 3
Watch it work on your screen
The assistant opens the app or the web page, clicks and types for real, and adds a check after each step, as a tester would.Each check is tried on the screen right there, and refused if it could not pass.The time it spends thinking is left out of the test.
- 4
Open the test, or run it
The result is an ordinary GUI Test Studio test.Open it on the timeline to see every step and check, run it in GUI Test Runner-Recorder, and read its Confidence Report.Nothing about the test says a model made it.See Create a test from a prompt.
How to write a prompt that makes a good test
A good prompt reads like a test case: where to start, what to do, and what must be true at the end.The last part is the one people forget, and it is the one that makes it a test.
Create a test for my app C:\Apps\Shop.exe: add a client named Atlas Textiles and verify it is listed.Test the sign-up form at https://shop.example/signup in Chrome, Edge and Firefox and on an iPhone 15.With GUI Test Studio, test the whole of C:\Apps\Shop.exe: list its main workflows and create one test per workflow, each verifying its outcome.Do
- Name the app's path or the page's address
- List the steps in the order a user takes them
- Say what must be true at the end
- Start with GUI Test Studio when the assistant could do it another way
Avoid
- "Test the clients page", with no expected result
- Ten workflows in one test
- Using the mouse while the assistant works
- Asking for a test script in a programming language
No cloud AI?A local model on your own computer
Some teams cannot send their screens to a cloud AI.GUI Test Studio can create the test with a local AI model instead: one we provide, downloaded once into its built-in engine, or one you bring through LM Studio or any OpenAI-compatible local server.
The local model uses exactly the same tools as Claude and ChatGPT.
- Your request and your screen stay on your computer.
- A set-up window recommends the model that suits your computer's graphics card.
- A local model is slower than a cloud assistant, and best with short workflows.
Where test data goes, and how to keep it in-house, is the subject of data privacy in test automation.
Change a test by prompt, with your approval
The same assistant changes an existing test in place: a check that should wait longer, a step to ignore, a new check after a given step.Nothing is recorded again.
In the Checkout test, let the "Order confirmed" check wait 10 s and continue the test if it fails.The assistant can also create a test from a Figma design before the feature exists, run tests, explain why one failed and file the failure as a Jira bug once you confirm.See tests from Figma designs.
Run it on every build, with no AI at all
A test made from a prompt is a test like any other. gtr-cli runs it on a Windows build machine, with no assistant, no AI call and no key: a failed test fails the build.
gtr-cli run tests --countdown 0 --junit results\junit.xmlThe run checks what the screen shows, so it fails for a real reason.
- Each element is found through the application's accessibility information first, and by sight when the application does not say: that is the GTS Engine + AI.
- Harmless rendering differences are ignored, and a changed word or a missing button is caught.
- When a picture differs, AI Smart Verification can explain the difference, as an option.
You describe it
One sentence, in plain words
The AI records it
On your screen, every check tried
A test you own
Read and edit it without code
Every build
Replayed with no AI
Key takeaways
- 1An AI test generator is judged by what it leaves behind, not by the demo.
- 2Generated code must be maintained; an AI at every run answers differently and costs every run.
- 3Every check should be tried on the real screen before it is kept.
- 4The best test is made by AI once and replays with no AI, the same way every time.
- 5You choose the AI: Claude, ChatGPT, Copilot, or a local model, ours or your own.
Questions
What is an AI test generator?
An AI test generator turns a description in plain words, such as "sign in and check the dashboard shows my name", into an automated test.Depending on the tool, the result is test code to maintain, an AI that performs the test at every run, or a recorded test that replays by itself.
Can AI write end-to-end tests?
Yes.An AI assistant can open the application, perform a workflow and add the checks a tester would add.What matters is what it leaves behind: a test whose checks were tried on the real screen, that a person can read and change, and that runs the same way every time.
Does a test made by AI need AI to run?
Not with GUI Test Studio.The AI creates the test once, then the test is an ordinary file that GUI Test Runner-Recorder or gtr-cli replays with no AI at all: the same steps, the same checks, the same answer on every run.
Which AI assistants can create GUI Test Studio tests?
Claude Desktop, Claude Code, ChatGPT with Codex and Copilot Chat in VS Code, through the connector installed with GUI Test Studio.Without an assistant, a local AI model on your own computer creates the test with the same tools: one we provide, or your own.
Can I generate tests with a local AI model?
Yes.GUI Test Studio can create a test with a local AI model, one we provide in its built-in engine or one you bring through LM Studio or any OpenAI-compatible local server.Your request and your screen stay on your computer.A local model is slower than a cloud assistant and best with short workflows.
How do I write a good prompt for an AI test generator?
Name the application or the address, the steps in the order a user takes them, and what must be true at the end."Add a client named Atlas Textiles and verify it is listed" makes a better test than "test the clients page", because the expected result is part of the request.
Can AI generate tests from a Figma design?
Yes.Give the assistant the design package exported with GUI Test Studio's Figma plugin, and it creates a test from it: every screen becomes a reference picture, every element a check, and a prototype flow becomes the click path, before the feature is built.
Claude, ChatGPT, Codex, Copilot, VS Code, LM Studio, OpenAI, Anthropic, Figma and Jira are trademarks of their respective owners.
Ready to record your first test?
Every feature, for 15 days.
Rather see it on the kind of app you test?Book a free 30-minute call with our team.