What Is Regression Testing? End-to-End Regression Tests Without Code
What is regression testing, and why end-to-end regression tests catch what unit tests miss: what to cover, when to run it, and how to automate it without code.
11 min readBy the GUI Test Studio team
What is regression testing?
Definition
Regression testing checks that a change to an application did not break something that already worked.
- A new feature, a bug fix, a library update or a new browser version can each break a part of the application nobody touched.
- The same tests are run again after the change, and a test that passed before and fails now has found a regression.
The word comes from going backwards: the application regressed, it does less well today than it did yesterday.That is what makes a regression so frustrating for users.Something they relied on simply stopped working.
Regressions happen because parts of an application depend on each other in ways nobody keeps in mind.A shared style, a rounding rule, a field renamed in an API: a small change in one place reaches screens far away from it.The person who made the change checked what they changed, and they were right.The break is somewhere else.
Regression testing vs retesting
The two are often confused, and they answer different questions.
| Question | Retesting | Regression testing |
|---|---|---|
| What does it check? | That one bug you fixed is really fixed. | That the fix, or any other change, broke nothing else. |
| Where does it look? | At the place of the bug. | Everywhere else, especially what users rely on. |
| Which tests? | The steps that showed the bug. | The same suite, run again after every change. |
| When? | Once, after the fix. | After every change, for the life of the application. |
Unit, integration and end-to-end regression tests
Regression tests exist at every level of an application, and each level sees a different amount of it.
How much of the application each test sees
Unit tests
One function, alone
Catches: A calculation that returns the wrong value
Integration tests
Two or three parts together
Catches: A service that no longer answers the way the next one expects
End-to-end tests
The whole application, through its screen
Catches: A total, a button, a page or a message the user sees broken
Unit and integration tests are fast and precise, and every team should have them.But they check the code as the developer thinks of it.A user never calls a function: they click a button, read a total and expect the next page to open.
Some regressions only exist on the screen, where unit tests never look.
- A style change hides the Pay button behind a banner.
- A total is computed right and displayed with the wrong currency.
- A dialog opens behind the window.
- A translation is too long and the label is cut off.
That is why end-to-end regression testing matters most: an end-to-end test drives the whole application through its interface, the way a user goes through it, and checks what the user sees.See what end-to-end testing is for the full picture.
Your users do not run your unit tests.They run your application, screen after screen.
Functional and non-functional regression testing
Functional testing checks what the application does: the order is placed, the report is exported, the error message appears when a field is empty.Non-functional testing checks how it does it: how fast, how it looks, how it behaves on another device or a slow network.
Functional regression
The checkout still places the order, and the confirmation still shows the order number.
Visual regression
The screen still looks as it did: no missing icon, no word cut off, no button in the wrong colour.
Non-functional regression
The page still loads in time, and the layout still holds on a phone and on a slow connection.
Most regression suites are functional, and most visual and layout breaks slip through them because a functional check asks whether an element exists, not whether a person can see it.Visual bugs your automated tests miss shows why.
What belongs in an end-to-end regression suite?
A regression suite that tries to cover everything is never finished and never trusted.Start small, with what would hurt most if it broke.
- 1
List the journeys users cannot live without
Sign-up, sign-in, search, checkout, the report your customers export every Monday.Five to ten journeys, each from the first click to the result the user expects.
- 2
Add what broke before
Every bug that reached users once deserves a test, so it can never come back unnoticed.That is how a regression suite learns.
- 3
Weigh by risk, not by screen count
A screen used by every customer every day matters more than ten settings pages.Money, data and sign-in come first.
- 4
Check outcomes, not only clicks
A test that clicks through checkout and never checks the total proves little.Each journey ends on what the user came for, and the test verifies it: the total, the confirmation, the exported file.
- 5
Keep it fast enough to run on every change
Mark a handful of journeys as the smoke suite, the ones that run on every pull request in a few minutes.The rest runs every night.
When should regression tests run?
The sooner a regression is found, the cheaper it is: the change is still fresh in the developer's mind, and nothing has been built on top of it yet.
| When | What runs | Why |
|---|---|---|
| Every pull request or build | The smoke suite: the critical journeys. | A break is caught before it is merged, while the change is small. |
| Every night | The whole suite. | Everything is checked once a day, without slowing anyone down. |
| Before a release | The whole suite, on every browser and device you support. | What users will really get is checked where they will get it. |
| After an update you do not control | The whole suite. | A new browser, operating system or library version can break the application without a line of your code changing. |
Checkout journey, last eight builds
- #38
- #39
- #40
- #41
- #42
- Started failing#43
- #44
- #45
#42 the last run that passed#43 the change to look at first#45 fixed, and checked again
Why automated regression suites get abandoned
Most teams agree regression testing should be automated.Many have tried, and many automated suites end up switched off or ignored.The reasons are almost always the same.
The suite costs more to keep than it saves.
- Coded tests find buttons through selectors, and every redesign breaks dozens of them at once.
- Only developers can fix them, and developers are building features.
- Tests fail for harmless reasons, a shade of colour or a pixel of anti-aliasing, so people stop reading the results.
- The full suite takes hours, so it runs once a week, and regressions wait days to be found.
Automate end-to-end regression testing without code
GUI Test Studio is built for end-to-end regression testing of web apps, Windows desktop apps, phones and games, through their interface, with no test code to write or maintain.
Record each journey once
Click Record in GUI Test Runner-Recorder and go through the journey in your real application, the way a user would.The clicks, the typing and the checks of what appeared become a test.You can also describe the journey in a prompt to your AI assistant, and the test is created for you.
The test checks what the screen shows, not the code behind it.
- A Visual Assert compares the picture of an element, a Text Assert reads its words.
- The GTS Engine + AI finds each element through the application's accessibility information first, and by sight when the application does not say, so a redesign of the code behind a button does not break the test.
Edit it like a video
Every test opens in GUI Test Studio on a timeline: its steps as blocks, its checks beside them, the recording above.Change a check, add a wait, group the steps of a journey: no code, and anyone on the team can do it.See the timeline.
Run it on every build
gtr-cli runs the suite on a Windows build machine and answers the way a pipeline expects: a failed test fails the build, a JUnit file fills the test tab, and every picture of what failed is kept.
gtr-cli run tests\regression --countdown 0 --junit results\junit.xml --artifacts results\artifactsIt works in TeamCity, Jenkins, GitHub Actions, GitLab and Azure DevOps.See Run tests in your pipeline.
A real regression, or harmless noise?
Every run ends in a Confidence Report.For each check, it says whether the element was found and whether it still looks the way it was recorded, with the expected and found pictures side by side and the difference outlined.
Noise is ignored and regressions are caught.
- Slight rendering differences that no person would notice are not counted.
- A changed word, a missing badge or a button that turned grey fails the check.
- When a picture differs, AI Smart Verification can take a second look and say what changed in plain words.
An intended change is accepted in one click
Not every difference is a regression: sometimes the new screen is the right one.In the Confidence Report, accept what the run found as the new expected picture, and the next run checks against it.For screens that differ on purpose, a dark mode or another language, baselines keep one version per environment.
Know when it started
Every run is kept in the test's History.When a journey starts failing, History shows the last run that passed and the first one that failed, and two runs can be compared check by check.
Every browser and device, and a bug in your tracker
A web test recorded once runs in Chrome, Edge and Firefox, on phones and tablets, in other languages and screen sizes: see cross-browser testing.And a failed run can become a Jira issue by itself, with the steps to reproduce and the pictures: see failed tests as Jira bugs.
A change is pushed
A feature, a fix, an update
The build runs
gtr-cli runs the suite
Confidence Report
Regression or noise, with pictures
Merge, or fix first
A failure blocks the change
Do
- A small suite of critical journeys on every build
- Checks of what the user sees at the end of each journey
- A test for every bug that reached users
- Every browser and device before a release
Avoid
- A suite so large it only runs once a week
- Tests that click through a journey and verify nothing
- Failures nobody reads because half of them are noise
- Tests only developers can change
Key takeaways
- 1Regression testing checks that a change broke nothing that worked. The same tests, run again after every change.
- 2Retesting proves a fix; regression testing proves the fix cost nothing else.
- 3End-to-end regression tests catch what users feel: the totals, buttons, pages and messages unit tests never see.
- 4Start small: five to ten critical journeys on every build, the full suite every night, every browser and device before a release.
- 5A suite must tell a regression from noise, and cost little to keep, or it will be abandoned.
Questions
What is regression testing?
Regression testing checks that a change to an application did not break something that already worked.After a new feature, a bug fix or an update, the same tests are run again, and any test that passed before and fails now points to a regression.
What is the difference between regression testing and retesting?
Retesting checks that one bug you fixed is really fixed.Regression testing checks that the fix, or any other change, did not break the rest of the application.Retesting looks at one place, regression testing looks everywhere else.
What is end-to-end regression testing?
End-to-end regression testing runs the important journeys of an application through its interface, the way a user goes through them, after every change.It catches the breaks that only show on the screen: a wrong total, a button that no longer opens its dialog, a page that will not load.
Is regression testing functional or non-functional?
Both.Most regression tests are functional: they check that features still do what they should.A regression can also be non-functional, such as a page that became slow or a layout that broke on a phone, and those deserve tests of their own.
How often should you run regression tests?
Run a short suite of critical journeys on every pull request or build, the full suite every night, and the full suite on every browser and device before a release.The sooner a regression is caught, the cheaper it is to fix.
Can regression testing be automated without writing code?
Yes.With GUI Test Studio, you record a journey once by using the application, or describe it in a prompt, and the test is created for you.It runs again on every build with gtr-cli, and nobody has to maintain test code.
What is visual regression testing?
Visual regression testing compares what the screen shows now with what it showed before, to catch changes a user would see: a missing icon, a word cut off, a button in the wrong colour.It is one kind of end-to-end regression check.
OpenAI, Anthropic, Jira, TeamCity, Jenkins, GitHub, GitLab, Azure DevOps, Chrome, Edge and Firefox are trademarks of their respective owners.
Ready to record your first test?
Every feature, for 15 days.
Rather see it on the kind of app you test?Book a free 30-minute call with our team.