Flaky Tests and False Positives: Why Tests Fail on a Working App

A red build should mean one thing: the application is broken.Too often it means something else: a test clicked too early, a font rendered a shade differently, a spinner was still turning.These are flaky tests and false positives, and they are one of the main reasons teams stop trusting their end-to-end tests.

This guide explains where they come from in coded and visual tests, and how to stop false positives without letting real bugs through.

What are flaky tests and false positives?

A false positive is a test that fails while the application works as it should.Testers also call it a false failure or a false alarm.The test raises an alarm, someone investigates, and nothing is wrong.

A flaky test is a test that passes and fails on the same code, with nothing changed between the runs.Every flaky failure is a false positive.Not every false positive is flaky, though: a test that fails every time because its expected picture is outdated is wrong, not flaky.

The opposite mistake is the false negative: a test that passes while a real bug is on the screen.It is the more dangerous of the two, because nobody looks twice at a green test.

What false positives cost a team

One false alarm costs an investigation.Someone opens the logs, reruns the test and finds nothing.

The real cost comes later.When most red builds are noise, people stop reading them: this is alert fatigue.Then come the habits that hide bugs: rerunning a test until it turns green, accepting every new screenshot as the baseline without looking, disabling the test that always fails.

Each of these habits turns today's false positive into tomorrow's false negative.The bug the test was written for walks through, because nobody believes the test any more.

What causes flaky end-to-end tests?

End-to-end tests drive a whole application through its interface, so they meet everything a real user meets: slow servers, animations, other windows.Coded tests add causes of their own.

Timing comes first for a reason.Selenium's own documentation calls the race between the test and the browser one of the primary causes of flaky tests.

Why visual regression tests raise false positives

Visual regression testing compares a picture of the application with an approved baseline picture.It catches what coded assertions miss: a wrong color, an overlapping label, a missing icon.It also sees differences no person would call a change.

A strict pixel-by-pixel comparison fails on every one of these.A loose one lets real bugs through.The craft of visual testing is telling the two apart.

How teams fix flaky tests today

In coded tests

Playwright waits before each action until the element is visible, stable, enabled and not covered by another element, and fails with a timeout otherwise.

Cypress retries its queries and assertions until they pass, for up to 4 seconds by default.It can also retry a whole failed test, which is off by default.

Selenium offers explicit waits that poll the application for a condition, and its documentation warns against mixing them with implicit waits.

Retries also serve to detect flakiness.Playwright reports a test that failed and then passed on a retry as flaky, and Cypress Cloud marks such a test with a Flaky badge.A retry makes the build green, but it does not remove the cause.

In visual testing

In Playwright, toHaveScreenshot takes a color threshold per pixel, 0.2 by default, and either an absolute maxDiffPixels or a maxDiffPixelRatio.It can mask dynamic areas, and by default it stops animations and hides the text caret.

BackstopJS allows a percentage of different pixels, 0.1 by default, and can hide or remove the elements you name before it takes its pictures.

Chromatic ignores anti-aliased pixels by default, and its diffThreshold, 0.063 by default, sets how much change it tolerates.

Argos stabilizes every screenshot by default: it waits for fonts and images, hides the text cursor and the scrollbars, and flags screenshots that change from run to run.

Percy freezes animations and other dynamic elements, and changes are approved by the build, by the group or one snapshot at a time.Its Visual Review Agent highlights the changes it considers meaningful.

Applitools Eyes compares with match levels.Strict aims to ignore what the human eye would not see, Layout checks only the arrangement, and Dynamic accepts any valid date, email address or URL.

Each of these helps.Each one also asks you to pick a number, a mask or a level, test by test, and to review new pictures by hand.To see these tools side by side, read the comparison study.

Why a looser tolerance is not the answer

A tolerance is a promise: this much difference is noise.The question is what this much is measured in.

A ratio grows with the picture.On a full screen of 1920 by 1080 pixels, 0.1 % is about 2,000 pixels, more than a changed word.On a button of 100 by 40 pixels, it is 4 pixels, fewer than one edge that rendered differently.

An absolute count does not grow with the picture, but it adds up noise scattered across the whole picture as if it were one change.A mask hides the noise, and it also hides any bug inside the mask.

How GUI Test Studio stops false positives without missing bugs

GUI Test Studio is an end-to-end test automation suite for web applications, Windows desktop applications, Android and games.You record a test once, or describe it in a prompt, edit it on a timeline and run it on every build.Its checks follow one rule: a test fails when a person would see a problem, and only then.

Elements are found two ways, with no selector to break

The GTS Engine + AI first asks the application where an element is, through its accessibility information on Windows, or the page's own elements in a browser.When the application does not say, it finds the element by sight on the screen.There is no locator to write, so a renamed class or a new framework does not break a test.See how GUI Test Studio finds every element.

Checks wait for the screen, never for a timer

Every check keeps looking until what it expects is on screen, up to its timeout of 5 seconds by default, and the test goes on the moment it is there.A Visual Absent Assert waits for a spinner to go away, so a slow page is waited for, not guessed.

When a Visual Assert or a Text Assert has not found its element in its search area after a second, it looks over the whole screen too, so a window that moved does not fail the test.Before a run starts, GUI Test Runner-Recorder compares your screen with the first moment of the recording and holds the run back when the application is on another page.See the settings of a check.

The Verification Phase counts what changed, and where

Once the element is found, the Verification Phase compares it with what you expect.It counts the pixels that clearly changed in the busiest small area of the element: an absolute number, never a percentage of the picture.

Noise is scattered: a softer edge here, a lighter shade there.A real change is packed in one place: a letter, a digit, an icon.That is why the same rule works on a small icon and on a whole dialog.

Each check has a Verification Strictness: Exact, Strict, Standard or Relaxed.Standard suits everyday buttons, labels and icons, and Relaxed suits large areas and busy screens.See visual checks.

The right moment, with the pointer where it stood

An element is often found while the screen is still settling: a menu fading in, a page still loading.The Enhanced Accuracy Pass looks at the frames that follow the match, keeps the one where the element looks its best, and stops where the test's next step begins.See the Enhanced Accuracy Pass.

In a recorded test, the run puts the mouse pointer back where it stood when the picture was taken.A button is then highlighted in the run exactly when it was highlighted in the recording.

A test you have just recorded, and not yet edited, gets extra help.When an element does not show up as it was recorded, the run adjusts the test smartly to find it, and it never does so to hide a real difference.

Different on purpose: Baseline Profiles

Some differences are real and wanted: dark mode, a laptop screen instead of a 4K monitor, another version of Windows, the application in French.A Baseline Profile keeps one expected version of a check per environment, so one test serves them all.Where to look, how long to wait and how strict to be stay shared by every version.See Baseline Profiles.

AI Smart Verification gives the final verdict

When a check finds changed pixels, AI Smart Verification can take a second look and decide, with a cloud model and your own key, or a local AI model on your own computer, one we provide or one you bring.It is off until you switch it on.

It passes a check when it finds only minor differences a user may not notice, while a changed word, number or value, or a missing icon or badge, is never minor and fails the check.The report says which it was.

The mouse pointer and the blinking text cursor never count as a difference, while a letter or an icon that changed still does.You can also give a check AI Instructions in your own words, such as: the date in the corner changes every day.See AI Smart Verification.

Enlarge

A report that says what changed

The Confidence Report shows every visual check in two phases.The Search Phase answers whether the element was found, and the Verification Phase answers whether it looks as expected.Pulse lights up what changed on the screen, and the verdict says it in plain words.

Enlarge

Its weakest links point at the checks that passed by the smallest margin, the ones most likely to fail next.History shows when a check started failing.When a change is wanted, you accept the new look in the report, and nothing is written until you save.See the Confidence Report.

Stop or continue: you decide what a failure means

Each check says what its failure does: stop the test, or continue it.A check the clicks after it depend on stops by default, because clicking without it would hit the wrong place.A check that only confirms what is on screen continues, so one run reports every problem it meets.

In a pipeline, gtr-cli can run a failed test once more with its retries option.Treat it as a last resort: the causes above are what to fix.

A checklist against false positives

Want to see this on your own application?Visual testing with GUI Test Studio shows the whole idea in one page.

Questions

Selenium, Playwright, Cypress, BackstopJS, Chromatic, Argos, Percy and Applitools Eyes are trademarks of their respective owners.