Smoke Testing vs Sanity Testing vs Regression Testing: Which One to Run, and When

Smoke testing vs sanity testing vs regression testing: what each one checks, when to run it, and how to run all three from the same recorded tests.

By Ait Nohaila, Business Development Representative 9 min read

Smoke testing checks that a new build starts and its main features respond. Sanity testing checks one fix or change in depth. Regression testing checks that nothing that worked before is broken. They differ in width, depth and timing.

Most teams know the three names and still run all three by hand, when there is time. When a release is late, the long regression pass is the first thing cut, and an old bug comes back in front of a customer.

Smoke vs sanity vs regression testing: the short answer

Picture your app as a grid: its features from left to right, and how deeply each one is tested from top to bottom. The three kinds of testing light up different parts of that grid.

Smoke testing

Wide and shallow

Is this build worth testing?

Sanity testing

Narrow and deep

Does this fix work?

Regression testing

Wide and deep

Did anything else break?

Each grid is the same app: its features from left to right, the depth of testing from top to bottom. The lit cells are what each kind of testing looks at.

What is smoke testing?

Definition

Smoke testing is a short set of tests that checks whether a new build is stable enough to be tested at all.

  • It starts the app and touches each main feature once: the app opens, a user can sign in, the main screens load, the key action answers.
  • It does not look at details.

A build that fails its smoke test goes back to the developers at once, and nobody loses a day testing it.

A smoke test is also called a build verification test, because it runs right after a build is made. Its value is its speed: a few minutes, on every build, with a clear yes or no at the end.

What is sanity testing?

Definition

Sanity testing is a quick, narrow check that one fix or one small change works, and that the part of the app around it still behaves sensibly.

  • It runs on a build that brings a bug fix or a small change.
  • It goes deep on that one area and leaves the rest alone.

When it passes, the build is worth the longer regression run.

Done by hand, a sanity check is often improvised: the tester clicks around the fixed area from memory. That is fast, but it leaves no record of what was looked at, and two testers rarely look at the same things.

What is regression testing?

Definition

Regression testing runs the existing tests again after a change, to check that nothing that already worked is broken.

  • It is wide and deep: every important workflow, with its results checked.
  • It is the longest of the three, which is why it runs at night or before a release.

This article keeps it short on purpose.The full guide, with the types of regression testing and what belongs in a suite, is what is regression testing.

Smoke testing vs sanity testing vs regression testing, side by side

What to compareSmoke testingSanity testingRegression testing
The question it asksIs this build worth testing?Does this fix work?Did anything else break?
How wideThe whole appOne feature or one fixThe whole app
How deepShallow: one touch per featureDeep, in one placeDeep, wherever it matters
When it runsOn every buildAfter a fix or a small changeEvery night and before a release
How long it takesA few minutesA few minutesAn hour or more, unattended
What a failure meansThe build is rejectedThe fix goes backA regression is caught before users see it

Smoke testing and sanity testing are the pair people mix up most. The way to tell them apart is what they look at: a smoke test looks at the whole build, a sanity test looks at one change.

Which one to run, and when in a release

The three are not rivals. Each has its moment in a release, and each one decides whether the next is worth running.

Every build

Smoke suite

A few short tests, a few minutes. A failure rejects the build.

After a fix or a small change

Sanity check

The test that covers the fix, run again, with its result checked.

Every night, and before a release

Regression suite

Every test, unattended. The report is read in the morning.

The shorter the test, the more often it runs. The longest one runs while nobody is waiting for it.

The order is always the same.

  • Smoke first: if the build cannot start or sign a user in, nothing else is worth running.
  • Sanity next: if the fix does not work, the build goes back before the long run.
  • Regression last: once the build is stable and the fix is right, everything else is checked.

One week, one app: the three in practice

ExampleAn invoicing app, three days

The numbers below are an illustration, with round figures.

WhenWhat ranWhat it found
Monday, a new build arrivesThe smoke suite: sign in, open a customer, create an invoice, export it. Five tests, four minutes.All pass. The build is worth testing.
Tuesday, a rounding bug in the tax is fixedA sanity check: the invoice test run again, with its total checked.It passes. The fix works.
Tuesday nightThe regression suite: all forty tests.One fails: the total of a credit note is wrong. The fix reached it.

The credit note bug is found on Wednesday morning by a report, not three weeks later by a customer. Without the nightly run, the fix would have shipped with it.

Why regression testing is the first thing skipped

Done by hand, the three compete for the same hours.

  • A smoke pass on every build takes a tester away from real testing several times a day.
  • A sanity check made from memory leaves no trace of what was looked at.
  • A full regression pass takes days, so it shrinks to the screens someone remembers when the release is late.
  • The bugs that come back are in the screens nobody had time for.
Nobody decides to skip regression testing. It gets squeezed out, one late release at a time.

Automating the three with a coded framework such as Selenium or Playwright moves the work to programmers: someone writes the scripts, marks them as smoke or regression in code, and repairs them each time a screen changes.The testers who know the product best cannot touch them.Manual testing vs automated testing looks at that trade in detail.

With recorded tests, the three are the same tests, grouped differently

GUI Test Studio automates end-to-end tests of any app without code. You do a workflow once in your app while GUI Test Runner-Recorder records it, and the test then replays by itself: the same steps, in the same order, at every run.

Once your workflows are recorded tests, smoke, sanity and regression stop being three pieces of work. They are three ways to group and run the same test files.

Smoke suite

3 of 6 tests

Sign inSearch a productCheckoutOrder historyRefundExport invoices

On every build

Sanity check

1 of 6 tests

Sign inSearch a productCheckoutOrder historyRefundExport invoices

After a fix to the checkout

Regression suite

6 of 6 tests

Sign inSearch a productCheckoutOrder historyRefundExport invoices

Every night

Six recorded tests, made once. A smoke suite, a sanity check and a regression suite are three ways to pick among them.
Kind of runWhat it is, with recorded testsHow it runs
Smoke suiteA few short tests, one per main feature.By gtr-cli, on every build.
Sanity checkThe one test that covers the fix, run again.With Run Test, or from the middle of the test.
Regression suiteEvery test you have.By gtr-cli, every night and before a release.

Three things keep the same tests usable in all three roles.

  • Every check waits for what it expects and moves on the moment it is there, so a short test stays short.
  • A group of steps such as Sign in is a Scenario that other tests reuse, so it is recorded once.
  • The GTS Engine finds each element through the app's accessibility information first and by sight when the app says nothing, so a test survives small changes of the screen.

Why a check that waits beats a pause of a fixed length is in fixed sleeps in tests, and Scenarios shows how one test's steps are reused in another.

A sanity check does not have to replay a long test from its start.In GUI Test Studio, put the playhead where the fixed part begins and choose Save & Run From Playhead Position: with the app put in the state it had at that moment, only the steps from there run.Run a test explains it.

How to build a smoke suite and a regression suite from the same tests

The whole setup is six small steps, and the first one is the only one that takes real time.

  1. 1

    Record each workflow once

    Click Record in GUI Test Runner-Recorder and do the workflow in your app, the way a user would. Each workflow that matters becomes one test file, with checks of what the screen showed.

  2. 2

    Share the common steps as a Scenario

    Group the steps every test starts with, such as signing in, into a Scenario named Sign in. Add that test to the others as a Scenario, so the sign-in is recorded once and reused.

  3. 3

    Pick the smoke tests

    Choose one short test per main feature: the app opens, a user signs in, the main screen loads, the key action answers. Keep their test files together in one folder, such as tests\smoke.

  4. 4

    Keep the deeper tests in a second folder

    Put every other test in a folder such as tests\regression. The regression suite is both folders together: every test you have.

  5. 5

    Run smoke on every build and everything at night

    On the build machine, gtr-cli runs the smoke folder after each build and both folders every night. A failed test fails the build, and every run keeps its report, its video and its pictures.

  6. 6

    After a fix, run one test again

    Open the test that covers the fix and click Run Test: that is your sanity check. On a long test, start the run from the playhead in GUI Test Studio, with the app put in the state it had at that moment.

A sanity check is one click on Run Test, and the Runs card keeps the result beside the earlier ones.

Run smoke tests on every build and regression tests every night

On a build machine, gtr-cli runs tests from the command line in any CI/CD tool. It takes test files, folders and patterns, as many as you list. A folder runs the test files that are in it.

The same tests, two moments
gtr-cli run tests\smoke --countdown 0 --junit results\smoke.xml
gtr-cli run tests\smoke tests\regression --countdown 0 --junit results\regression.xml
  • The first line belongs to the build job, and a failed test fails the build through the exit code.
  • The second line belongs to a nightly job.
  • Each run writes a results file your CI tool reads, and keeps its Confidence Report, its video and its pictures.

In the morning, a failed regression test is read, not guessed at: the Confidence Report shows the step that failed, what was expected beside what was found, and the history of earlier runs.How to read a test report shows what to look at first, and Run tests in your pipeline lists every option of gtr-cli.

What to know before you start

LimitWhat it means
A run uses the real screenTests run one after the other on a machine, unless a web test runs without a browser window. Give the nightly regression run a build machine nobody is using.
The same starting pointA test replays from the state it was recorded in. Each suite must put the app back there, and a web test opens its own clean browser window.
A test checks what it was told to checkA green smoke suite says the main features respond. It does not say every detail is right: that is the regression suite's job.

Key takeaways

  • 1Smoke testing is wide and shallow. It decides in minutes whether a build is worth testing.
  • 2Sanity testing is narrow and deep. It confirms one fix before the long run starts.
  • 3Regression testing is wide and deep. It proves the change broke nothing that worked.
  • 4Each has its moment: smoke on every build, sanity after a fix, regression every night and before a release.
  • 5With recorded tests, they are one set of test files. GUI Test Studio groups and runs them three ways, without code.

Questions

What is the difference between smoke testing and sanity testing?

Smoke testing checks the whole build in a shallow way, to decide whether it is stable enough to test. Sanity testing checks one fix or one small change in depth. Smoke comes first, on every build, and sanity follows a build that brings a fix.

What is the difference between smoke testing and regression testing?

Smoke testing touches each main feature once and takes minutes. Regression testing runs every important workflow with its results checked, and can take hours. A smoke test says the build is alive, and a regression test says nothing that worked is broken.

Is sanity testing a part of regression testing?

In practice, yes. A sanity check is a small slice of the regression suite: the tests that cover the area that changed. It runs right after the fix, and the full regression suite runs later, at night or before a release.

Which comes first: smoke, sanity or regression testing?

Smoke testing comes first, on every new build. Sanity testing comes next, when the build brings a fix or a small change. Regression testing comes last, at night or before a release, because it is the longest of the three.

What is a build verification test?

A build verification test is another name for a smoke test. It runs right after a build is made and answers one question: is this build stable enough to be tested further.

How many tests belong in a smoke suite?

Few: one short test per main feature, usually five to ten in all, so the whole suite runs in a few minutes. A smoke suite that gets slow stops being run on every build, and then it protects nothing.

Should a failed smoke test fail the build?

Yes. A build that fails its smoke test is not worth testing further, so the pipeline should stop and the team should be told. With GUI Test Studio, gtr-cli ends with an exit code the pipeline reads, so a failed test fails the build.

Can smoke and regression tests be automated without code?

Yes. With GUI Test Studio you record each workflow once by doing it in your app, and the test replays by itself. The smoke suite and the regression suite are then two groups of the same test files, run by gtr-cli on every build and every night.

Selenium, Playwright and Windows are trademarks of their respective owners.

All articles