TestingBudget

Run

False positives per run

Test failures that are not product defects — tests the product broke plus flaky failures. The model counts both, on separate rows, because they have different owners.

Also called: false alarms · spurious failures · non-defect failures · noise

Unit
count per run, derived
Where you enter it
Not entered. Read the Run tab's failures-per-run breakdown; the two halves come from the flaky rate (Run) and the breakage rate (Maintain).

What it means

A false positive is a test that fails when there is nothing wrong with the product. Every one still costs a person the time to look at it and prove it is not a bug, which is why the phrase turns up in every vendor demo: "eliminating false positives" is the maintenance saving self-healing and AI tools are selling.

The model does not have a single false-positive row, deliberately. False positives are two different things with two different owners: tests the product broke (deterministic — the screen changed and the test did not; the product team's doing, priced through the test breakage rate) and flaky failures (non-deterministic — timing, environment, brittle locators; the tool's doing, priced through the flaky rate). Together with the real defects they make up the failures a run produces.

Be aware the phrase is occasionally used the opposite way, borrowing the statistical sense: a test that passes while a bug is present. In test-automation practice, and on this site, a false positive is the spurious failure.

Why it matters to the cost

In the default scenario a 300-case suite on a monthly cadence produces about 5 broken tests and 6 flakes a run against 5 real defects — so two failures in three are false positives, and each costs the same triage time as a real one until it has been ruled out. Halving the flake rate and the breakage rate is worth more triage time than any execution speed-up, and separating the two tells you whether the fix is a better tool or a more stable test environment.

Typical values

Ten to twelve per run for a script-based tool on the default suite; three to five for an AI-native tool with self-healing; none for manual execution, where the tester decides in the moment.

Typical values describe a category of tool — AI-native, codeless, script-based, homegrown, manual — never a named product. Why we do not price named products.

The formulas it lives in

Run — every release

tests broken per run
Total test cases × Test breakage rate ÷ Runs per year — found here, repaired under Maintain
flaky per run
Total test cases × Flaky failures per run
Failures per run
Defects found per run + tests broken per run + flaky per run — derived, never asked
Run effort
Prepare and start a run + Failures per run × Triage one failure + Defects found per run × Write up one defect + Report and sign off — an automation tool
Run effort
Prepare and start a run + Total test cases × Execution time per case + Defects found per run × Write up one defect + Report and sign off — a manual approach — the tester is the run
Machine time per run
Total test cases × Execution time per case — automation only; duration, not labour
Run cost per year
Run effort per run × Runs per year × Effective hourly rate + Token cost

Every name is a link to its definition. See the whole model.

Try it with your numbers

The calculator works false positives per run out from the rows it is built on. Change those and every total on the page moves with them — free, in your browser, nothing to install.

See how false positives per run impacts the calculation of test automation costs

Related terms

  • Flaky failures per run The share of cases that fail for no product reason — timing, environment, brittle selectors — and still have to be investigated.
  • Test breakage rate The share of the suite that the product breaks in a year — changed screens, renamed fields, altered flows — and so needs repairing.
  • Failures per run Everything a person has to look at after a run: defects found plus tests the product broke plus flaky failures — derived, never typed.
  • Defect validation The human work of proving that a failed test is a real product defect and getting it in front of a developer — priced as triage per failure plus write-up per defect.
  • Triage one failure The time to look at one failing test and decide what it is: a real defect, a broken test, an environment problem, or a flake.

Last reviewed 2026-09-15. All terms.