TestingBudget

Run

Failures per run

Everything a person has to look at after a run: defects found plus tests the product broke plus flaky failures — derived, never typed.

Unit
count per run, derived
Where you enter it
Computed; shown on the Run tab.

What it means

The number of failing cases a run produces, and therefore the number a person must triage. It is never entered, because an evaluator comparing tools cannot know it per tool. It is derived from three things the model does know: the defects a cycle finds (an application figure), the tests broken by product change (the annual breakage rate spread over the year's runs), and the tool's flaky-failure rate.

For a manual approach only the defects count: manual cases neither break nor flake, and the tester decides what a failure means in the moment rather than triaging afterwards.

Why it matters to the cost

Failures are the dominant per-run human cost of automation. In the default scenario a tool sees around 5 defects + 5 broken tests + 6 flakes = 16 failures a run, and at twenty minutes each that is over five hours of triage per run before anything is fixed. The breakdown shows where to push: flakes are the tool's problem, breakage is the product's.

Typical values

Ten to twenty per run for a 300-case suite on a monthly cadence; hover the figure in the calculator to see its breakdown.

Typical values describe a category of tool — AI-native, codeless, script-based, homegrown, manual — never a named product. Why we do not price named products.

Where it appears in the model

Run — every release

tests broken per run
Total test cases × Test breakage rate ÷ Runs per year — found here, repaired under Maintain
flaky per run
Total test cases × Flaky failures per run
Failures per run
Defects found per run + tests broken per run + flaky per run — derived, never asked
Run effort
Prepare and start a run + Failures per run × Triage one failure + Defects found per run × Write up one defect + Report and sign off — an automation tool
Run effort
Prepare and start a run + Total test cases × Execution time per case + Defects found per run × Write up one defect + Report and sign off — a manual approach — the tester is the run
Machine time per run
Total test cases × Execution time per case — automation only; duration, not labour
Run cost per year
Run effort per run × Runs per year × Effective hourly rate + Token cost

Every name is a link to its definition. See the whole model.

Try it with your numbers

Failures per run is a row in the calculator. Change it and every total on the page moves with it — free, in your browser, nothing to install.

See how failures per run impacts the calculation of test automation costs

Related terms

  • False positives per run Test failures that are not product defects — tests the product broke plus flaky failures. The model counts both, on separate rows, because they have different owners.
  • Defect validation The human work of proving that a failed test is a real product defect and getting it in front of a developer — priced as triage per failure plus write-up per defect.
  • Defects found per run How many real bugs a regression cycle typically finds — a property of your product, entered once, and the only "failures" figure you are asked for.
  • Test breakage rate The share of the suite that the product breaks in a year — changed screens, renamed fields, altered flows — and so needs repairing.
  • Flaky failures per run The share of cases that fail for no product reason — timing, environment, brittle selectors — and still have to be investigated.
  • Triage one failure The time to look at one failing test and decide what it is: a real defect, a broken test, an environment problem, or a flake.

Last reviewed 2026-09-15. All terms.