Take one test you’re thinking about automating. A checkout regression that takes 15 minutes to run by hand, which you run on 200 builds a year. That’s 50 hours of somebody’s year. Eight hours to automate it and five hours a year to keep it working recovers the build cost in just over two months, and it keeps paying every year after.
Now take a different one. A quarterly admin report check, 20 minutes, run four times a year. That’s 1.3 hours a year of manual work. Six hours to automate it against two hours a year of upkeep never reaches break-even at any horizon, because the annual maintenance alone costs more than doing the thing by hand.
Same team, same tooling, opposite answers. The variable that decided it was frequency, and it’s the one vendors ask about least. Nobody asks how often you run it.
We build and maintain automated suites as part of delivery at gmware, and we’ve inherited enough abandoned ones from other teams to have opinions about how they died. Framework choice almost never turns up in the post-mortem. What turns up is a backlog full of the second kind of test. If you’re working out what testing costs generally, rates and engagement models are a separate question; this page is about which tests deserve to exist before anyone quotes you for building them.
Automated tests are liabilities that pay rent
The mental model that causes the trouble is treating an automated test as something you build once and own. It behaves more like a subscription you pay in engineering time.
Every automated test sits against a product that changes. When the product changes, enough of the suite breaks that somebody has to determine whether each failure came from a brittle test or from genuinely wrong code. That triage is the recurring cost, and it appears in no proposal anywhere.
Which gives you the rule that, in our experience, predicts survival better than any coverage metric:
A suite dies when the annual cost of maintaining it exceeds the manual testing it replaced.
You can’t know your maintenance cost precisely in advance. You can bound it well enough to decide, and you should do that per test rather than per suite, because “should we automate?” has no answer while “should we automate this?” has one you can compute.
The four numbers
Start with M, the minutes to run it manually and honestly, including setup and checking the result. Then F, how many times a year you actually run it, which is different from how many times you imagine running it. Then A, the hours to automate it to the point of being reliable rather than to the point of passing once. And finally X, the hours a year you’ll spend fixing it when the product moves.
Manual cost per year is M × F ÷ 60 hours. Automation costs A + X in year one and X every year after. The comparison is that simple, and doing it on a whiteboard for ten candidate tests takes an afternoon and routinely eliminates half of them.
Annual hours: manual against upkeep
Two of the four inputs get systematically distorted, and knowing which helps.
Teams overestimate F for anything not already on the every-commit path, because the aspirational run frequency and the real one diverge fast once the suite gets slow. And in the suites we’ve inherited, X was consistently the number nobody had estimated at all, because it stays invisible until the first big refactor lands. If you’ve never measured it, take the vendor’s number and double it, then check whether the answer changes. A test that only clears the bar at the vendor’s estimate is a marginal test.
The remaining input worth interrogating is stability, which is really a question about your roadmap rather than your code. Automating against a screen the design team is rebuilding next quarter means writing the test twice and paying for both. Ask what’s scheduled to change, not what changed last quarter.
What actually breaks when you automate the wrong thing
The damage a badly chosen test does to its own build hours is trivial next to what it does to the suite around it.
A test automated against an unstable interface fails often and usually for reasons that aren’t defects. Its failures teach the team that red doesn’t necessarily mean broken. Once that lesson lands, the team starts skimming failures instead of reading them, and the genuinely valuable tests in the same run lose their signal too. A handful of consistently noisy tests can neutralise several hundred good ones, which is why deleting tests is so often the highest-value work on an inherited suite.
The second failure is subtler. A test that duplicates human judgement, checking that a layout “looks right” or that a flow “feels wrong”, produces assertions nobody can maintain, because the thing being asserted was never crisply defined. Six months later nobody can say what the test was protecting, so nobody can safely change it or delete it, and it sits in the suite accumulating maintenance.
The layer question is a stability question
Where a test sits determines what it costs to keep.
Unit tests are fast, cheap to maintain, and break only when the logic they cover changes, which is usually correct behaviour. They should be the bulk of your automation and they largely belong to the developers writing the code rather than to a QA vendor.
API and integration tests are the best value in most engagements. They exercise real behaviour across service boundaries, run in seconds, and don’t break because a button moved.
UI and end-to-end tests are the most compelling to a stakeholder and the most expensive to own. They break on layout changes, they’re the main source of flakiness, and they’re slow. Keep them scoped to the flows where a regression actually costs revenue or trust, which for most products is a handful rather than a hundred. An inherited suite that’s mostly UI tests is the strongest warning sign we see, and if a vendor proposes one, ask why. The usual answer is that UI tests demo better.
What to ask before you sign
“Who maintains this in month nine?” The most important question and the one most proposals dodge. If the answer is your team, they need to be in the build from week one, or you’re receiving a codebase nobody in the building understands.
“How many tests, at which layers?” A proposal quoting a test count without a layer split is quoting volume, and volume is precisely what you don’t want. Pair it with a question about flakiness, because a serious vendor has a flake budget with a number and a policy attached: quarantine, fix within N days, delete if unfixable. A vendor who hasn’t thought about flakiness will hand you a suite the team stops trusting inside a year.
“What do you recommend we do not automate?” The most revealing question on the list, and the one a vendor selling hours has no reason to answer well. Someone who names three specific things you should keep manual has run this before and watched it fail. Ask alongside it whether the suite runs in your CI, on your infrastructure, from day one, because tests that only run on the vendor’s machine aren’t integrated, and integrating them later is a second project at a second price.
The engagement shape worth asking for
For most teams below a few hundred engineers, the useful split is sequencing rather than a straight choice between hiring a vendor and building in-house.
Buy the build. Framework selection, CI wiring and the first tranche of critical-path tests are well-scoped work with a finish line, and an experienced team does it faster and with fewer wrong turns than a team learning as it goes. Have your own engineers pair on it so ownership transfers with the code rather than after it.
Own the upkeep. Maintenance needs somebody who knows what changed this sprint and why, which your engineers know and a vendor learns expensively.
Then keep the suite small enough to stay green, because a team that trusts 200 tests ships faster than a team ignoring 2,000. If you’ve inherited a large suite, run the four numbers across it before you spend anything more on it. Start with the tests that failed in the last quarter without finding a real defect, because that list is usually shorter to fix by deletion than by repair, and our product development practice will run that assessment with you if you want a second read on it.