Every test automation proposal you receive will quote a build. In the proposals we see, very few quote the part that actually determines whether the investment pays, which is what it costs to keep the suite green for the next three years.
That omission is why so many automation projects are quietly abandoned. The build usually works. The suite decays, the failures get ignored, and someone concludes automation does not work here. What actually happened is that a two-part cost was approved as a one-part cost.
If you are still deciding whether to buy testing as a service at all, our overview of software testing services and what they cost covers the rate card and the manual-versus-automated split. This post is about the line that comes after the invoice.
The two-part cost of test automation
We’re gmware, a software development firm headquartered in Austin, TX with engineering centers in Bangalore and Mohali, India. We build and run production systems ourselves, including Shield Suite, a retail-intelligence product covering 60,000+ beverage-alcohol storefronts, so the maintenance line in this post is one we pay too.
Read the quote as two numbers
Build cost is engineering time: designing the framework, standing up test data and environments, writing the tests, wiring the whole thing into CI so it runs without a human. It is estimable. Test count and application complexity get you close, and any competent vendor can defend the estimate line by line.
Maintenance cost is the recurring engineering time to keep those tests passing as the application changes. Every renamed field, restructured screen, changed API response, and new authentication step breaks something. Somebody fixes it, every sprint, indefinitely.
Maintenance is the number that decides the outcome, and it is the number most proposals omit. So make it a procurement requirement: ask the vendor to state expected annual maintenance as a percentage of build effort, in writing, and to say what assumptions that figure depends on. The right answer references your rate of change, not an industry average. A vendor who has never operated a suite past its first year will not have a figure. That is useful information.
It is worth being precise about who pays that line, too. If the vendor built the suite and you own the maintenance, you have bought an asset with an unfunded liability attached. If the vendor maintains it, you have bought a subscription, and the contract should say what happens to the tests when the relationship ends. Neither arrangement is wrong. Discovering which one you signed a year later is.
The break-even, as arithmetic
Strip away the debate and automation is a straightforward trade: you spend engineering hours once, plus some every year, to stop spending tester hours repeatedly.
A test is worth automating when:
(manual execution time saved × runs per year) > (annualized build cost + annual maintenance cost)
Work an example with your own numbers rather than borrowed ones. Take a regression case that costs 20 minutes to run by hand. Run it once per release, with monthly releases, and you save four hours a year. Against any realistic build-plus-maintenance cost, that test loses, badly. Run the same case on every merge, twenty times a month, and you save roughly eighty hours a year, and it wins comfortably.
Same test, same build cost. The only variable that changed was frequency, and it flipped the answer entirely.
This is why “how much does automation cost” is the wrong opening question. The useful question is which tests clear this bar in our situation, and the answer is usually a smaller and more specific set than the sweeping coverage a proposal will offer you. Two properties decide it:
- Execution frequency. Anything gated on every merge accumulates value fast. Anything run quarterly almost never does.
- Stability of the target. A test against an interface that changes every sprint spends its life being repaired. The saving is real and the maintenance eats it.
High frequency plus high stability is the zone where automation is close to unarguable. Low frequency plus high churn is where these projects go to die, and it is exactly where enthusiastic first attempts tend to start, because those features are the ones everyone is currently arguing about.
There is a second-order effect worth naming. The formula above treats maintenance as a constant, and it is not. A suite that grows without pruning raises its own maintenance line every quarter, so tests written when the break-even was comfortable can quietly fall below it as the total upkeep bill climbs. That is the mechanism behind suites that felt like a good idea when they were built and feel like a tax a few years later. The fix is not to write fewer tests. It is to re-run the arithmetic on the suite you already have, at least annually, and retire what no longer clears the bar.
Layer order changes the cost more than tool choice
Teams spend weeks choosing a framework and minutes deciding what layer to test at. The second decision has far more effect on the bill.
API and integration tests are the value center. They run orders of magnitude faster than UI tests, they do not care what the screen looks like, and they cover the business logic where real regressions live. They break when the contract or the behavior changes, not when the screen does, which is precisely why they are cheapest to build and cheapest to keep.
UI end-to-end tests are the most expensive tests you will ever own, on every axis. Slowest to run, most sensitive to cosmetic change, and the most flake-prone layer in the suite. They are still worth having, because some failures only appear when the whole system is wired together. The discipline is to keep them few and deliberately chosen: the handful of journeys where failure means lost revenue or a support queue. Sign-up, checkout, login, the core workflow. Not a broad sweep of every screen.
Getting this order wrong is the most expensive mistake available. A suite that is mostly UI tests over a changing product will consume its maintenance budget and produce failures nobody trusts, which is the precondition for abandonment.
The practical test when reviewing a proposal is to ask for the test count broken down by layer. A plan that is heavy at the UI level needs a specific defense, and “we wanted realistic end-to-end coverage” is not one, because the realism is exactly what you are paying for in maintenance.
Flakiness is a cost line
A test that fails intermittently is worse than no test. It consumes triage time, trains the team to ignore red, and by doing so destroys the value of every reliable test sitting next to it. Once a team routinely re-runs the build to get green, the suite has stopped being a quality signal and become a tax.
Budget for this explicitly. Flaky tests get fixed or deleted, not retried, and someone owns that call. It is a process commitment rather than a tooling one, and no framework choice will substitute for it.
The reason this belongs in a cost discussion rather than a quality one is that flakiness compounds. Each unreliable test adds triage time to every run, and it also erodes the response to genuine failures, which is the thing you bought. A suite at ninety-five percent reliability across a hundred tests, if the failures are independent, fails somewhere on almost every run. That is enough to make “just re-run it” the rational local choice, and once that habit is established the suite’s signal is gone while its maintenance bill continues.
What automation does not buy you
It does not replace testers. It replaces the specific activity of re-running known scripts by hand.
Automation verifies that things which used to work still work. That is regression insurance, and it is genuinely valuable, because it is the work humans do worst: repetitive, attention-dependent, and boring by the fortieth repetition. What it cannot do is explore, exercise judgment about whether a feature is coherent, or catch the problem nobody thought to write a test for. Those remain human work, and freeing people to do more of it is the actual return.
Any business case built on headcount reduction is fragile for this reason. The defensible case is faster release cadence with fewer regressions reaching production, and a testing team spending its time on work that only humans can do.
Scoping questions worth asking
- Quote the build and the maintenance separately. Refuse a single number.
- What annual maintenance percentage do you assume, and on what basis? Get it in writing.
- What ratio of API to UI tests are you proposing, and why? UI-heavy proposals need a defense.
- Who owns a failure at 2am? Automation running in CI creates an on-call surface.
- How will we know this is paying off in six months? Agree the measure before you start.
- What is the plan for retiring tests? Suites that only grow eventually collapse under maintenance.
- Who owns the suite if we stop working together? The tests, the CI wiring, and the knowledge to change them.
A vendor with good answers to all seven has usually run a suite through its second year. A vendor who answers with coverage percentages and a tool comparison has run a pilot. The difference will show up in your maintenance line long after the build invoice is paid.
How gmware sizes an automation program
We start by pricing the maintenance, not the build, because it is the number that decides whether the program survives. That means looking at how often your product changes and where, deciding the API-to-UI split from that rather than from a template, and being explicit about which candidate tests do not clear the break-even so you are not paying to maintain them.
The work sits inside our product development practice, where QA is planned with the feature rather than bolted on before launch, and we fold it into digital transformation engagements where the release cadence is the thing actually being changed. Reach out with how often you ship and where quality currently bites, and we will tell you which tests are worth automating and which are not.
If you are weighing this against building the capability internally, the trade-offs are the same ones covered in fixed price versus time and materials and staff augmentation versus a dedicated team: who carries the recurring cost, and who is accountable when it grows.