AI adoption vs AI impact in software teams
Usage is not results. This guide covers why the gap exists, how to recognise adoption theatre, and how to measure whether AI is reaching your P&L.
Adoption is usage: licences active, prompts sent, suggestions accepted, developers reporting they "use AI daily". Impact is movement in metrics the business runs on — margin, delivery predictability, quality, cost to deliver — measured against a baseline. The uncomfortable finding across the industry: the first number is easy to grow and the second one usually isn't, and vendors only report the first.
Why the gap exists
Three mechanisms, all of them structural rather than attitudinal:
Time saved gets reabsorbed. Writing code faster shifts the constraint to reviewing, testing and integrating it. If those stages haven't changed, the system delivers at the speed of its slowest stage — same as before, with busier upstream stages.
New costs appear that nobody baselines. Review burden on senior engineers grows, and plausible-but-wrong output creates rework. Both are real costs of adoption; both are invisible if you only track usage. The mechanism is the same one behind AI-driven technical debt.
Local gains don't equal system gains. A developer being 30% faster at one activity does not make the team 30% faster at delivering value — bottlenecks, queues and dependencies decide that. Software delivery is a system; AI accelerates parts of it.
The signals of adoption theatre
How to measure AI impact — and ROI
Baseline first, independently of the tools: rework rate, review time per change, cycle time, delivery predictability and cost per delivered unit. Then let real usage run for 60–90 days and compare. ROI is total cost of adoption — licences, enablement, and the added review burden — against measured gains in those delivery economics.
One evidence hierarchy worth internalising: measured delivery metrics > repository-level data > self-reported time savings. Surveys of how much time developers feel they save are the weakest class of evidence — they measure perception, and perception has never paid an invoice.
What turns adoption into impact
The pattern that works is boring and specific: a baseline, one workflow redesigned end to end with explicit guardrails, measured against that baseline, then scaled only if it won. That's the entire logic of our Pilot Sprint — including the commitment that if the pilot doesn't beat the baseline, we recommend not scaling, in writing. Impact is a property of the operating model, not of the licence count.
If you want a fast read on where your team sits today, the 10-minute Scorecard returns your AI delivery maturity level and your top profitability leak immediately.
Related questions
Compare rework, review time, cycle time, predictability and cost per delivered unit against a pre-adoption baseline. No movement after 60–90 days of real usage means activity, not productivity — regardless of how it feels.
The mechanics are universal; the stakes differ. In a 10–100 engineer NZ or Australian company, senior review capacity is scarce and margins are watched closely — which makes unmeasured adoption more expensive and disciplined adoption more valuable than in a large enterprise that can absorb waste.
No — pausing loses learning and momentum. Keep using the tools; add a baseline and guardrails around one real workflow now, and expand from evidence. Measurement is something you add to adoption, not a gate in front of it.
Nicolás Espinosa — Founder, Optimum Agile. Former Director of Product at PikPok and Senior Delivery Manager at Trade Me; postgraduate specialisation in AI Product Management (Duke University). Based in Auckland, working with NZ and Australian software teams.