Coding was never the bottleneck.
Brian Houck, Applied Scientist at DX, on where the next wave of AI investment should go.
Yesterday I published a Signal about verification debt and the GetDX longitudinal study - 65% more AI tool usage, ~10% more output:
After the article went live, I sent three questions to Abi Noda (CEO of DX), and he connected me with Brian Houck, Applied Scientist at DX and co-author of the report.
Brian’s answers arrived at midnight. I read them twice. With his permission, sharing here in full.
Brian Houck Applied Scientist at DX
Co-author: https://getdx.com/report/ai-and-engineering-velocity-a-longitudinal-analysis/
Author: https://queue.acm.org/detail.cfm?id=3807964 — ACM Queue
10% is not a disappointment.
My first question was:
Your study shows 65% more AI tool usage but only ~10% improvement in PR throughput. What are engineering leaders actually doing with that finding? Are they slowing AI rollouts?
Brian answered:
I haven’t seen any evidence of organizations slowing AI rollouts (and certainly not because throughput gains are “only” around 10%). Though increasing token costs do appear to be a factor that some organizations are starting to pay closer attention to when looking at overall AI usage.
It’s worth remembering that a sustained 10% increase in throughput across an engineering organization is a very large outcome.
For most companies, achieving that through AI is dramatically less expensive than hiring enough additional developers to produce the same gain.
What we’re seeing is that AI adoption and productivity gains don’t scale linearly. Software engineering is a complex socio-technical system. Writing code is only one component of delivering value; code still has to be reviewed, tested, integrated, deployed, and maintained. As AI accelerates code creation, other constraints become more visible.
In many organizations, engineering teams aren’t slowing AI adoption.
Instead, they’re shifting attention toward the downstream effects and trying to answer the next wave of questions:
Can our review process absorb the increased volume?
Are we creating more rework?
Are we increasing operational complexity?
Are we helping developers make better decisions, or just write code faster?
If the answers are “no,” then begins the harder work of changing the system around the technology.
The organizations seeing the most success tend to treat AI as a system-wide transformation rather than a coding assistant rollout.
The next frontier is coordination, not code.
My next question was:
The coding was never the bottleneck - planning, review and handoffs are. If that’s true, then what should the next wave of AI investment actually target? And who in the org would own that decision?
Brian replied:
I think we’re entering a phase where the highest returns come from attacking coordination costs rather than code generation itself.
While coding may be a bottleneck in some contexts, many organizations are discovering that accelerating code generation simply exposes other constraints that were previously harder to see (but always there).
One example is code review. As AI increases code production, review capacity becomes increasingly important. This is why tools like Meta’s RADAR are interesting—they focus on reducing friction in the review process rather than generating more code (https://arxiv.org/abs/2605.30208).
In my recent EngThrive paper, I detail a case study where Azure SRE Agent dramatically reduced the time developers needed to spend resolving incidents. I expect we’ll see more investment in these types of specialized agents that target friction in non-coding parts of the SDLC (https://arxiv.org/abs/2605.04259).
These are the more obvious next-wave investments, but there are also some less obvious opportunities that I’m interested in watching:
Requirements synthesis and clarification
Knowledge discovery and context retrieval
Design review
Cross-team coordination and handoff support
In other words, helping people understand what to build, why they’re building it, and how their work connects to everyone else’s work.
Personally, I’m interested in improving both sides of the human-agent collaboration. Developer Experience focuses on the human side of the relationship, while Agent Experience approaches the problem from the perspective of the agent (i.e., does the agent have the environment, context, and support needed to do its best work?).
I discussed this in a DX webinar last week.
The ownership question is interesting because these problems sit between traditional organizational boundaries. Engineering owns some of it, but product, platform, developer experience, architecture, and knowledge management teams all have a stake.
The most effective organizations I’ve seen usually have a senior engineering leader sponsoring the effort, but they approach it as an organizational workflow problem rather than an engineering tooling problem.
AI amplifies the system it's introduced into.
And my last question was:
The 10% PR throughput gain - is there any slice of your 400 companies where AI is delivering above that average? what do those have in common? Any patterns that you observed?
Brian’s response:
YES.
There is substantial variation between organizations. Some are seeing much larger gains than the overall average, but even at the far end of the distribution, we are not seeing 10x outcomes (or even 2x outcomes). Abi and I discuss this briefly during our fireside chat at DX Annual:
What’s particularly interesting is that the organizations with the largest gains don’t appear to cluster by company size, geography, industry segment, or many of the other demographic variables people often expect.
The strongest predictor appears to be culture rather than demographics.
The organizations seeing the largest gains tend to have visible executive support, active enablement programs, strong peer learning networks, and a culture that encourages experimentation rather than passive tool adoption. I discuss some of these characteristics in my SPACE of AI article (https://queue.acm.org/detail.cfm?id=3807964).
One pattern I keep coming back to is that AI appears to amplify the characteristics of the system it’s introduced into. Healthy organizations often improve noticeably, while unhealthy organizations frequently discover their bottlenecks faster.
This is one reason I’m increasingly interested in concepts like Developer Experience, Agent Experience, and context quality. The next frontier may not be making models smarter. It may be creating environments where humans and AI can collaborate effectively because the underlying context, workflows, and organizational systems are healthy.
Thanks again for the questions!
That last answer is the one I’ll be thinking about for a while.
Brian - thank you.
This is exactly the kind of primary-source depth that must not be lost in the noise.
The piece argues coding was never the bottleneck - that the real frontier is coordination, review, handoffs. I half agree. But framing it as "coding vs. coordination" lets senior leaders off the hook for the harder diagnosis.
Here is the precise mechanism: AI didn't reveal a coordination problem. It revealed a *judgment* problem. Who decides what to build. Who owns the review call. Who closes the loop between agent output and a real decision.
Coordination is the symptom.
Unowned judgment under pressure is the cause.
Fix the org chart of decisions, not the workflow diagram.