Who cares about developer productivity anyway?

I have recently come across a lot of talk about how LLMs are increasing developer productivity. Though I agree that LLMs can make the developer’s job easier, not all motion is progress. If companies were to talk honestly about the subject, they would have to admit 3 things:

1. They don’t measure productivity
2. They don’t know what productivity means
3. The bottleneck is not the coding, but the refinement

Companies don’t measure developer productivity

Companies create metrics expecting to measure developer productivity, but they often fail. Let’s take a company that uses a democratic process to estimate effort, for example, poker planning and story point estimation. Developer A consistently completes 15 story points in a 2-week sprint, developer B completes only 8. One would think that developer A is more productive, but have you considered:

• Are tasks assigned randomly?
• Is the nature of the work different?
• Do tasks belong to the same project? Are both projects of equal complexity?
• Do all developers have good knowledge of the project and technologies so that estimations are accurate?
• Is the code written by these developers of the same quality?
• Are developers assigned to other projects or working on other tasks as well?

Consider an even bigger issue. Since they know they are being measured, developers will focus on pushing more story points, instead of creating a better system with a lower level of complexity. This is Goodhart’s law in its full splendor (“When a measure becomes a target, it ceases to be a good measure”).

I imagine we have already learned that counting lines of code is a terrible way to measure developer productivity. Easy to hack with awful consequences for doing so. Given a set of features, less code is usually better than more code. Sometimes deleting code or updating big chunks of structural code may be the best course of action.

Some companies don’t even try to track developer productivity. Others try hard to do so, but they end up with a bad proxy that also makes the codebase worse. Only a few understand that this is a hard subject and focus on removing hurdles for developers.

Nobody knows what productivity means for intellectual work

We know that productivity means output / input. The input for intellectual work is clearly time, but there is no right answer for what the output should be. In the case of software developers, there is no one single output that would satisfy the job requirements. We all want beautiful software that solves all our problems without bugs, but building it requires trade-offs. For example, here are some of the things software developers have to do:

• Develop a system (usually by coding) to solve the user’s problem
• Doing it in a reasonable amount of time
• Follow requirements as outlined by Product (or functional analysts, or clients)
• Reject requirements when incomplete or unfeasible
• Making the software fail as little as possible
• Making the software easy to change
• Communicating with people involved in the project (stakeholders) and providing project updates
• Communicating technical possibilities to stakeholders
• Coordinating the work among multiple software developers and teams
• Increase company’s profit, by increasing revenue or reducing costs. If the company is not a software company, this usually means reducing the time that certain processes take or reducing risk.
• Increasing user happiness

The same thing happens with other white-collar jobs. With so many goals to consider, focusing on a single metric is bound to fail. And some goals are also hard to track. Clearly, it’s impossible to measure how much a single developer contributes to the overall company’s profit, or how hard the software will be to change.

To make matters worse, in complex jobs (such as coding), there is a huge gap between what the average contributor and the best can do. I have a friend who coded the back end, front end, and mobile app for a startup, and also did infrastructure and deployment. Linus Torvalds invented both Linux and Git. Bret Taylor is said to have rebuilt an early version of Google Maps in a weekend, and I am inclined to believe that stories as such are possible. On the opposite side, there are employed programmers who wouldn’t pass the most simple coding tests, just as there is a subset of lawyers who don’t know the basics of their job. And you need an actual license to practice law.

Bottlenecks

In production processes, the speed is determined by the speed at the slowest point, often referred to as the bottleneck of the process. Even if you have a team of staff-level programmers, if you don’t give them the right tools, or make them wait for requirements, or have convoluted QA processes that take away their time, you won’t have a productive team. Individual productivity doesn’t always translate to team productivity, and a lot of productive teams don’t always make a productive company, since coordination can be subpar. If software developers are constantly facing friction, they can’t do their best work.

Some common bottlenecks that can lead strong individual developers to not be as productive as expected include:

• Too much work
• Not enough work
• Lack of clear priorities
• Incomplete requirements
• Incomplete development/test environments
• Broken deployment processes
• No individual ownership or responsibility
• Lack of management buy-in (leadership not giving enough importance to the technical side)

Improving coding speed only translates to shipping features faster if that’s the bottleneck. But the slow part of developing software is the gathering and refinement of requirements, that is, knowing exactly what to build. No one, no matter how talented, can write a perfect specification. Code is the only precise description of the software’s behavior.

Why AI won’t make a big impact to the bottom line

”[…] I’ve had a giant productivity boost in the portion of my job which is typing code at a computer. And I would estimate I am two to three times more productive, faster at turning thoughts into working code, than I was before. But that’s only 10% of my job.”

― Simon Willison, co-creator of the Django web framework

AI won’t be the revolutionary force we are being promised for two main reasons. The first one is that writing code is only a portion of what developers do. Finding and refining requirements, managing complexity, talking to the several stakeholders, and coming up with interesting tactics to get the job done in time are no less important.

The second reason and most important reason is that AI code assistants fail to address essential complexity, that is, the complexity that is present in the real process. In software development, there is a distinction between accidental complexity and essential complexity. Let’s take for example an e-commerce website. The product model has a price field. Whether to use a decimal or an integer is a decision developers must make due to accidental complexity, that is, due to the modeling process. But let’s say that we want to keep track of historic prices, and track who updated the price and when. Then price, instead of being a single field, would have to be another model that relates to the product model, so a product can have multiple prices, only one of which will be active at a given time. That’s essential complexity that is needed for the model (codebase or system) to be useful, not an implementation accident.

Software development is a kind of design problem, meaning that it can’t be perfectly defined without trying to solving it. The only valid approach is iteration. By the time the problem is defined, it is solved. It would be several times easier to develop software if developers knew what to implement with such precision that no doubts about business logic ever arose. But making such a precise description would only be possible in code. Knowing exactly what to build to solve the user’s problem is the hard part of software, and LLMs do little for that part. Sure, LLMs do help with accidental complexity, but only if properly guided by an experienced developer who knows what they are doing. Otherwise, the codebase can quickly become unmanageable.

With all the talk about the importance of increasing developer productivity, you would imagine there is no more low-hanging fruit to pick, yet companies often ignore the basics. Many low-cost, low-effort interventions are never considered. They could improve developer experience by allowing developers to choose their physical working environment and their tools, providing more context about the business and domain, and giving them the time and resources for good software practices (like early refactoring), which quickly pay off.

I would like to be proven wrong, and for LLMs to be a boon to developer productivity and the world of software in general. And I would like to write that “time will tell”, except it won’t. With so little good data on developer productivity, we won’t ever know.