What is Engineering Effectiveness?

Software Engineering is hard to reduce to well-known patterns of management.

Most industries have followed recognizable evolutions: from nascent technologies to empirical methods of management to fully scalable operations. The reason this happens over and over again is the pursuit of economic growth, which is inherently tied to making profits.

Software companies are subject to the same economic forces and they are currently transitioning from empirical notions of productivity to a more scientific approach. This is mainly driven by the fact that almost every company is becoming a software company.

There remains a challenge though: how do we define productivity in software?

Software is different

In the classic sense, productivity is the ratio of outputs to inputs: people try to make and sell more of the same thing at a cheaper cost to themselves. However, measuring the output of a development organization is notoriously hard.

For example, the idea of measuring lines of code as a productivity proxy has long been identified as a dead-end. Attempts such as measuring agile sprint points or “number of tickets closed” have never really taken hold. Any engineer knows intuitively why this approach has failed:

  • Computer science relies on symbolic representations. The number of symbols used to describe a complex algorithm bears little correlation to the value it creates.
  • Software engineering is a social activity involving many engineers and teams. Human beings don’t like being measured as if they were widgets in a factory, and employers are not in a position to ignore these feelings because of the general penury of software engineers.

As former engineers ourselves, we also think it is counter-productive to try and measure the output of engineers in this way.

Focusing on inputs

This difficulty has led some engineering leaders to give up on trying to understand productivity:

  • In some cases, productivity is deliberately ignored or replaced with qualitative frameworks such as “OKR completion rate”.
  • In other cases, “working on the right things” is presented as being superior to improving productivity, which is a false dichotomy.

The unfortunate consequence is that such organizations unfairly reward story-telling: missing deadlines or moving more slowly than the competition can always be explained away. As a result, the organization becomes more political and the best employees leave for greener pastures.

This handbook instead focuses on the other side of the productivity equation: inputs. Taking into account the realities of software development, we define inputs as anything that affects the workflows of engineers:

  • How much available time do individuals have to think and code?
  • How good is the suite of developer tools?
  • How long does it take to get code reviewed by a peer?
  • How easy is it to access documentation?
  • How often do engineers receive pages outside of business hours, interrupting their sleep or family life?

In software engineering, inputs are easier to measure and more actionable than outputs - they should be the focus of productivity. To further illustrate this point, let's look at side-by-side examples:

Measure ... (INPUT METRIC)Don't measure ... (OUTPUT METRIC)
The time it takes to do code reviewsThe throughput of pull requests
The amount of interruptions affecting engineersHow many hours engineers have been working
The amount of unplanned work affecting a sprintSprint points burned by the team

In the rest of this handbook, we will thus define engineering effectiveness as the optimization of inputs, ie the removal of bottlenecks and inefficiencies in the life of an engineering team.

For simplicity, we will also use the words productivity and effectiveness interchangeably.

Edit this page on GitHub Updated at Tue, May 24, 2022