Over the last decade, my cofounder and I have engaged in hundreds of conversations with engineering leaders on our journey to improve engineering. We talked with Silicon Valley startups just getting started as well as established enterprises with thousand-person headcounts.
What we discovered surprised us: Not only can software engineering effectiveness be measured; most engineering teams follow the exact same evolution.
From a two-person startup to a thousand-person team, engineering success always comes from the same fundamentals: 1) Hire excellent engineers and 2) Remove the blockers that prevent their success.
Different team sizes, however, change the ways these fundamentals manifest. Broadly, they fit into five stages:
- Qualitative
- Data-Curious
- People-Driven Engineering Effectiveness
- Software-Driven Effectiveness
- Engineering Effectiveness as a Strategic Advantage
While you're perusing these stages, keep in mind that every engineering org starts somewhere and every org can reach stage 5. Regardless of the size or age of your team, your engineering effectiveness would benefit from:
- Objectively assessing your current engineering org: What stage are you in and why?
- Solidifying a strong foundation: Ensure you've built all the components of your current and lower stages.
- Aiming higher: Add advanced functions to accelerate your achievement.
Of course, individual organizations have their own unique traits, so you should feel free to make this framework your own. On the whole, however, here's how to grow:
Stage 1: Qualitative
Most engineering teams start small. At this point (usually around 1-20 people), the team is evolving rapidly and there aren't many objective metrics to measure success.
In stage 1, the most successful engineering organizations build a strong social, cultural, and behavioral foundation. This foundation should include implementing widely-known best practices like:
- Sprint retrospectives to identify patterns of effectiveness
- Test-driven development and version control with small, frequent commits
- Flexible hours with high employee autonomy
With stage 1 being almost metric-free, engineering success relies almost entirely on your first-line managers. We've found these managers to be the differentiators between successful stage 1 orgs and those that flounder. If your org is in stage 1, be on the lookout for strong managers: ones who are either experienced and well-trained or inexperienced but exceptionally fast learners.
To support a stage 1 engineering org, company leadership should create a high-trust environment in team meetings/all hands (to encourage feedback) and keep a pulse on morale through high-quality one-on-ones. It's also helpful for long-term culture to invest in manager growth and development, particularly around people skills and EQ. (We've found Google's framework to be particularly effective at helping managers become servant leaders.)
In stage 1, engineering revolves around culture: you'll succeed if your managers can support and align your team.
Stage 2: Data-Curious and Reactive
Stage 2 is when most orgs first become aware of engineering effectiveness as an area to improve, so they start dabbling in data.
As most stage 2 orgs are small and fast-growing (20-50 people), they must still rely heavily on first-line managers for qualitative measures like:
- OKR completion
- Team engagement
- Hiring targets
Stage 2 will also bring your first quantitative measures, typically in the form of ad hoc metrics like:
- Bug counts
- Alert rates
- Employee satisfaction surveys
On the dev ops side, successful stage 2 orgs typically begin investing in a pipeline aimed at achieving continuous delivery. These investments often include custom scripts or open-source tools that provide snapshots of their DORA metrics.
These instantaneous measurements and metrics help make stage 2 engineering orgs more effective than those in stage 1. That said, these stage 2 improvements are typically ad hoc, one-off, and isolated. Instead of a comprehensive, real-time dashboard or even a long-term feedback loop, stage 2 orgs act on instantaneous information as it arises, necessarily making the org's improvements reactive.
Stage 3: People-Driven Engineering Effectiveness
The typical stage 3 engineering org will have 50-250 engineers and feel like it's in a transition stage between independent qualitative assessments and fully-automated metrics.
In stage 3, the best engineering teams establish a dedicated function for engineering effectiveness, most frequently through a dedicated internal team or by hiring an engineering chief of staff. This dedicated function will begin to make:
- A regular cadence of investments in dev tech, processes, and tooling
- Looker dashboards to capture DORA metrics in real time
- SLAs/SLOs for engineering effectiveness
- Tangible, objectively-verifiable effectiveness metrics that will hold managers accountable
While stage 3 brings both long-term metrics and systems that provide a holistic view, most of these activities will be performed by hand. For example, the engineering Chief of Staff or program manager might ask directors to fill out a spreadsheet, which will then evolve into a powerpoint presentation that prompts the VP to make adjustments.
A successful stage 3 engineering org will implement a dedicated engineering effectiveness team to measure metrics and incorporate adjustments at a reliable cadence.
Stage 4: Software-Driven Effectiveness
In stage 4, dedicated software enters the picture, bringing continuous improvement to every engineering stage.
The typical stage 4 engineering org will have hundreds or thousands of engineers. At this scale, treating engineering like a black box is no longer acceptable. Instead, managers will require precise, quantitative assessments instead of qualitative or imprecise measures.
The most successful stage 4 teams automate a high variety of engineering metrics, either through dashboards built by a full-time engineering effectiveness team or by leveraging third-party software
These dashboards gather actionable, real-time effectiveness metrics at every level of management:
- The CTO sets high-level metrics and goals, typically through a dashboard they share with the other executives.
- Directors/mid-level managers set goals for their sub-orgs and monitor their metrics for early signs of issues.
- First-line managers provide root-cause analysis on the specific factors contributing to high-level metrics.
- Every engineering OKR includes goals around effectiveness and improvement.
Before stage 4, engineering teams often aim to measure everything. In stage 4, they aim more precisely at high-value metrics like engineer utilization, blockers, and work-life balance. Stage 4 brings the ability for an org to diagnose specific symptoms all the way down to their root causes, where they can form coherent, data-backed stories that inform pinpointed improvements. The most successful stage 4 teams will even start uncovering personalized metrics that they find particularly correlate to their success.
By combining automated, real-time metrics with the culture of continuous improvement built in stages 1-3, stage 4 teams can evolve their effectiveness into an always-running, well-oiled improvement machine.
Stage 5: Engineering Effectiveness as a Strategic Advantage
Stage 5 turns engineering effectiveness into a strategic lever that helps the company achieve precise business goals.
Even though a typical stage 5 team will contain hundreds or thousands of engineers all around the world, the best stage 5 orgs run like a well-coordinated symphony: individual contributions come together to create a single unit that's much more than the sum of its parts.
On top of stage 4's software-based measurement and org-wide culture of improvement, stage 5 adds a strategic lens. The best stage 5 organizations can make calculated risks involving conscious trade-offs. Perhaps the org extrapolates a concerning quality trend and adjusts its features long before engineers or customers start to complain. The best stage 5 organizations can calculate specific risk levels and readjust without unpleasant surprises.
High-performing stage 5 teams typically engage in:
- Industry/peer-group benchmarking (to understand their effectiveness compared to other companies)
- Automatic implementation of the latest effectiveness research (to accelerate constant improvement)
- Anticipatory activities at every stage in the org (to predict potential problems before they arise)
- Thought leadership on new best practices of engineering effectiveness (naturally uncovered as a result of their experience)
- Full transparency/understanding of engineering metrics, even outside of the engineering org (to aid company-wide improvement)
- Calculated risks (to achieve precise business aims)
From the outside, a stage 5 team looks like a strong engineering brand. It can accelerate and adjust, attract top talent while achieving business aims.
Toward Engineering Effectiveness
Engineering effectiveness is too expensive to be left to chance. Start by assessing where your team currently is. Are you:
- Sufficiently small that metrics are still a "nice to have"?
- Data-curious and ready to react?
- In need of a dedicated effectiveness team?
- Positioned to produce a continuously improving organization ?
- Able to precisely tune your priorities to enable the company's long-term strategy?
Then, solidify your position at your current stage, building a solid foundation on which you can expand.
While every engineering team has its own individual nuance, most will follow this consistent evolution. We uncovered these stages through years of observation, but there's still more improvement to be applied to engineering.