Article | Development

How to Measure Developer Productivity in a Multi-Project Environment

Reading time: ~ 4 minutes

I recently had the pleasure of chatting with Kovid Batra from groCTO about implementing engineering metrics for our team at Planet Argon. We discussed my experience with measuring developer experience using the DX tool (more on this later), my exploration of the DORA & the SPACE frameworks, and the challenges we face as an agency implementing standardized, automated metrics collection.

I'll share some of the key takeaways from our discussion below. You can also listen to the full episode here:

Ben Parisot from Planet Argon | DORA & the SPACE framework by groCTO Community

The DORA lab - Episode #03

Read on Substack

The Research

Being an agency and not a product team, getting consistent, automated metrics on our team's work has always been a challenge. Our developers work across multiple repos representing multiple apps for various clients, meaning they're rarely in the same code base or even working in the same development domain week over week. The apples-to-orange nature of our team's work makes collecting and comparing metrics and tracking their work challenging, especially in a fair way.

Despite these challenges, I knew that I needed to implement some baseline productivity metrics to evaluate the team's performance more easily and accurately, celebrate their wins, identify areas for improvement, and ensure we were delivering the highest-quality work for our clients.

I took on the challenge of finding a process that would work for us by exploring the DORA and SPACE metrics frameworks and researching some company-specific metrics measured by top tech companies.

The following is background information from my research.

DORA

DORA (the DevOps Research and Assessment) group was formed to test the following hypothesis: _How a team delivers software directly correlates to how well that team achieves its business goals. _The DORA/Accelerate metrics are:

Deployment frequency: How often you deploy to production.
Change lead time: How long it takes to deploy to production.
Change failure rate: Percentage of changes to production fail.
Mean time to recovery: How long it takes to restore service after a failure.

Though DORA is focused explicitly on metrics related to releases, historically, it's been used to measure the performance of all team members, even if they have nothing to do with DevOps. Currently, DORA is considered an incomplete measure of developer productivity, as most metrics focus on software delivery performance and don't track anything before deployment.

SPACE

The SPACE framework is an evolution of DORA and was intended to capture a more holistic measurement of a developer's experience. It covers the following areas:

Satisfaction and well-being: How fulfilled developers feel with their work, team, tools, or culture, and how healthy and happy they are.
Performance: Best evaluated as outcomes instead of output.
Activity: A count of actions or outputs completed while performing work.
Communication and collaboration: How people and teams communicate and work together.
Efficiency and flow: The ability to complete work or make progress on it with minimal interruptions or delays.

Unlike DORA, the SPACE framework doesn't prescribe specific metrics to be tracked for each area. Instead, it describes each area critical to a productive developer's success and outlines how a manager might measure them. These areas are specific enough to ensure the majority of a developer's workday is being examined but broad enough to allow different types of teams to create metrics frameworks that work best for them.

The Prep Work

Most of the papers and reports I read suggested combining 3-5 metrics from the DORA and SPACE frameworks based on the nature of the development work, the makeup of the team, and the business goals to which the development work contributes.

In order to choose the best metrics for the Planet Argon engineering team, I first had to investigate the nature of our team's work and determine what kind of data set could capture the value we were providing for our clients and the well-being of our developers.

I considered The following questions when researching and choosing the right metrics to measure for our company size, business model, and team makeup.

How do we measure productivity consistently across developers working in different code bases and on different types of tasks?
How do we measure productivity fairly based on the skill and experience level of individual developers?
What metrics measure the impact and value of work being done?
Once metrics are chosen, what actionable steps can a developer take to improve them? Are these metrics actionable?
How do productivity metrics differ from performance metrics?
How much do productivity metrics play into role advancement or performance reviews?
How does measuring developer productivity improve developer experience?

These questions were not easy to answer; in fact, some of them had multiple, conflicting answers! My attempt to answer them involved speaking with our engineering team as a group and individually, researching other companies' existing metrics frameworks, and looking for direct feedback (whether positive or negative) we'd received from clients over the past year.

The Outcome

The following two images list the metrics I selected for our team, each metric's type, the reasoning behind each selection, and the tool(s) we could use to measure and track each.

It's important to note that these metrics may change as the makeup of our team and the needs of our clients change. This, in fact, is an important feature of any good metrics framework – versatility and the willingness to update what's being measured to match the current development and company environment.

metrics table image 1

metrics table image 2

Some of these metrics are quantitative and won't be hard to measure. Others are qualitative and will require more manual collection and analysis of the data for any use. Exactly how we will collect, consolidate, and analyze these disparate metrics is still being researched, but we're confident this new framework will provide a great roadmap for improving our team's productivity, experience, and value to our clients.

A Note on Developer Experience

You might have noticed that while I mentioned developer experience as a consideration in some of these metrics, I did not include any DevEx metrics in this endeavor. That's because, for the past three years, we've been using the wonderful DX tool. This Slack-integrated, data-driven dashboard tracks and summarizes the following self-reported satisfaction metrics: