The 2 key metrics delivery managers should care about first
Management thinker Peter Drucker is often quoted as saying, “you can’t manage what you can’t measure.” Managing the delivery of digital products is no exception to this rule. So, the big question for delivery managers is what measures are most useful to manage teams towards efficient delivery of valuable business outcomes.
Before we get to the answer to this question, let’s take a look at the metrics in the context of modern management of product development.
Management of product development
Managing products instead of projects
At Emergn we know the best way to manage products is to follow and manage the product lifecycle. That means not using projects as the model for managing delivery, and not managing the software development lifecycle separately from the product or service the software is used for.
Principles for managing product development
We live and teach the fundamental principles of Value, Flow and Quality, which underpin all successful product development. In short, the best way to manage product development is to deliver Value early and often, optimize the Flow of work end-to-end, and discover Quality through fast feedback.
Product Management in combination with Delivery Management is the recommended way to best manage the development of digital products and services.
Product management and delivery management
Product Managers’ key responsibility is managing scope; their role is to continuously evaluate and prioritize the most valuable work. This work doesn’t end until the product is sunset. Any change, whether a new feature, a bug fix, or a technical improvement, needs to be prioritized by the value they are expected to generate once they are realized.
The Delivery Manager’s job is to make sure the work gets done and delivered. As business value is only realized once the outputs reach customers and end-users, this means managing the flow of work end-to-end. And along each step of the way, ensure that feedback on quality is collected and actioned.
The 2 key metrics to start managing delivery
Choosing the wrong metrics is ineffective and wasteful, worst case, even counterproductive. On the one hand, there is no single magic metric that captures the complexity of product development. On the other, there is no shortage of useful metrics to optimize workflows, but the first two to measure and optimize against are End-to-end Lead Time and Release Frequency.
DevOps specialists will recognize these as closely related to the two ‘DORA metrics‘ of Lead Time for Changes and Deployment Frequency. This is for the very good reason of keeping product development steered by feedback from business operations and real customers. There are however some key differences too, so don’t stop reading just yet.
End-to-end Lead Time
Lead time measures the total time it takes for work to flow from the moment an agreement is reached to pursue the idea and to include it in a future release (intake) to the time it is actually released to customers or users (release). Lead time is measured in actual time, usually hours or days, and this includes processing times, as well as time work spends sitting in queues or waiting to be started.
Benefits of measuring Lead Time
- By measuring lead time over a set period of time, you can assess the impact of changes to your process. You can answer questions like “did those changes result in delivering value to customers and users more quickly?” or “did your time-to-market increase?”.
- Over time, effective improvements to technology and work processes should shorten the average lead time for all work.
- You should be able to see the effects of prioritization reflected in the lead times. For work that is of roughly the same size, the most highly prioritized work should have shorter lead times than work of lower priority.
- You’ll also see the effect of grouping (or batching) work together. When this is effective, smaller work items should, on average, reach the market faster than bigger items of the same priority.
- Lead time also helps you become more predictable by quantifying the probability that work will get done in a particular time frame, allowing for more accurate forecasting.
How to measure end-to-end Lead Time
Start with gathering data from release notes, work management tools or other documentation that captures when work is initiated and when work is completed.
Determine the right entity to measure
From being an idea or a reported problem to reaching users and customers, work can take many shapes and sizes. The entity to use for measuring lead time is the one that best represents the smallest discreet piece of business value. This is typically a product feature, but can also be a requirement, a user story or a customer service issue. One simple test is to check that the headline or short description makes sense to stakeholders and senior managers.
Capture the endpoint and identify the work items
The endpoint to look for is the one that best represents the moment when the output of the work is ready to start generating benefits. This is typically when it has reached the hands of customers or users.
Next, choose a set time period, typically a month or a quarter – depending on how often new products and product features are released to customers and users. List all the items considered to have a discreet business value that were released during that window, noting the exact date of their release. For example, list all stories released in the last 90 days.
Remember that for software development, release notes are a very good starting point for finding this information.
Finding the earliest reliable point of record
The most challenging part of the process is finding the point when work was first recorded. It’s often the case that ideas or problems change shape by being redefined and rescoped as more information surfaces. If all ideas are not diligently recorded, it’s better to opt for a more reliable source a little later in the process, for example, the time when benefits are estimated, or when business cases are created.
Requirements management and project management tools are usually good sources of information regarding the time when ideas and problems are recorded, but customer service and time management systems can also provide the data.
Calculating the Lead Time
Lead time calculations are very straightforward. For each item, simply subtract the earliest recorded date from the release date. The result should be presented in calendar days for each item released. Lead times can be presented as an average over a set period, typically a month or a quarter, depending on how often new products and product features are released to customers and users.
Potential pitfalls
- Averages are not helpful when comparing items of very different sizes in terms of effort or comparing items that might have been completed using a different process or resources. In this case, use the distribution.
- Sometimes the same tools are used to plan and report on activities where the output is not part of the end-product running in production, for example, knowledge transfer. These activities should not form part of your lead time measurement.
- There is usually a considerable amount of time spent before decisions are made to ‘start’ building a product or product feature. Make sure you include this time as part of the lead time. All time spent before business value is realized is part of the lead time.
- Don’t confuse lead time with cycle time. Lead time includes waiting times in queues and measures the end-to-end duration (until value is released), whereas cycle time measures how long it takes for work to go between specific start and end points.
Release Frequency
Release frequency measures how often updates to products and solutions are delivered to customers and end-users. It’s a leading metric that gauges the capability of a product team to react fast and deliver value early and often.
Benefits of measuring Release Frequency
The business value of a digital product or solution is not generated until it reaches customers or end-users. The more frequently valuable increments are delivered, the more opportunities the team has to deliver value early and often. Release frequency also helps to understand the all-important ability to react quickly and resolve acute and unforeseen issues – for example, the expected time to respond to security and fraud alerts.
How to measure Release Frequency
To start measuring you will need to collect release notes, production log files or other records capturing the time that deployments are made to production environments.
Record the date and time of each deployment to production environments
It’s important to select only the releases that reach customers and end-users with new fixes and features. The type of release does not really matter (they could be alpha, beta or live releases) as
long as they are anticipated to provide business value to a select group of customers or end-users. Release notes are typically a very good starting point as the place where this information is captured. The precision increases with the number of data points. On the other hand, it should show trends and changes. We recommend looking at the last 8-12 releases. If releases are made daily, record the time, otherwise, the date alone should be sufficient.
Calculate the elapsed time between releases
Simply calculate the difference between the first release date and the last release date recorded in step one above. Convert this into a number of time periods such as days, weeks or months. The frequency of release events can be stated in any time unit: per hour, per day, per week, per month, etc. Choose one that is meaningful to your business and provides a frequency that is easy to relate to. When in doubt, weeks are a good starting point.
Calculate the Release Frequency
Count the number of releases from step 1 and divide by the number of time periods from step 2 above.
| Release | Live Date |
|---|---|
| v.1.0 | Friday, Jan 31, 2020 |
| v.1.0.1 | Friday, Feb 7, 2020 |
| v.1.0.2 | Friday, Feb 14, 2020 |
| v.1.1beta | Monday, Mar 2, 2020 |
| v.1.1 | Friday, Mar 13, 2020 |
| v.1.1.1 | Monday, Mar 16, 2020 |
| v.1.1.2 | Monday, Mar 30, 2020 |
| v.1.1.3beta | Friday, April 10, 2020 |
| Count of releases in time period | 8 |
|---|---|
| Weeks from v1.0 to v1.1.3beta | 10 |
| Release Frequency | 0.8/week |
Interpreting the results
First, the frequency should be high enough to enable timely responses to unplanned events and to be able to revert mistakes. This would be highly dependent on the operational business context. Secondly, a higher frequency minimizes the amount of waiting time for value to reach the market to generate business benefits (cost of delay). This means the release frequency should be high enough not to cause unwarranted delays and waiting time added to the end-to-end lead time. Over time, effective improvements to technology, tools, and work processes should enable increasingly higher release frequencies.
Potential pitfalls
Measuring releases other than those to a target market. Releases only generate business value when they are used in an operational business environment. Therefore, releases that only make it to a test environment do not count, even if the responsibility shifts from a supplier to its client. That said, a target market could be internal or external, and the target is contextual; for example, releasing to a subset of users or customers would count.
Counting the same increment multiple times. Subsequent releases when rolling out the same product version to more customers or end-users (incremental roll-out) would not count as a separate release if the product or solution is unchanged. There are other measures that better capture feature adoption of releases. Incremental releases with bug fixes and patches that are released to end-users are valuable and should be counted separately, even if there’s no additional business value generated compared to initial assumptions.
More on metrics and more metrics
End-to-end Lead Time and Release Frequency are, of course, not the only metrics a delivery manager needs to know to manage the end-to-end flow of work. But as a starting point, they provide the context for a more detailed analysis of quality metrics such as failure load, throughput, and flow efficiency metrics.
If you want to know more about metrics or our VFQ Analytics solution for capturing them, please get in touch with our team.
Additional insights
Metrics that matter: How UX measurements drive product success
When product management theory meets cold reality
Measuring the flow of real work
