About six months ago, I wrote a blog post called Top 10 Tips for Measuring Agile Success, and the reality is that it wasn’t necessary a set of tips as it was a blog about the to ten ways people responded to the VersionOne State of Agile survey and some related metrics that support them. Way before that blog was ever published, the question of how to measure agile success was a common one that I and many other agile coaches would receive when working with organizations and executives. Since the blog was published, I’ve had more questions and in some cases some rather odd reactions to the concept of measuring agile success. Some questions are very direct — “which metrics really work?” Or, “which metrics should be used at the various levels of the organization?” Then there are the reactions or questions like, “aren’t you aware of the impact of metrics?” Or, the statement, “suggesting the one way is ridiculous.” Or, the best reaction, “dude, I hate metrics.”
Okay, I can accept all this and I get the confusion and general concern, and trust me — I share some of these sentiments. Instead of looking at the question from the stand point of which metrics are the best, let’s explore the topic or the questions of how do we measure agile success and why is it important.
Let’s start with the “why”, and I think the primary “why” is obvious — the cost of change can be significant. There’s not only a tangilble investment in training, coaching, reorganization, staff changes, and even re-engineering the physical environment, but there’s also the significant intangible cost associated with productivity loss due to teams reforming, working through the chaos, and emerging through the change usually with something that looks much different than what you started with. I don’t think I’ve been around a team or organization going through the change associated with adopting agile that hasn’t had staff turnover, fits-and-starts, and a brief time of general struggle both for the people and the software output as everyone comes up to speed. So, trying to understand the return or the off-setting value gained is an important reason to measure agile success. To that end, it’s not really measuring agile success, it is better stated as measuring the success of the process investment change that organization is embarking upon or has recently spent six-months enduring.
Another “why” for measuring agile success is to enable the PDCA loop. The PDCA loop (a.k.a. the Deming Circle or Plan-Do-Check-Act [Adjust]) is a core business and leadership practice and it is called out in all lean and agile approaches. The concept is simple — establish a goal, decide what you are going to do, get it done, inspect the results, make adjustments based on observations, and then do it all over again as you march to the goal — the essence of iterative development and continuous improvement. Measuring the organizations progress and performance allows for the inspection to occur; thus, you adapt and get better the next time around.
So, we need to ensure that the organizational change we’ve embarked on is making the positive impact we expect and a key part of ensuring this is measuring to enable continuous improvement.
How we measure our agile success is a bit more complex — mostly because there are two things to measure. First, we need to measure the adoption of agile principles, processes, and practices. Second, we need to measure how our organization is performing to assess the impact of changing to agile.
The approach to measure agile process success is generally around leveraging Agile Assessments which hope to identify where your organization is on an “agile maturity” spectrum. There are several long established approaches that internal and external coaches use. The concept of measuring maturity is simple, conduct a self-assessment based on both quantitative and qualitative measures in several areas including team dynamics, team practices, requirements management, planning activities, and technical practices (just to name a few). For these measures to mean anything, you need to start with a baseline (how mature are you today?) and then select a reasonable cadence to re-assess on your road to … more maturity? There are some very-useful existing maturity assessments out there including Agility Health, the classic Nokia Test, and about 20+ others listed on Ben Linder’s blog.
Agile assessments do have some aspects of measuring impact; however, the focus is generally isolated to certain areas and or used to reflect the success back to the process. Measuring agile success from the standpoint of impact on the organization should be more focused on The Moneyball metrics of the business. Measuring impact is much more difficult sometimes because it can be difficult to tie a direct correlation between the agile delivery metrics and the traditional business metrics. It is also difficult because the lack of understanding of the agile delivery metrics. Making matters worse is how people sometimes focus on the wrong ones, which takes me back to The Moneyball reference. It’s important for organizations to select the right metrics to focus on and the right ones to tie together. As mentioned by Michael Mauboussin in his HBR article The True Measures of Success, leadership needs to understand the cause and effect of metrics. What this means, metrics if not selected correctly can provide misdirection and can result in misbehaviors — basically people will make bad decisions and game the metric.
To give you an example of a [not so solid] agile success impact metric, lets look at a common metric that people often argue about – sales revenue tied to the delivery organization’s velocity based on story points (e.g. revenue / velocity). The first challenge with this is using the term story points [and velocity], you tend lose or confuse people not familiar with the concept and, if they do, an argument about estimation generally ensues and people often change their point measuring stick. To avoid this challenge, go with safer, lean metrics or simply put, the count of stories or things (great advice from Jeff Morgan – @chzy). The next challenge with this metric is that it may be too generalized and not really lead to better results. There may be better goal focused measure such as publication mentions after a release that leads to an increase in the number of product trials. Or possibly a goal of reduces support tickets which leads to improvements in customer retention or renewals. All of these are good, but alone they don’t necessarily provide an ability to measure agile success. To help assess your agile success, correlate the impact metrics with the lean, agile metric — the number of stories delivered during the same period. For example, use the number of stories delivered to normalize product revenue, number of web visitors, number of trials, and the number of support calls. Watch and assess these trends over six months and see the impacts.
I recently read a book called RESOLVED: 13 Resolutions for LIFE by Orrin Woodward. Although the book is aimed at leadership development, one of the resolutions talks about establishing and maintaining a scoreboard. The idea is that we should have a set of metrics that we constantly visit that help to power our PDCA loop. This is a long running practice in business, and if you don’t already, I suggest you establish a scoreboard that helps you measure your agile success. It should include metrics from your process adoption assessment as well as your organization’s agile-adapted Moneyball metrics. In agile we often talk about big-visible charts, your agile success scorecard should be one. Share the results of your agile journey and the impact it is having on your organization, and help people understand what the metrics mean and what decisions or actions should be made based on the indications of the metrics. There will be times things don’t look good, but done right, your agile success scorecard should help spur and inspire an environment of continuous improvement that embraces agile principles and practices you’ve embarked on implementing.
Although, I don’t call out any specific examples of agile success scorecards — it would be great if you would share your examples or metrics you like or resources that can help others.
There are many worthy reads on this topic, but a couple more that I like are Agile Fluency, established by Diana Larsen and James Shore, as well as this article by Sean McHugh, How To Not Destroy your Agile Teams with Metrics.