A Bug's Life

I've written before about useful metrics for software development organizations, and it's past time for another. The metric I'm proposing now, I call "A Bug's Life". Because who doesn't like Pixar? And it's descriptive.

What is this

The basic idea is to track how long it takes a defect to be found and fixed. At some point, a developer will write some code that has undesirable behavior, and thus a bug is born. Some time later, someone will notice it, and then finally it gets fixed. That sequence is the basic lifecycle of a bug. Tracking this lifecycle is useful and provides good data points to evaluate your team's maturity and growth. It should also be fairly easy to do. Each individual data point simply consist of 3 dates. The dates for when the bug was introduced and when it was fixed come from your VCS system. The date for when it was found can come from your application lifecycle management system, or test case management if you're fortunate enough to have a good system for that. And best of all, the data can be mined from those systems after the fact, so you don't need to interrupt your regular work flow to support this metric.

How do we do it

The basic process to gather this metric goes like this:

1. At the end of a sprint, you look through your completed tasks and pull out the bugs. That gives you the found date.
2. You then check the commit logs to get commits that fixed each bug. That gives you the fixed date.
3. And then you inspect the commit history with a blame/annotate tool to find the commit where the bug first appeared. This gives you the introduced date.

If any of those steps are difficult, that's also a good indicator that you have room to improve your process. You need to use source control and task trackers. That's just basic stuff. You should integrate them so that tasks can be referred to in commits and vice versa. You should then be making those references. And you should be writing good, structured, and descriptive commit messages. What exactly that means is the topic of many an internet flame war, but I'd suggest any structure that makes this analysis difficult doesn't qualify as good or descriptive.

Once you have all of your dates, you write them down somewhere. Maybe put them on calendars or timelines. And after a few sprints, you will have a trend, and you can start using that to inform how you approach future sprints. Maybe you'll now find yourself armed with visuals that demonstrate that you need more support for testing. Or that too much attention is paid to adding features instead of preserving quality. Your analysis could turn up a myriad of things, but the point is now you have the data. And going forward, you'll have a way to measure the progress of your team's self improvement.

But wait, there's more

I think this is a particularly good metric for a lot of reasons. For one, it's multidisciplinary. It covers the work of all (or at least most) of the roles in a typical development team. So everyone has skin in the game, and if anyone shows improvement, that should translate to a total improvement for the whole team. And I don't see any way to game this metric, short of just lying about when things happened. There's no artificial way to change the result. Improvement means that bugs actually get found and fixed in less time.

Another reason is that it makes you actually look at your development history. In hindsight, you might see some patterns appear. You might notice that bugs are often introduced while working on large stories, or in particular areas of code, or when code review gets skipped. The metric itself doesn't tell you this, but gathering the data gives you a chance to notice it on your own.

The last reason is pretty meta and subjective, but I think still valid. Gathering this data will require you to actually use a lot of the artifacts that you produce in your development process. How easy or difficult it is to gather this data can give you a sense of how good that process is, or how well it's being followed. If you can automate some portion of this data gathering, that's a very good sign. If you can automate all of it, then I seriously envy you and maybe you should be the one writing about this stuff. And either way, it is useful prep work for your retrospective.

So there it is, that's A Bug's Life. All that's left is for the caterpillar to turn into a beautiful butterfly.

Haha! Metaphore.