Metrics Driven Product Design

Cross-posted to Zolo Labs.

I’ve had some downtime from work recently, and have been working on a side project with a friend. (Isn’t that what vacations are for?) We’re calling it Zolodeck. One of the first things we did is decide that we want this to be driven completely by future users. So this is an experiment in user-driven product design, you might say.

Technically, we’re tracking everything that a user can do. This data gets fed into our data-digester. Yes, that’s the technical term. And out comes Insight. The idea is to use these insights to decide how to evolve the project.

We’re looking at several ways in which we can get the infrastructure to support all this, including building out our own. I know it will be a complete distraction from the goal of the project, so I’m certain we won’t go down that route. We’re still evaluating what we’ll end up with, so stay tuned on that. We do want to document our experiment here, so this notion of “metrics driven product design” should be a recurrent theme on this blog.

BTW, we’ll also add convenient text-boxes that users can use to drop us notes (mostly appreciative suggestions, but also hate-mail if they choose :-) ).

Announcing the Bay Area Lean Software Development Meetup

Hi folks, I’d like to announce the Bay Area Lean Software Development Meetup. The first meeting is scheduled for the end of the this month – on the 30th of April.

I’m going to present on the topic that I’m presenting at the Lean & Kanban 2009 conference – and the topic is Lean Software Development for Startups.

This is just the first meetup, so it will be general – exchanging ideas about what different folks in the valley are doing with their software processes… and things like that.

Join the group – and attend, it should be fun!

Adopting lean software development: What is a user story?

I recently got an email from a reader (Dmitry Lobanov) who had some questions about the process stuff I’ve written about here. With his permission, I’ve reproduced the contents of his email and have responded to his queries. It may help other folks that are in early stages of adopting lean/agile methods -

Hello Amit. You keep interesting blog with a lot of useful information about project management. To my opinion the most attractive idea in your blog is user stories decoupling.

Let me tell how I found it interesting and then I’ll ask a few questions, Ok?

In our project we use scrum with some XP techniques. I found link to your blog in an overview of Mingle at Thoughtworks website. While reading I found your idea of decoupling stories into 2 story points tasks very interesting. And I think, that it makes sense. We tried to compare two approaches (yours and card poker) while estimating amount of time required for 1 specific user story. Well, it is not 100% your approach in fact, because we still tried to use estimation and ideal days, but we took your idea concerning 2 story points tasks. But we are not ready to forget about estimation yet :)

First we estimated using card poker, and then repeated estimation using your approach. And we found the following advantages of your approach:
– your approach gave us complicated tree of concrete task. That means that we could just take them and start working on this user story, we know what and how should be done. In other words we have got a plan! Card poker didn’t force us to decouple user story into atomic tasks and just gave us list of generalized tasks (which themselves can be user stories).
– card poker estimation gave us some number of story points (we used ideal days according to scrum guidelines), let it be X ideal days. And then we used your approach for estimation: number of atomic tasks multiplied by 2 story points (we still use ideal days :) ), let it be Y ideal days. And we have been amazed, that Y was 10 times greater than X! 10 times! That’s too much.

This experiment shows us that standard estimation lacks of accuracy a lot… well, we suspected it, but only after experiment we saw it clearly. And using card poker (i.e. bad decoupling) we constantly have problems with developers got stuck in development process, and as a result they do 5-days task for 2 weeks or even more. We have one user story, which had been estimated as 8 days, but two developers work on it for 2 sprints already. It’s no good at all.
So in the near sprint we’ll try decoupling user stories into 2 story points tasks and watch the result.

I have a couple of questions about decoupling and sprint planning, and it would be nice if you provide your point of view concerning them.

In your post about requirements management (http://epistemologic.com/2007/04/08/requirements-management-user-stories-mind-maps-and-story-trees/) you touched only initial planing. But what to do with bugs and technical tasks? I’ll explain. Look, our project has been in development for 2 years, we develop payment acceptance system. And our system has been in use for a bit less than 2 years (it took approximately 3 moths to issue first release). So it changes constantly, we issue new release every 2-3 weeks. But when the development started, team suffers a lot from a lack of project management experience. I came to project few months ago (I’m not a project manager, just developer, who wants to improve project). Now situation seems to improve. But we have a LOT of bugs. As you understand, these bugs can’t be attached to specific user story, moreover we hadn’t user stories before, in fact we introduced scrum only 2-3 moths ago.

For example we need to fix one bug (one of many) concerning working with hardware. And we clearly understand, that we need to refactor part of our system, which works with hardware. It could take a lot of time, we understand that we need to add hardware manager, change hardware interfaces and so fouth in order to unify working with hardware. And it should be done, because if we just fix bug not changing anything, then literally tomorrow we’ll have to fix similar bug, and then another one, e.t.c. How this task should be registered? It is not a user story, because product owner (or stake-holders) wouldn’t see the result, they wouldn’t be able to “touch” it, it can’t be demonstrated to them. And it is not a defect for particular user story, because it is the behaivour of legacy code, we haven’t corresponding user story. What should we do?

I see only one solution. We should wait till stakeholders give us user story concerned this functionality. For example, stakeholders want us to add support for a new type of device. Hence we can include refactoring task to a bunch of tasks under this user story.

But our system has been in development for a long time, and we added support for a lot of devices already. So it can take a lot of time till suitable user story will appear. And we should just wait and fix bugs?

And how to register bugs if we don’t have corresponding user story?

Another one situation we faced yesterday. If we have user story, which involves a lot of changes in architecture. That is user story contains a lot of task, but these task can’t be accomplished separately, and they are not user stories themselves, because they are technical tasks (refactoring and researches). It’s clear that we can’t show them to stakeholders, but we can’t accomplish user story not accomplishing these tasks. And we think, that we won’t be able to accomplish all these tasks in one sprint. And I don’t know how we can decouple 1 user story intro several user stories. What should be done?

Correct me please if I am wrong in my interpretation of user stories. I think that user story is “something” useful to stakeholder. Stakeholders can “touch” user story, they can take a look at it, they can play with new functionality and so fourth. Hence anything that can’t be demonstrated to stakeholders is not a user story, is it so? Stakeholders are not interested in tasks, they want to know only 2 things: What and When, they don’t want to know How.

And I have another one question concerning terminology. What does “epic story” mean in terms of user stories? How does it compare to user stories?

Looking forward to your reply.

Best regards,
Dmitry Lobanov

OK – so here’s my take on the questions raised:

User stories: I’ve had to deal with the question of what a user story is quite a few times over the years. There are many theoretical definitions – and I don’t care about most of them. In my opinion, the only thing that matters is this – a user-story should add business value. That raises the obvious – shouldn’t everything you do while working on a project satisfy this criteria? Yes, it should – and this is usually something that business stake-holders understand quite well.

There are several kinds of stories. The first is the obvious “feature” stories that have a GUI etc. This is the category of stories that can be “touched” and is “tangible”. Justifying this one is easy – indeed, these are the ones most commonly requested by the business.

Then there are the “technical” stories that don’t have a UI – e.g. – “store the uploaded data in a compressed format because the space used up by the system right now is costing us too much money”. Again, the value is obvious.

The next category is bugs – fixing these delivers clear value and are no trouble to manage as such. See below for more information.

One other type of story is the spike. These are easy to deal with since it is obvious what value they deliver – namely an experiment to determine if an approach might work.

The final variation is stories that deal with paying down technical debt. These are harder to justify because the business is usually not technical enough to understand these. There is no easy answer to how to sell these – but there is usually a very tangible benefit that can be attained by playing these cards. Usually this benefit is deferred – allowing the team to build new features faster, or reduce the number of bugs being discovered in a subsystem. When put in these terms the business is usually quick to understand – and then it boils down to a question of ROI. The cost of implementing these cards should be less than the savings they represent. It is important for everyone on the team to see the whole – and make these decisions together. The key is to sell the business on the value these stories delivers as opposed to the technical details. And by the way, this should be done whenever the debt starts to weigh heavier than what it should. A clean code-base is a happy code-base.

How to deal with bugs: The short answer is I usually treat bugs as stories. They don’t need to be associated with a specific user story. A bug is a bug – it doesn’t matter how it came to be – (of course, it is often related to a story – and most often the same developer ends up fixing such bugs) – and I track them in the same system used for stories. I also let the same prioritization process determine which ones get fixed and when. Further, some bugs are so tiny that they may even get fixed without there being any record of them at all. Some are so large that they might actually result in multiple stories – especially if a redesign is required. Keeping in mind that it is cheaper to fix bugs sooner than later, bugs can be managed just like any other story.

So – to recap, I don’t care for the theoretical definition of user-stories – that they should be tangible and what not. Instead, I recommend that folks do what makes sense for their situation. If I had to give only one tip around stories, it would be this – keep them small. This mnemonic might help.

Finally – about epic stories. An epic is merely a feature (or some technical aspect of the system) that is way too large to be completed as a single user-story. It is epic, because it is a large story :) I always keep playable stories down to less than 2 days in length – as far as possible. Epic stories, therefore, must be managed by breaking them down into incremental chunks of functionality. This is possible about 95% of the time – sometimes however when some spike is involved or some hairy technical refactoring is needed – longer stories can be played.

I hope these thoughts answer some of the questions. I’m sure other folks have different experiences and different solutions to these issues – and that’s fine too, since agile is all about adapting.

Conference – Lean & Kanban 2009

I will be speaking at the Lean & Kanban 2009 conference which is being held in Miami, Fl in the coming February (18th to 20th).

There is a great line-up of speakers, so if you’re at all interested in the Lean Software Development space, this could be a great conference to visit in 2009.

Take a look here – http://www.leankanbanconference.com/

What iteration-less agile really means

I’ve written about the idea of an iteration-less process before, and we’ve been using the spirit of the idea at the startup I work at (Cinch) for the past few weeks.

The idea is simple, and has two parts. First, eliminate the concept of formal iterations (use a pull-system to let work flow through the pipeline – use Kanban cards if you like). Second, replace the rhythmic heart-beat of iterations with actual releases to customers (internal or external).

These releases do not have to be spaced out by a week or two or any arbitrary span of time. They ought to be, in fact, the shortest amount of time that the software system can be re-stabilized after undergoing change (through adding features or fixing defects). Hence, if it takes 2 days to do a release, do it then. If, for some reason, it takes 6 days (instead of the old five), then so be it. However, the focus of the team should be on getting quality software into the hands of the final users. After all, everything else is inventory.

Pretty simple, eh? We have combined this with the idea of the estimation-less process, for even greater effect. I have been hearing from some people that I have worked with in the past, and they’ve reported excellent results through just such a process. So, anyway – that’s what iteration-less means. :)

P.S.

In a past life I had spoken about this with Jeff Sutherland when I attended Deep Lean (in 2007) – and he agreed that this made sense. He didn’t recommend it for any but the most experienced team – and I certainly agree with that. If your team is new to agile or is lacking in some other way, getting it going with the standard approach to agile is probably the best thing to do.

However – if your team can pull this off, you might want to give it a shot and see how much productivity increases over time.

Why I don’t use real hours for sprint planning

OK, so several of my friends and colleagues pointed me to Mike Cohn’s recent blog post about why he doesn’t use story points for sprint planning. I left a couple of comments on his blog, but I couldn’t wait until he moderates them… so I thought I’d post my thoughts here.

In my opinion, using either story-points or real-hours for sprint planning makes one assumption. The assumption is that it is somehow very important to get a sprint “just right”. In other words, it is important for a team to estimate a sprint’s worth of work well enough so as to make it fit perfectly within a sprint.

I believe that it’s not important at all. Here’s why.

First of all, the length of a sprint is picked by the team in a mostly arbitrary manner. Some pick 30 days, some pick 1 week. For the most part, it could be any number in between there. So if the duration is not that critical, then why should fitting work into it be? In any case, when a team happens to under-estimate, the undone work gets pushed into a future sprint. And when a team over-estimates, the stakeholders gleefully add more work to the sprint. So… what was the point of all the real-hours-based estimation?

Next, lets talk about predictability. Take an example of a basket-ball team. They can make a general statement about their long-term average, but can they predict what they’ll score in the next game? No… but even more critical, is it even important to try to make this prediction? The answer is not really, it is only important that they do their darned best. A software team can certainly spend time breaking stories down into tasks and then even more time estimating each in real hours, and then bargaining with the stakeholders to make sure whatever they pick all fits in… but it’s just more important that they work on the highest priority cards, with as less distraction and road-blocks as possible.

Whether they fit it all in, into that one sprint or not, is a mere detail.

Now, it’s not to say that having dates is not important. It is, but you can’t be scope-bound AND time-bound. Pick one. My default choice is time-bound with variable-scope. In other words, I like to set a release-date, and then ensure that the team is burning down the highest priority items at all times, as that date approaches. When the date is nigh, release! We’re all writing production-ready software, aren’t we? Then, repeat.

This approach then makes it possible to simply drive the whole project off the product-backlog… without strict sprint-backlogs. It allows velocity to be measured in story-points (from the product-backlog) and prediction is a matter of total-story-points divided by velocity-in-story-points. Simple. If anyone asks what will be done this sprint, the answer is the highest priority stories on the backlog, as many as the team can, here’s our best guess (average-velocity-in-story-points). This approach is actually faster than sitting and re-estimating individual stories in real-hours, because well, the team doesn’t have that distraction anymore.

In fact, from my experience, when the team gets comfortable doing this, they even ask… OK, so if the length of the sprint was set fairly arbitrarily, and as long as we work on the most important stuff at all times, then why are we bothering with the concept of sprints at all? These teams go beyond what I described above and eliminate the waste of iterations. Fun, eh?

If I talk about this stuff to people, sooner or later, someone brings up the happy concept of rhythm – how it is important for the team to have a pulse, a beat to which they work – sprint-planning, development, sprint-close, retrospective, repeat. It gives them satisfaction. It probably does, but I think we can do better. I prefer my teams to not just close an iteration based on some rhythm, but even push the software into production each time. Let the rhythm be set not by an arbitrarily picked number, but through customers getting their hands on the next release of the software. They still do the retrospectives regularly, and have team outings all the time. But the rhythm of the team is set through actual releases. When teams get better and better at this, and realize the critical importance of cycle-time, they do all they can to reduce it.

Anyway, I can talk about this stuff for a long time. Back to sprint-planning. My take is – do it like agile architecture/design. Think of it as evolutionary. We’ve eliminated the big-up-front-design, it is time to eliminate the big-up-front-sprint-plan. Prioritize the back-log constantly, always ensure that the team is working on the highest priority item. Ensure there are as less distractions and road-blocks as possible. And release quickly, as early and as often as possible – make a rhythm out of it.

And if the stake-holder asks when a particular story will be done, see where it stands in the prioritized backlog. The answer to the question is, of course, after all the ones above it are completed. Ask also if they would like to re-prioritize. That’s all there is to it!