A simulation to show the importance of backlog prioritization

June 8, 2011 at 6:36 pm | Posted in Agile, Uncategorized | 10 Comments

A successful agile software project depends on a prioritized backlog. You can have a perfectly functioning development team that efficiently finishes one feature after the other, but if these features hold no business value it still is waste.

To show just how important this prioritization is I did a Monte Carlo simulation, using Google Docs spreadsheet. Here are the assumptions I made to create the model:

• the business value of a user story is randomly generated from a uniform distribution between 1 (very low value) and 100 (very high value).
• the effort to implement this user story is generated by drawing from a standard set of Scrum poker cards with the values 0, 1/2, 1, 2, 3, 5, 8, 13, 20 and 40 (100 and ? are omitted). The drawing was done first selecting an index based on a normal distribution with average 5 and standard deviation 1. This (zero-based) index is used to select a card from the range of poker cards. For example index 5 corresponds with the fifth card which happens to be 5 as well. Index 6 corresponds with the value 8, etc.
• the assumption was made that there is no correlation between the business value and the effort needed to implement the corresponding story.
• the whole backlog can be delivered in 10 sprints.
• during the first 4 sprints the velocity is increasing in a ration 1, 2, 3, 5. This means that the development team is 5 times as fast in the forth sprint compared to the first. After that the velocity stays constant at a value of 5.

With these assumptions I calculated the delivered business value (as a percentage of the business value of the whole backlog) using 4 different prioritization algorithms:

• no prioritization. The development team picks up the stories in the (random) order from the top of the backlog.
• prioritization using the business value of the stories. Stories with high business value are built first.
• prioritization using the needed effort to build a story. Stories with low effort are built first.
• prioritization build on the ratio between business value and effort. Stories with a high ratio (high business value, low cost) are built first.

The results can be seen in the following graph:

As you can see from this graph is that not prioritizing your backlog will deliver business value quite late. Since there is no correlation between business value this will be a straight line after the first 3 iterations in which we have a low velocity.

The other extreme is the prioritized backlog using the business value / effort ratio. This is the red line. Here you can see that after 5 iterations you already have delivered 80 % of the business value. For the backlog without prioritization this takes more than 8 iterations!

What looks a bit surprising initially is the difference between the prioritization on business value (the yellow line) and prioritization based on effort (the green line). This is introduced by the fact that in our simulation we have created a bias towards more expensive user stories: the number of user stories with effort lower than 5 is equal to the number of user stories with an effort higher than 5, but remember that the Scrum poker card values are not symmetric: 0, 1/2, 1, 2, 3 versus 8, 13, 20 and 40. That is the reason that in this simulation it is better to prioritize on effort instead of business value.

There is one more interesting aspect to investigate. What is the impact of the standard deviation that we use to draw from our poker cards? I repeated the simulation using the business value / effort ratio for the prioritization and different values for the standard deviation. The result:

The results are easy to explain: when using a higher value for the standard deviation you will get more extremes (either 0 or 40) for the effort to implement a user story. This means that there will be more user stories which require little effort and deliver good business value. These can be build first.

Bottom line: make sure your product owner works hard on a prioritized backlog. Let him quantify the business value of his user stories. As you can see from this simulation half of the effort (and cost) needed to complete the whole project might already be sufficient to fullfil his business needs. This is a huge money saver!