Effort estimates for software considered evil

February 24, 2010 at 10:23 am | Posted in Agile, Programming | 1 Comment

Recently I had the opportunity to look at effort estimates for a big (about 4000 function points) software project. The interesting thing was that there were already quite elaborate specifications, so in theory that would be a good basis for an estimate. The estimates were done by three different parties:

  1. An external consultancy firm. They counted function points based on a document with use-cases.
  2. The internal business analysts.
  3. The developers. They split-up the use-cases in user-stories and assigned story points.

Both the business analysts and the developers made relative estimates instead of absolute estimates in for example hours. What I had expected was a reasonable correlation (0.90 or higher) between the three estimates. This turned out to be quite different. Lets first have a look at a graph that plots the value of the estimations from the consultancy firm against the business analysts:

Consultancy firm versus business analysts

On the horizontal axis you can see the estimates in function points. On the vertical axis are the estimates from the business analysts in an arbitrary (but relative) unit. As you can see from this graph the correlation is pretty poor. In fact, when you do regression analysis, the correlation is only 0.73. This means that about 50 % of the estimates of the business analysts can be explained by the estimates from the consultancy firm. The consultancy firm at the same time claims about 15 % accuracy in their estimates.

When we look at the difference between the business analysts and the software development team we see a similar picture:

Development team versus business analysts

There are a lot less data points here because the development team only estimates those user stories that they will pick up in a next iteration of 2 weeks. Again the correlation between the two estimates is pretty poor: 0.71. So once again only 50 % of the estimates of the development team can be explained by the estimates from the business team, assuming there is a linear relation between the two.

If we correlate the estimates from the development team with the consultancy firm we get similar data. What worries me most is that the approved budget and lead time will be based on the estimates of the consultancy work which apparently bears hardly a relation with the effort that the development team has to invest. The estimated effort could easily be 100 % of more wrong. Guess who is going to be held responsible for that…

1 Comment »

RSS feed for comments on this post. TrackBack URI

  1. This is very interesting. One fundamental assumption in Scrum and XP is that story (or complexity) points are so easy to estimate so you can rely on them for planning. It seems that, in this case, that is not true. I hope you client can deal with the unpredictable nature and focus on value.

    Are you planning to track all the velocity in all 3 scales? I predict the FP estimates will produce the most reliable velocity number.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: