Effort estimates for software considered evilFebruary 24, 2010 at 10:23 am | Posted in Agile, Programming | 1 Comment
Recently I had the opportunity to look at effort estimates for a big (about 4000 function points) software project. The interesting thing was that there were already quite elaborate specifications, so in theory that would be a good basis for an estimate. The estimates were done by three different parties:
- An external consultancy firm. They counted function points based on a document with use-cases.
- The internal business analysts.
- The developers. They split-up the use-cases in user-stories and assigned story points.
Both the business analysts and the developers made relative estimates instead of absolute estimates in for example hours. What I had expected was a reasonable correlation (0.90 or higher) between the three estimates. This turned out to be quite different. Lets first have a look at a graph that plots the value of the estimations from the consultancy firm against the business analysts:
On the horizontal axis you can see the estimates in function points. On the vertical axis are the estimates from the business analysts in an arbitrary (but relative) unit. As you can see from this graph the correlation is pretty poor. In fact, when you do regression analysis, the correlation is only 0.73. This means that about 50 % of the estimates of the business analysts can be explained by the estimates from the consultancy firm. The consultancy firm at the same time claims about 15 % accuracy in their estimates.
When we look at the difference between the business analysts and the software development team we see a similar picture:
There are a lot less data points here because the development team only estimates those user stories that they will pick up in a next iteration of 2 weeks. Again the correlation between the two estimates is pretty poor: 0.71. So once again only 50 % of the estimates of the development team can be explained by the estimates from the business team, assuming there is a linear relation between the two.
If we correlate the estimates from the development team with the consultancy firm we get similar data. What worries me most is that the approved budget and lead time will be based on the estimates of the consultancy work which apparently bears hardly a relation with the effort that the development team has to invest. The estimated effort could easily be 100 % of more wrong. Guess who is going to be held responsible for that…