Software development effort estimation is the process of predicting the most realistic use of effort required to develop or maintain software based on incomplete, uncertain and/or noisy input. Effort estimates may be used (and are often used) as input to many other business processes (project plans, iteration plans, budgets, investment analyses, pricing processes and bidding rounds).
Published surveys on estimation practice suggest that expert estimation is the dominant strategy when estimating software development effort. With this method typically, effort estimates are over-optimistic and there is a strong over-confidence in their accuracy.
Overconfidence: Why You Suck At Making Development Time Estimates posting recommends to read a a lengthy post Coding, Fast and Slow: Developers and the Psychology of Overconfidence about why software developers often struggle to estimate the time required to implement their projects. Drawing on lessons from a book called Thinking Fast and Slow by Dan Kahneman, the writer explains how overconfidence frequently leads to underestimations of a project’s complexity. Unfortunately, the nature of overconfidence makes it tough to compensate.
Coding, Fast and Slow: Developers and the Psychology of Overconfidence article gives reasons why we’re so bad at making estimates:
The first is the sort of irreducible one: writing software involves figuring out something in such incredibly precise detail that you can tell a computer how to do it. And the problem is that, hidden in the parts you don’t fully understand when you start, there are often these problems that will explode and just utterly screw you. And this is genuinely irreducible.
The second problem is overconfidence. Specifically, in many, many situations, the following three things hold true:
1- “Expert” predictions about some future event are so completely unreliable as to be basically meaningless
2- Nonetheless, the experts in question are extremely confident about the accuracy of their predictions
3- And, best of all: absolutely nothing seems to be able to diminish the confidence that experts feel
Alright, So Let’s Stop Being So Overconfident! The promise of the “full specification before we start coding” approach is that you can get more accurate estimations. Congratulations, you’ve just invented Waterfall. But that totally fails. Like, always.
Using short sprints in development are one way on trying to handle the bad estimation problems: they place a hard limit on the cost of a horrifically bad estimate.