The Art and War of Estimating and Scheduling Software

In my experience, the majority of estimates and schedules for software development projects are derived from hunches, guesswork, and gut instincts.  We say things like “that should take about 4 hours of work” without backing it up with data.  And we set timelines based on those estimates without adjusting for past accuracy in estimating.  And what about the tendency of the customer / stakeholder to change scope midstream?  That should drive how big the “buffer” is in the schedule for changing requirements.

I’m as guilty as the next person in many cases.  I fight an internal battle on this issue.  Part of me wants to just be a carefree developer with little to no regard for estimates.  I dare say that my time as a computer science student was when I got stuck in that trap.  Nobody taught me to estimate my work, let alone track the accuracy of my estimates over time.  I usually had ample time to do assignments and the code flowed easily for me so I never really worried about how long I spent working on things.  [In recent years, I know that people like Rick Wightman at the University of New Brunswick have been working on teaching students how to think more like professional developers.]

The professional developer in me knows that good estimates are essential to everyone in the development chain.

The businessperson in me cherishes accurate estimates.

ACM Queue has an interview with Joel Spolsky in which he talks a bit about evidence-based scheduling (which apparently is somehow supported in FogBugz):

In evidence-based scheduling, you come up with a schedule, and a bunch of people create estimates. And then, instead of adding up their estimates – instead of taking them on faith – you do a little Monte Carlo simulation where you look at what speeds developers had in the past, vis-à-vis their estimates. You use that same distribution of probabilities, as we call them, that you had in the past and run a simulation of all of your futures. What you get, instead of a date, is a probability distribution curve that shows the probability that the product will ship on such-and-such a date.

Nice!  I need to try that.  Sounds so much better than just using spreadsheets to track history.

What’s most interesting and useful about tracking estimate-to-actual history on a per-developer basis is that you get data about how accurate the hunch / gut instinct of each person really is.  This is so much more powerful than just collecting team or project based accuracy.  Of course there will be outliers in the data, like when a developer who does a lot of similar tasks has to estimate something new or unrelated.

I asked a friend who is a project manager that I respect (and trust) to comment on this.  (I was wondering if there was a more popular term than “evidence-based scheduling” or other good tools to support it.) At his old job he used to teach an estimating course for developers.  Check out the part I highlighted:

Well when you run a project you have a bunch of planned dates (when are you planning to be done), and you are really supposed to keep track of actual dates (when did the work actually finish).  If you are really keen (and perhaps have a database to track this by resource -and perhaps by technology and task type)… you should be able to form some projections on developers… I.e. Billy always takes twice as long to do a design but he does a detailed job so the coding goes twice as fast.  You can use this information to do a gut check on teams too… but Joel is right … these estimates are as complex as the people who are on the teams and the types of tasks they are working on.  There is also the issue of doing something for the first time… very hard to estimate.  It is only when you do something multiple times that you get good at estimating… and only if you are looking back on your estimates looking for trends.  What Joel is describing is putting some trends to the old estimate / vs. actuals data.

Not sure what to call it.  But PMs are always trying to log history of missing or hitting deadlines and you very quickly get a sense of which developers have good estimates, and which developers you need to double, triple, etc… On my last project I had a developer tell me something would be able to get something done by Friday of that week and it actually took about 3 months of 8 people full time… 😉  I’ve heard the statistic of 8:1 (fastest developer to slowest developer).

To my memory, I have never had a developer highlight his/her accuracy in estimating on a resume or during an interview.  That would be an interesting thing to do.  It would blow me away if someone told me they had a spreadsheet with their personal estimation accuracy showing their estimates-to-actuals history.  That would be an impressive artifact!