The importance of evidence in software development
Hakan Erdogmus is the Editor-in-chief of the IEEE Software magazine.Â In the May/June 2010 issue of the magazine, Hakan has written an impassioned plea for the importance of using evidence as the basis for software development decisions, especially when deciding on the adoption of software engineering ideas. I totally support Hakan’s position.Â Too many of us forget the “science” part of computer science and the “engineering” part of software engineering.Â The testing of hypotheses in science and the use of prototypes in engineering are fundamental to those disciplines.Â Yet in software development, we rely on methods more akin to building a bridge by adding one brick at a time on either side of the river and hoping that it will meet in the middle and be strong enough (while ignoring those budgets, deadlines, materials and colleagues that get swept away by the river during construction). For deciding on the adoption of software engineering ideas, Hakan categorizes three types of evidence in increasing order of rigor:
- Feasibility Check – While it is the weakest of the three, a feasibility check should be able to:
– quickly evaluate major advantages, risks, limitations – roughly estimate the size of the problem – assess the novelty of the solution – test the cost effectiveness – help to decide if the idea is worth exploring further and under what controlled but real-world circumstances
- Anecdotal Evidence – This is effectively “crowd sourcing” your evidence gathering in support of a new idea with all of the advantages (the pioneers are the ones with all the arrows in their backs) and disadvantages (has the author really tested the idea under similar circumstances as you?Â Do they have a vested interest in supporting the idea?) of that approach.Â Nonetheless, external data points ARE valuable and a significant step up on a purely internal feasibility check.
- Systematic Evidence – Of course, researched and documented evidence of the merits of an idea is the best type of evidence but it is difficult to find or produce if the ideas are new.Â Mature organizations can make this type of evidence much easier to discover and use by adopting a methodical approach to capturing lessons learned from projects.
Hakan makes the excellent point that the SIZE of the decision or idea should dictate the threshold of the quality of evidence required.