Agile Governance – a little history and an illusion

In order to better appreciate the benefits of Agile, it is worth understanding a little bit about the history of project governance, and how things got to where they are today.

The history of incremental and iterative development goes back to the 1960s and even earlier. Various individuals and organisations were experimenting with ways to improve software delivery, including large companies such as NASA and IBM.

The seems that the waterfall model was introduced as a reaction to the ad hoc ‘code and fix’ practices of the early days, which lacked rigour and controls. Although interestingly, many of the sources often quoted to support the use of a waterfall delivery model are a lot more lukewarm than they seem, including the US Department of Defense standards that other large organisations followed and the well known paper by Winston Royce.

In his book Agile and Iterative Development, A Manager’s Guide, Craig Larman calls this “the historical accident of waterfall delivery”.

In the 1990s, as the number of software projects exploded, failure rates of projects also reached problematic proportions. Waterfall was seen as the control methodology that could get things back on track. It was hoped that stricter governance could put a lid on these spiralling failure rates.

The thing about waterfall delivery and governance is that it sounds like it should work. On paper it makes sense. It is simple and linear, whereas iterative development is more complex to understand.

But we are delivering software in the real world, and real world issues get in the way of well crafted plans every time…

Single pass waterfall delivery gives the illusion of orderly, predictable, accountable and measureable process, with easy to define milestones that can be marked by simple deliverables (e.g. requirements document complete).

“Plan the work, work the plan”

The problem is that it is an illusion.

For example, using a gating process to allocate money to a project creates an illusion of less risk at each stage as you are achieving a ‘milestone’ before more money is handed out. And you are assessing the progress of the project at each gate.

The trouble is that you are really not reducing risk. No-one is ever brave enough to cancel a project where thousands or hundreds of thousands of dollars have been spent, and the only thing we have to show for it is a couple of documents and some code that isn’t integrated with anything. The numbers are fudged at each ‘gate’ because no-one can bear the idea of so much lost money.

Contrast this to an agile project where working software is delivered every few weeks. If a project is deemed to no longer be worth spending money on, you at least have something to show for all your hard work. The ‘milestones’ on agile projects are not documents, but working software that delivers value to the business. This means that it is much more likely that a decision will be made to stop work on a project when that is the right thing to do.