The problems of software development (2002) | Brussels Blog

The problems of software development (2002)

This is a note I wrote in 2002 and 2003. It is not easily accessable elsewhere. It is still relevant although the examples are old. There are, of course, more recent examples.

It identifies procurement methodology as a key element in the failure of large software projects. As such, it has implications wider than software development.

The Problems of Software Development

Introduction

This study document investigates a particular aspect of the problems of software development which is one of the main causes of failure for many medium and large-scale software projects. It draws upon a combination of informal and respected formal sources. The conclusions reached are what I believe to be the best picture of the current situation and the best way forward for future software developments commissioned by medium sized companies.

Development Approaches and Financial Models

Within the IT community there is considerable debate upon the merits of different software development methodologies. This debate has recently become more intense because of the wellpublicised failure of a number of large public projects. Software development is now national news.

The problem of managing software development in a market economy has a scope that ranges from the generalities of economic theory to the specifics of computer systems. Few have a grasp of the full range of issues involved.

It is becoming increasingly apparent that the financial environment is a fundamental part of that  process. This document starts with two idealised financial models: the “Cost Plus” model and the “Fixed Payment” model.

The Cost Plus Model

In the Cost Plus Model a contractors are paid for the resources that he uses together with a margin for profit. Contractors need not given a comprehensive initial specification nor indeed need the customer know at the outset what exactly it is that is required. The situation is akin to that of a person who tasks a jobbing horticulturalist to produce a pleasing garden. General instructions will be given at the outset that will be changed or refined as that garden comes to life. In the meantime, time and materials will be paid for.

The potential disadvantages inherent in this approach are obvious. Who would support a department spending potentially large sums of money when it does not appear to have a clear idea of its requirements? Perhaps the most obvious disadvantage is that there is a clear incentive for an unscrupulous provider to proceed slowly and inefficiently. The longer it takes the more he gets. For these reasons, cost plus contracts attract widespread criticism.

The Fixed Payment Model

This may be a project with a fixed price single payment or it might be more complex. For example, the fixed price may be replaced by a stream of income that is related to the continued use of the software similar to the “Public Private Partnership” schemes. The important characteristic of the Fixed Payment model is that the deliverables must be defined by a comprehensive specification prior to work commencing.

In simple terms, a price is given for a job. The advantages of this approach are as obvious as the disadvantages inherent in the cost plus model. The process is transparent. The financial commitment to the customer is known at the outset. He can expect a certain product at a certain price at a certain time.

It is for these reasons that the Fixed Payment model is favoured within the current economic orthodoxy. Sadly however, within the context of software development contracts rarely deliver as expected.

Software Development Approaches

There are several different approaches to developing software. Here we will divide them into two main groups: the “Big Bang Approach” and the “Step by Step Approach”.

The Big Bang Approach

This approach is associated with large projects where the finished work is delivered as a complete package. An initial specification describing what the client requires, and sometimes how the job is to be done will be issued before implementation. Acceptance tests will be written into the contract. On delivery these acceptance tests will have to be satisfied before payment is made.

This form of software development naturally aligns itself to the Fixed Payment financial model. The usual sequence of events is as follows. The client (or his agent) writes a specification of the software to be delivered; the job is defined. The work is then put out to tender and the successful bidder is awarded a fixed price contract.

This approach places great emphasis on the specification stage because it is then that the customer must comprehensively, unequivocally and immutably set out what is to be delivered. However, once the contract has been agreed, any significant change demanded by the customer in the specification of the product has the effect of setting aside the obligations placed upon the provider.

This is the reason that many fixed price contracts run over-time and over-budget without the sanction  of legally enforceable damages. As soon as the change is requested, the terms of the original contract can fall away and the parties will enter what is essentially a cost-plus financial relationship or worse. In effect, the contractor no longer needs to produce the agreed product for the agreed cost unless the  customer forces completion of a contract for something he no longer really wants.

There is a school of thought that in the real world of project development the Big Bang approach fails because it is tied to the dependence on rigid initial specification. It is rarely possible to predict a full specification for a large project or even a medium sized project. Such predictions are however  fundamental to the form of contractual relationship imposed by the Fixed Payment financial model.

Of course, a customer will often have a vision of the product that he requires. This vision may be of a large project. In itself this is a legitimate expectation, there is nothing wrong per se with big visions. But, such a vision can rarely be worked out in its finest detail at the outset of its conception. To be tied to such a detailed concept inevitably disallows the useful tools of trial and error and serendipitous development.

Furthermore, to leave the testing of the project until the end of its productioni, invites disaster.

The Step-by-Step Approach

The step-by-step approach to software development is one in which functionality is developed and implemented in small steps. The direction in which the customer is proceeding may be envisioned as a goal but not as a clearly defined product with pre-ordained developmental procedures. The incremental steps towards the general goal may be well-defined micro-projects or they may be less well defined – the sort of activity that may be hidden under the term “maintenance”. However, with well-managed projects, each stage has a complete range of tests can be performed to ensure a working system.

The difficulty with this approach is that compared with the big-bang model, it is more difficult to put a job out to tender, harder to write a contract and even harder to fix a price for the contract. In the case of development that is carried out internally, these problems may be hidden. For both external and internal development, however, the problem is: how can resources be efficiently used and how can value for money be judged unless there is a clear idea of what is required?

The advantages of the step-by-step approach are

1. For each small step, size and complexity are also small.
2. When difficulties arise changes to functionality can more easily be changed
3. Important possibilities for improved functionality may be discovered
4. Changes in requirements can be incorporated quickly
5. Projects are much easier to terminate early.
6. Complete system testing can be performed at each stage in development(iiiii).

Also, when a modular approach to development is taken it becomes easier to change to another supplier if an existing one proves unsatisfactory. This is in sharp contrast with the big-bang / fixed-payment approach where there is a tendency for the customer to be locked into a relationship with the contractor. This shortcoming has recently been highlighted by the difficulties experienced by the Inland Revenue in its current attempt to find bidders for a partner to maintain its legacy systems and transform its business IT beyond the year 2004. These difficulties are described in some detail at a later stage in this study.

In summary, it is obvious that the step-by step approach does not easily fit into the Fixed Payment financial model. It naturally fits with the cost plus model, but that does not itself accord with the  current economic orthodoxy.

Large Software Project Failures

The failure of software projects has become a high profile issue. A recent example is the trouble that the Child Support Agency has experienced in the implementation of their software projectiv. In 1999, the Select Committee on Public Accountsv considered 25 large public software projects, which have failed seriously over the previous decade. They included

• The United Kingdom Passport Agency
• The National Insurance Recording System (NIRS)
• The Immigration and Nationality Directorate Casework Programme
• Benefits Agency: Jobseeker’s Allowance
• Crown Prosecution Service
• The Hospital Information Support Systems Initiative
• Northern Ireland Vehicle System Replacement Project
• Inland Revenue-Pay and File

Readers of the computing press and now even the broadsheets can name other projects that have failed, been cancelled or are seriously late and over budget: London Ambulance, DVLA, Air Traffic Control,  etc. Large successful software projects are more difficult to identify.

In the United States, the situation is much the same but there, more systematic work has been done in recording the failures. The CHAOS report in 1994vi by the Standish Group, which surveyed a sample  of 365 respondents and representing 8,380 applications, says

The Standish Group research shows a staggering 31.1% of projects will be cancelled before they ever get completed. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg.

The successful projects were 16% of the total. By the year 2000, they report elsewherevii, the percentage of successful projects had risen to 28%. During this period the abandoned projects had reduced from 31% to 23%. This dramatic improvement has thus seen progress from extremely bad to  very bad.

Project Size

A key factor that has been identified in the failure of software projects is sheer size. A more recent article in Software Magazine by Jim Johnson, chairman of the Standish Group saysviii “Most of these new [successful] projects are well within The Standish Group’s criteria established in “Recipe for Success, 1998,” which limits project duration to six months and  project staff to six people.”(ix)

Kent Beck, a software guru with long practical experience, states in his seminal book on software development methodology, Extreme Programmingx, that a team as big as ten programmers is “doable”. However, his methodology dictates that these programmers should work in pairs. This therefore would represent five independent programming units.

My experience over several decades suggests four programmers as being a practical maximum, albeit that there may be some additional support staff. Indeed, on a recent visit an experienced software developer/manager with one of the worlds biggest software companies put the maximum size for a successful project as five programmers.

Plans and Specifications

The problem here is that it is all but impossible for a large software project to be planned in a way that enables the final product can be clearly seen. So key is this issue that a group of software gurus have formed themselves into a group, the Agile Group and published a manifesto(xi).

This manifesto emphasises how difficult it is to have successful projects that rigidly define their outcome through detailed specification. This school of thought also emphasises that close collaboration with the customer should take precedence over contract negotiation and that responding to change is to be valued over following a plan.
One of the “Principles behind the Agile Manifesto”(xii) states

“[We] welcome changing requirements, even late in development.”

Another principle of the Agile Group is to “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” These principles are in line with the judgement in the introduction that “the most successful IT projects are those that start small and are incrementally developed over a period of time, with significant testing at each stage.” Indeed, the Extreme Programming method requires the design of a suite of automatic tests. The full range of tests is applied at every stage in the incremental programming steps. When a change to the project software is proposed its validation tests are added to the project test suite prior to the work on that change commencing.

It is worthy of note that the main protagonists in the Agile Group are hands-on workers within the industry rather than academics.

Why do Software Companies accept he Big-Bang Approach

For those tasked with the responsibility for procurement, it is easy to make the case for one large project that has clearly specified objectives and in which a fixed payment is made on completion of acceptance tests. It is an attractive option for all involved. The shareholder or, in the case of public projects, the taxpayer is offered a transparent process that promises value and assurances of quality.

For the procurer the process of putting well-specified projects out to tender provides the shield of delegated responsibility. If things go wrong it cannot be the fault of his department if the procedure has been followed in the correct fashion.

The problem with this approach for large projects is that, once a contractor has been accepted, that contractor often becomes a monopoly supplier. The more important the project is to the customer, the more power is given to the contractor. He benefits by being locked into the project. For a casein point  see Appendix 1, “The Power of an Incumbent Supplier”

Even in those instances where a project is detected as going awry during the development stage there are strong incentives to stay with a supplier. The financial cost of restarting with another provider may be prohibitive. The loss of time may be inconvenient. The expertise accrued by the original supplier will be lost if there is a change. Furthermore, if the fault lies with the initial specification then it lies in the interests of the procurement department to avoid rocking the boat lest any of their staff should fall overboard.

Staff of the procurers of software development may also have an incentive to go for a Big Bang approach. The involvement of client staff is likely to be greater in a Step-by-Step approach. This is often not welcome for those with a heavy workload. More importantly, they perceive it as not allowing enough distance between themselves and the software project. With the current track record of  software development, who would wish to be closely associated with it?

In the Big-Bang approach, responsibility for the failure of the project can be passed to the contractor who in turn will blame the inadequacies of the initial specification. These inadequacies will in their turn be palmed off as resulting from changes in circumstances that have arisen after the initial specification was written. There is in this merry-go-round no culpability other than that ascribed to the unpredictable hand of fate. It is for this reason that despite the manifest failure of so many projects there are few reports of individuals or companies shouldering the responsibility for these failures.

Ultimately the losers in this system are the shareholders or taxpayers who fund it and the ultimate consumers who suffer from degraded services.

Resolving the Specification Problem

There is a fundamental problem in the production of effective software. It is the problem that bedevils any attempt accurately and fully to specify a finished software product before the shaping of that product has begun. The problem is that the full specification of a product is to be found in the codes that make up the program itself. Any specification will fall short of defining absolutely and exhaustively the steps that will take it from a construct of language to the rigorous syntax of a computer program. Even the simplest of tasks, if described in terms of their every changing condition, take on an unexpected complexity. It is these unexpected complexities that prove the cause of the failure of the “up-front” specification process. It is worthy of note that that Joel Spolsky, a manager on the Microsoft Excel project said that his specifications changed on a daily basis(xiii).

The dilemma is this. On one side there is the economic orthodoxy (with some reason) that requires specification as an integral part of the procurement process. On the other side there is the real world experience that projects based on the initial specification model simply do not deliver the goods.

In my view the problem can only be resolved in the following way.

• First, we must abandon reliance upon the requirement for full initial specification. It does not and cannot work.
• Second there must be a rapport between the customer and the developing product itself. Given the complexities inherent in translating real world requirements into a computer program, every effort should be made to remove all impediments in the process of communication.
• Third, there must in place of an initial “up-front” specification be a continuous dialogue between the customer and the provider. It is only in this way that the fuzzy edge of real-world requirements can be fitted to the structure of a computer program.

The conclusion is that, in the majority of cases of software development, the Step by Step approach is the correct one to follow but that care must be taken in its management. The following steps should be taken:

1. A goal with an appropriate business purpose should be set. This can be described in general terms and should be agreed on the basis that its form may be subject to change. Although it may be desirable to set a wider context, this goal should be at most six months and employ a maximum of four programmers.
2. In planning for this goal, developer and customerxiv should jointly identify significant landmarks, each of a small size, which ideally should be chosen to have identifiable business value.
3. Each landmark should be developed as a series of smaller programming steps. The developers should write automatic tests for each of these steps. The whole system should be checked against the whole test suite after each step is implemented.
4. Frequent functional evaluation should take place in conjunction with the customer.

The main feature of this process is that development grows incrementally and at each stage there is a working system.

Measures that can help with the Cost Plus Model

The problem with the development that I have proposed is the use of the Cost Plus model. In the minds of those that know little of the process of software development the Fixed Payment model seems much more attractive because it holds out the promise of market driven value for money. However, the Fixed Payment system skews the development process by putting too much emphasis upon the requirement for a fixed initial specification. It is the financial model of the Big Bang approach.

But the Step by Step approach is not suited to the Fixed Payment model. It fits with the Cost Plus model and seems therefore, to invite wasted costs and a lack of expediency in the completion of a project. But real world experience of IT project disasters leads to the conclusion that the Big Bang approach never works for large projects, leaving the Step by Step approach as the only workable solution. This means large projects must use some form of Cost Plus model.

It may however, be possible to have measures, which get better value for money than previously. These measures may include some or all of the following

Increase the responsibility of in-house staff in the development of projects. If this is done, then the steps taken in that development should rise no higher than the levels of skill that the staff believe that they have. Of course, the incremental approach will reduce both the possibility of failure and the weight of responsibility that would follow in its wake. This approach requires a high degree of consultation and co-operation from the staff.

If outside consultants and contractors are charged with the primary responsibility for development there should be close contact with in-house staff. This will serve a number of functions. It will provide training for future development work and will familiarise staff with the new product. Furthermore, it will place the outside contractor under the constant scrutiny of the representative of those by whom he is to be paid.

Ultimately the customer should establish a small pool of suppliers with whom he has established a relationship of trust. Each supplier should be made aware of the fact that he is not the sole provider and cannot therefore develop a locked-in relationship with the client. In this way, the benefit of competition will work within the pool of suppliers in order to drive down cost and enhance efficiency. It must be accepted that from time to time a supplier will, despite his best endeavours, fail to deliver. If the reasons for this failure do not adversely reflect upon the general competence of the supplier then he need not be excluded from the pool. Partnerships between the supplier and client should be developed over a period of time within which factors of trust and the development of symbiotic working practices are given primary importance.

Whenever a project is seen to be heading towards a dead end, there should be no hesitation in shutting it down. The great benefit of the step-by-step approach is that it reduces the risk of such an outcome and mitigates the loss when it does occur.

Postscript

Government, under the guidance of the Office of Government Commerce has developed the Gateway Review process to help the management of IT projects. It is a tightly controlled process of some 30 stages to make large Big Bang projects work.

It is a matter for debate whether problems arising in tightly managed projects have as their solution an even tighter framework.

Geoff Beacon
May 2002
Aug 2003

Acknowledgement

Thanks to Robert Collins for redrafting this report.

Appendix 1 – the Power of an Incumbent Supplier

A good example of the power of an incumbent supplier is the case of the contract to run the Inland Revenue’s IT operations. According to Computer Weekly for the 21st March 2002, a ten-year contract, which was valued at £1bn in 1995 was awarded to EDS but the cost of the contract has risen to £2.8bn. This is £400 million more than in April of the year 2000 when MP’s expressed concerns about rising costs. Computer Weekly also reports that the Inland Revenue is so concerned by the monopoly held by its supplier that it is offering millions of pounds to major computer companies in an effort to persuade them to enter bids. So difficult will it be for an outsider to pick up the threads of the work, that few if any are expected to respond. Another aspect of the situation, even more disturbing in the spectre that it raises is described in the article in the following terms:

The Revenue says the risk of lock in to EDS is manageable. Its chairman Nick Montagu has told MP’s ‘we will make sure that the competition is not just a token to meet European requirements, but a proper one’.

IT suppliers say the only serious rival to EDS was Accenture (formerly Anderson Computing), which runs the Revenue’s national insurance recording system, Nirs2. However, Computer Weekly has discovered that EDS and Andersen are planning a joint bid for the new contract.

Some IT executives believe that such a bid would be unassailable, partly because of the risks to the continuity of service if the nation’s tax and insurance systems are passed to a company with no experience of running both systems.

If such a joint bid is proffered there will be, despite Mr Montagu’s assurances, no competition either real or by way of a token as a sop to European legislation.

Appendix 2 – Review of Major Government IT Projects

The Cabinet Office report Successful IT: Modernising Government in Action, published in May 2000, gave an analysis of the problems of software procurement. By and large, their focus is concentrated on government IT projects that are much greater than medium sized companies would contemplate commissioning. However, they make recommendations, which are ostensibly similar to those in this report. Under the heading “Making large projects more manageable” they state

Governments in the UK and abroad and the private sector have recognised that an effective way to reduce risk is to break large projects into smaller, more manageable components.

However, they give the following as evidence

Evidence: A large insurance company undertook a project in which the staffing grew to such an extent that the management overheads were not worthwhile and communication became difficult. The company now insists that senior management can see ‘both ends of the tunnel’ at all times. This is achieved by making all projects modular, for example, letting them run for no more than 18 months and involving no more than 50 staff.

This is well over the limits described above – a maximum of five staff for a project (this may be doubled for paired programming).

Clearly, the authors of the report are not coming down to the scale of project used with Agile Programming methods.
Indeed, there is a clear reason why the Agile Programming approach would be rejected by the authors of this report. It is that they insist that the benefits of a project should be understandable before a project starts and delivering these benefits, and these benefits only, should be the measure of success. They state

6.1 A project or programme is only successful if it delivers the benefits for which it was initiated.

and

Recommendation 15: A post-implementation review must be undertaken of all projects or programmes and benefits realised assessed against projected benefits outlined in the original business case or subsequent amendments.

The conclusion drawn from these points is more control from the Centre

6.6 Money for large projects or programmes is authorised by HM Treasury for the realisation of a specific set of benefits contained within a business case. Therefore, the Centre has a key role to play in monitoring the extent to which those benefits are, in fact, delivered.

Clearly any project, which starts with a business case that must be put to the Treasury, cannot be a small one. Both the scale of such a project and the “projected benefits only” rule means there is little scope for the customer and programming team getting together to “work things out as they go along”. The authors do, however allow for this possibility in another way – the use of prototypes. Under the heading “Purchase of preparatory work” they say

5.18 During the pre-contract phase, one process that can help to firm up requirements is the use of prototypes. … Most are throwaway systems that are built as cheaply as possible to help to clarify requirements or prove a concept. Due to the short-term nature of these prototypes, they are unlikely to be suitable for procurement under a PFI arrangement.
…To encourage the bidders to follow this approach, each has been paid £100,000 from [a UK government] agency’s own funds. The agency views this as an acceptable price for the risk reduction it offers.

The question the authors do not answer is “What should be done if the prototypes actually work?” One of methods of Agile Programming is to progress through a series of small stages, which should be production quality code and should deliver business benefit. Typically, these stages will cost a fraction of the £100,000 allocation the agency set for each prototype.

Notes:

(i) It is often the case that a large or medium sized project is divided into sub-projects, which are integrated towards the end of the project. Testing at this stage, integration testing, often uncovers many unexpected difficulties.

(ii) It is characteristic of the best step-by-step methodologies that each small step is integrated with the whole system and is tested against all the current system tests.

(iv) Child Support Agency reforms delayed, BBC News Online Wednesday, 20 March, 2002.

(v) House of Commons, Public Accounts – First Report, Session 1999-2000.

(vi) The CHAOS Report (1994), Standish Group

(vii) Collaborating on Project Success Jim Johnson, Karen D. Boucher, Kyle Connors, and James Robinson, Software Magazine, February/March 2001.

(viii) Op. cit.

(ix) This article is based on information from the company’s latest paper, “Extreme CHAOS 2001.”

(x) “Extreme Programming Explained: Embrace Change”, Kent Beck, Addison Wesley, 2000

(xi) Manifesto for Agile Software Development

These principles can be found at http://agilemanifesto.org/principles.html

(xii)These principles can be found at http://agilemanifesto.org/principles.html

Principles behind the Agile Manifesto:

We follow these principles:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals.
• Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development.
• The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity–the art of maximizing the amount of work not done–is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

(xiii) Joel Spolsky is a strong advocate of the prominent use of specifications. He explains this colourfully on Joel’s website, . But he also says Specs Need To Stay Alive. Some programming teams adopt a “waterfall” mentality: we will design the program all at once, write a spec, print it, and throw it over the wall at the programmers and go home. All I have to say is: “Ha ha ha ha ha ha ha ha!”

But he is using the term specification in a wider sense. As program manager, his specifications are being continually written and re-written throughout the project. He also accepts that programmers will sometimes refuse to implement certain things so he changes his specifications. In this paper, unless stated otherwise, I mean “up-front” specification – the type that is required by the Fixed Payment financial model.

(xiv) Ideally a single individual, who understands the business requirements, and who works closely with the programming team, should represent the customer and be in charge of the project. Such an individual is called a “Senior Responsible Owner” in the report “Successful IT, Modernising Government in Action” by the OGC.