Software Failures

JKS & Associates

Software Failures

Software Consulting

Up

Why do software development projects fail so often?

You've either seen them or belonged to a team that was on a failed software project. The project seemed to be properly sponsored and, in theory, should have been delivered on time and on budget to the customer. Somehow, the project took a life on its own and quickly went out of control.

The Standish Group, a research organization in Massachusetts, has been conducting an on-going survey (1) since 1993 on software and information technology failures. It is called the "Chaos Study." The study is trying to determine "the scope of software project failures, the major factors that cause software projects to fail, and the key factors that can reduce project failures." The study involves 365 respondents and represented 8,380 applications. This is a big study. In 1995, the Standish Group estimated that US development houses will spend 81 billion for cancelled software projects.

What the study has found is rather thought-provoking and interesting. They have found that only about 16% of the software and IT projects initiated by the responding companies will be completed successfully. Success is defined as on-time and on-budget. About 53% of the projects were over budget and behind schedule, while 31% were deemed failures (cancelled).

Cost overruns averaged 189% of the original cost estimate for all the projects. Time overruns averaged 222% of the original time estimate. Only 61% of the originally specified features and functions (requirements) were available on the released project. Re-starts were common on many of the failed projects.

What Can Be Done To Prevent This From Occurring?

In order to prevent future failures we must learn from our successful projects and attempt to incorporate these "good" factors in future projects. These success factors must be recorded and passed on to other projects and to new project managers. A good way to do this is to perform a project post mortem after the completion of each project and record the methods, measures and data that made the project a success. In short, these "good" practices must be institutionalized. With past project failures we need to identify those factors that caused the project to fail and prevent these factors from arising in the future. We must document and perform post mortems on failed projects in order to prevent repeating these "bad" behaviors. Software history is a valuable teaching tool for us.

The Standish study identified several factors that would improve a project's chance of success. The top four responses included : 1) User Involvement, 2) Executive Management Support, 3) Clear Statement of Requirements, and 4) Proper Planning. The study participants ranked Incomplete Requirements and Lack Of User Involvement as two most common factors that cause projects to fail or be cancelled.

These project success and failure factors can be used as a checklist to determine if a new project has the potential of being successful. Other findings by the Standish group "suggest that reducing the time frames and delivery of software components early and often, will increase the success rate" as well. The shortening of the development time frames often causes developers to adopt a more iterative form of development, which results in more prototyping, designing, testing, and fixing. Since the software components are smaller, it is easier to estimate, manage, and plan. Smaller components are also less complex, which has the additional benefit of making the overall software project simpler. In my consulting experience I have seen the detrimental effect of complex applications and making software too complex.

Software Runaways

Software author Robert Glass recently published an interesting book on software runaways (2). Glass defines a runaway project as "that which goes out of control primarily because of the difficulty of building the software needed by the system." "Out of control" is taken to mean "schedule, cost, or functionality that was twice as bad as the original estimates." This book profiles 16 of the largest and most highly publicized software failures over the last decade. Familiar development projects that were profiled included the Denver Airport baggage handling system, the IRS modernization project and the American-Hilton-Marriott-Budget Confirm project. Glass does a tremendous job at dissecting the projects and identifying the cause or set of causes that contributed to each of the project’s downfall. The book is laid out in a format that lists the major failures types and illustrates each failure with one of the 16 failed software projects.

The predictable findings from the various projects included (2) :

Many of the runaway projects are (or were) "overly ambitious." It is well known in the field that large projects are problematic.

Most of the projects failed from a multiplicity of causes. There may or may not have been a dominant cause, but there were several problems contributing to many of the runaways.

Management problems were more frequently a dominant cause than technical problems. But see the list of surprising findings below.

Schedule overruns were more common (89 percent) than cost overruns (62 percent).

The surprising findings from the study included (2) :

Survey respondents thought that there would be more runaways in the government and financial sectors, and fewer in service and manufacturing. But the survey findings found all such sectors equally susceptible.

Respondents were optimistic about the trend in runaways; 42 percent believed they would decrease in number, while only 8 percent felt they would increase.

The use of packaged software did not help in reducing the incidence of runaways. Of the runaway projects studied, 47 percent consisted of mixed custom and packaged software; 24 percent were custom software; and 22 percent were packaged software.

Runaway projects showed their true colors early in the project history. More than half started showing symptoms during system development, and 25 percent showed those symptoms during initial planning.

In spite of the above, visibility into the existence of a runaway came first of all from the project team (72 percent); only 19 percent were spotted initially at the senior management level.

Technology is dramatically increasing as a cause of runaways. "Technology new to the organization" was the fourth most common problem in the runaway projects.

Risk management appears more and more frequently in the software management literature. But 55 percent of the runaway projects had not performed any risk management, and of those 38 percent who did (some respondees did not know whether it was used or not), half of them did not use the risk findings once the project was underway.

Summary of Causes of Runaways

Glass summarizes the analysis of these failures in the following format. It is interesting to note that many of the failure factors are similar to the findings in the Standish Chaos Study. Runaway projects can therefore be categorized by their primary cause which include (2) :

Project Objectives Not Fully Specified-51 percent

Bad Planning and Estimating-48 percent

Technology New to the Organization-45 percent

Inadequate/No Project Management Methodology-42 percent

Insufficient Senior Staff on the Team-42 percent

Poor Performance by Suppliers of Hardware/Software-42 percent

Other-Performance (Efficiency) Problems-42 percent

Just like what was found in the Standish Chaos Study, project requirements/user involvement is the number one problem for these failed projects. The typical problems with requirements that I see are similar to what Glass has described as well. This includes too many requirements, unstable or constantly changing requirements, ambiguous requirements or incomplete requirements. In most organizations I find that incomplete or missing requirements seems to be the most prevalent requirement's issue. The second major study finding has to do with project planning and estimation, which are activities typically performed by management. These categories are similar to the Executive Management Support and Proper Planning found in the Chaos Study. Somehow we are just too optimistic when it comes to delivering software. We have still failed to nail down the schedule issue. Most of my experiences seem to concur with Glass’s findings. I rarely see organizations using proper scheduling techniques; often schedules are determined by marketing with no justification and attention to what has to occur to build the system.

Other Pertinent Observations By Glass

Here are some additional observations by Glass on these runaway projects (2). These observations get right to the heart of the problem in many instances. Also, they provide insight and a simple strategy for preventing software failures or runaways in the future if properly acted upon.

"The most common problem in building software systems is not the construction of them itself, but rather the estimation of the costs of that construction."

"It is all too common for a software project to fail to meet its costs and schedule targets, because those targets themselves were simply (and grossly!) wrong."

"For at least a decade, the software industry has heard the advice "do a post mortem." This advice falls on deaf ears usually. However, it is only through analyzing our successes and failures that we can possibly improve."

"A project is a death march if the parameters for that project exceed the norm by at least 50 percent."

"Because we know so little about accurate schedule estimation, and because estimates are usually made by the people who are least able to make accurate estimates (e.g. marketers and customers), it is simply the norm that schedule targets and thus targets, are unreasonably short. Thus project achievement of them is at best problematic."

Suggestions For Reducing Software Runaways

One of the best techniques to reduce software runaways, according to Glass, is to perform Software Risk Management. Risk management is a method that anticipates the most serious problems that could occur on a project, and provides ways to mitigate the risk and lists the necessary steps to handle them. Software measurement expert, Caper Jones, has identified some of the most serious software risks. They include (2) :

Inaccurate metrics

Inadequate measurements

Excessive schedule pressure

Management malpractice

Inaccurate cost estimating

Silver bullet syndrome

Creeping user requirements

Low quality

Low productivity

Canceled projects

The second method to reduce software runaways is to perform proper Issue Management. Glass defines "issues" as obstacles that tend to arise that threaten to disrupt project progress. He further states that "risks are problems that are (or at least may be) anticipated prior to the beginning of project work. Issues, on the other hand, are problems that arise while the project is in process. If all issues could be identified in advance, they would probably be addressed as risks. But, given the complexity of software construction, there will always be plenty of (surprise) issues arising as work proceeds." Issue management is essentially project management by responding properly to issues.

Remedies That May Be Attempted During Runaways

Risk and issue management have been considered more academic approaches to preventing software runaways. But what should you do if you are suddenly faced with a software runaway? Some possible solutions that you might implement according to respondents to a KPMG study on software failures in the United Kingdom are (2) :

Extending the schedule (the paper's authors called this "more time")-85 percent

Better project management procedures-54 percent

More people-53 percent

More funds-43 percent

Pressure on suppliers by withholding payment-38 percent

Reduction in scope of project-28 percent

New outside help-27 percent

Better development methodologies-25 percent

Pressure on suppliers by threat of litigation-20 percent

Change of technology used on the project-13 percent

Abandoning the project-9 percent

Other-9 percent

Lessons Learned and Action Plans For Your Organization

What can you learn from these two studies? I believe a number of things. Here are some for you to think about, digest, modify and use in your organization.

1. Be very cautious if you are going to build an unprecedented software system. Don’t try to integrate or perform the massive big bang build at the end of a project for a large and complex system. Prototype sections of the system and incrementally integrate to reduce the risk of failure.

2. If you are new to the area of software improvement, focus on requirements management and project planning/estimation if you want to really improve your software development efforts. These items will allow you to leverage the most on a given project. They are the two most cited reasons for software failures.

3. As Glass explains, always consider Hofstader’s Law. This law states that software development always takes longer than you think, even when you consider Hofstader’s Law.

4. Use a detached neutral observer (or external consultant) to review your project if you think it may be going down the wrong road. Don't wait to the end of the project to review the status with this neutral observer.

5. Use defined processes and methods to evaluate the software capability of a supplier or out-sourcing organization. Check their track record on previous projects. The Software Capability Evaluation (SCE) or the Software Acquisition Capability Maturity Model (SA-CMM) defined by the Software Engineering Institute (SEI) are formal and rigorous ways in evaluating software organizations who want to build an application for you.

6. Avoid large protracted time frames. Use smaller development time cycles and smaller deliverables. They are easier to build and manage.

7. Higher the best people. Don’t try to use cheap untrained or in-experienced help to do the project. You really get what you pay for.

8. Manage your contractors and sub-contractors carefully. Make sure that accountability is defined at all levels before the project starts. Track your progress in a systematic and routine (weekly or even more frequently if necessary) manner.

References

1. Standish Research Paper, Chaos Study, 1995, available on-line at http://www.standishgroup.com

2. Software Runaways : Lessons Learned from Massive Software Project Failures, Robert L. Glass, Prentice-Hall, 1998.

� 1998-2004 by John Suzuki
Last Revised : 9/15/2004

[Up] [Home Page]