Measuring Project Success and Managing Expectations by Ivar Jacobson

There are a number of studies that cite poor performance of software projects - The Standish Group being the authors of one of the more often cited, their Chaos Report (and old version from 1995 is posted here, and although the data is old the conclusions are not dramatically changed).  The gist of these studies is that the majority of projects (as high as 70%) fail when measured against original schedule, budget, and expected features. I would be the last to argue the general conclusion: that it is very hard to manage a project to success. 

Most projects lack clear direction and purpose, and many are rife with disagreements about what success looks like.  There is, however, something in the assumptions behind these studies that rings hollow: that the initial schedule, budget and expectations for projects is a reliable milepost against which to measure. Most projects are vaguely conceived at best - they often lack a clear understanding of why they should exist and what problems they need to solve.  At their initiation they are usually poorly scoped and vaguely purposed, and the funding associated with them is often assigned based on an  allocation of an arbitrarily assigned budget.  Their schedules, at least those produced at the start of the project, are largely speculative endeavors, a mixture of gut and guesswork, that bears little basis in reality.  Measuring project performance against the initial schedule, scope and budget is of little value except to illustrate the point that there is a large disconnect between the expectations of business sponsors and the ability of teams to deliver against those expectations.  There are, to be sure, rampant problems with performance, but there are also widespread woes of expectations that are just as important to address.

Where should we start?  The first place is probably with project funding and measurement. The real thing of importance to measure is whether the project produced (or exceeded) the business value  expected of it.  If a delay in the project caused a market window of opportunity to be missed, that is significant, but it is the decline in value delivered that needs to be measured, not a schedule variance that cannot be correlated with economic activity.  Forcing a focus on business value produced would also put the right attention on the role of the business in following-through on their assertions of the value that will accrue from having the solution.  Requesting projects based on business needs has an opportunity cost - choosing one project over another should affect the value delivered to shareholders - and accountability for assertions by the line of business is just as important as accountability for project delivery.

If we shift our attention to value delivered rather than meeting schedule and budget, we may free the development team to find better ways to deliver the value, which may or may not include the initial set of features envisioned by the business sponsor.  Initial feature lists are usually vaguely conceived and don't provide a very good target for delivery.  Work is usually required to ascertain the real needs from this initial list of "features", some of which contribute to satisfying real needs but many of which are simply good initial starting points for discussion about real needs.  It may very well take longer than expected to solve the real problems (it usually does, as we all tend to be more optimistic than we should about how long things will take).

The problem is that most teams are set up to fail from the start.  By measuring them against budgets and schedules based on arbitrary assumptions and often a poor understanding of the real business value that needs to be produced, we find them constantly struggling against a plan that cannot possibly succeed.   Measuring against initial schedule, budget and expected features is not merely meaningless, it's actually part of the problem.  We need to shift our focus to better articulating problems to be solved and needs to be satisfied, and measuring business value produced.  Once we start to do that, we can focus on the plans and milestones needed to ensure the delivery of business value.

2 Comments
  1. Kurt Bittner | March 20, 2008 at 7:38 pm Reply

    avatar

    Yes – there is little that can be done to guard against poor planning on the part of the business. What I have found useful in cases like this is to get very clear about what outcomes need to be delivered in those short time frames, and then to work backwards to figure out what needs to be developed. Forcing ranking of outcomes really helps, too. Then you come up with what needs to be developed to deliver the “must” outcomes.

    From there a rough estimate of the effort required to deliver the outcomes you will either know whether it can be done or whether some of the outcomes need to be dropped from the plan.

    I think it is better to time-box – i.e. deliver something in the 1-month timebox – than to ask for more time. If you identify outcome and then scope based on delivering those outcomes it will give you a way to decide what really needs to be done.

  2. claire mcfarlen | March 19, 2008 at 7:51 pm Reply

    avatar

    Even bigger problem is having no opportunity to realstically develop budget and schedule when the first stated requirement is “Now I need this by the end of next month”. So then everything after that is about either a) managing the client and begging for more time, and b) manipulating the deliverables to conform to old stuff built for someone else that can be cloned. Alternatetive? “OK, I’ll just outsource it or buy a vendor product” which will promise much and maybe deliver something (whether was what they wanted or not does not matter so much), for big cheque….