Measuring Project Success and Managing Expectations

March 3rd, 2008

There are a number of studies that cite poor performance of software projects - The Standish Group being the authors of one of the more often cited, their Chaos Report (and old version from 1995 is posted here, and although the data is old the conclusions are not dramatically changed).  The gist of these studies is that the majority of projects (as high as 70%) fail when measured against original schedule, budget, and expected features. I would be the last to argue the general conclusion: that it is very hard to manage a project to success. 

Most projects lack clear direction and purpose, and many are rife with disagreements about what success looks like.  There is, however, something in the assumptions behind these studies that rings hollow: that the initial schedule, budget and expectations for projects is a reliable milepost against which to measure. Most projects are vaguely conceived at best - they often lack a clear understanding of why they should exist and what problems they need to solve.  At their initiation they are usually poorly scoped and vaguely purposed, and the funding associated with them is often assigned based on an  allocation of an arbitrarily assigned budget.  Their schedules, at least those produced at the start of the project, are largely speculative endeavors, a mixture of gut and guesswork, that bears little basis in reality.  Measuring project performance against the initial schedule, scope and budget is of little value except to illustrate the point that there is a large disconnect between the expectations of business sponsors and the ability of teams to deliver against those expectations.  There are, to be sure, rampant problems with performance, but there are also widespread woes of expectations that are just as important to address.

Where should we start?  The first place is probably with project funding and measurement. The real thing of importance to measure is whether the project produced (or exceeded) the business value  expected of it.  If a delay in the project caused a market window of opportunity to be missed, that is significant, but it is the decline in value delivered that needs to be measured, not a schedule variance that cannot be correlated with economic activity.  Forcing a focus on business value produced would also put the right attention on the role of the business in following-through on their assertions of the value that will accrue from having the solution.  Requesting projects based on business needs has an opportunity cost - choosing one project over another should affect the value delivered to shareholders - and accountability for assertions by the line of business is just as important as accountability for project delivery.

If we shift our attention to value delivered rather than meeting schedule and budget, we may free the development team to find better ways to deliver the value, which may or may not include the initial set of features envisioned by the business sponsor.  Initial feature lists are usually vaguely conceived and don't provide a very good target for delivery.  Work is usually required to ascertain the real needs from this initial list of "features", some of which contribute to satisfying real needs but many of which are simply good initial starting points for discussion about real needs.  It may very well take longer than expected to solve the real problems (it usually does, as we all tend to be more optimistic than we should about how long things will take).

The problem is that most teams are set up to fail from the start.  By measuring them against budgets and schedules based on arbitrary assumptions and often a poor understanding of the real business value that needs to be produced, we find them constantly struggling against a plan that cannot possibly succeed.   Measuring against initial schedule, budget and expected features is not merely meaningless, it's actually part of the problem.  We need to shift our focus to better articulating problems to be solved and needs to be satisfied, and measuring business value produced.  Once we start to do that, we can focus on the plans and milestones needed to ensure the delivery of business value.

Theory X and Theory Y

February 18th, 2008

Management styles have a huge effect on software development teams.  A significant shift required in  the movement toward a more agile approach is a change in the way that teams are managed and measured.Management styles can be described in  a number of ways - Theory X and Theory Y are popular approaches dating back to the 1960's but still applicable today.   Basically, Theory X assumes that people are basically untrustworthy slackers who need to be constantly monitored and told what to do and are only working because they need the money.  Theory Y assumes that people want to do a good job and are motivated by more than money, and that people produce their best results when they are working in a supportive environment that frees them to be creative and productive.It should come as no surprise that a necessary condition for a movement toward agile teaming is a "Theory Y" management culture - a team trying to adopt an agile approach in an organization with a "Theory X" management culture is doomed to failure and frustration; it will constantly be fighting the management system and the surrounding culture. In these organizations, the management culture (and its supporting measurement system) must change along with the team approach in order for both to be successful. Before you can change it, however, you need to understand it.

Agile or not Agile – that is the question ? Or is it ?

February 4th, 2008

I often get called into companies who are thinking of going "Agile".  They have invested many thousands of dollars on a very complex SDLC taking the best ideas from the industry, but that process is not being followed and their teams are effectively not working together, and now think that Agile will solve those problems.  They come to IJC with a simple question "how do I adopt agile ?".  But when you dig, you find that this question is not as easy as it would appear.  When you question their motivation and their constraints, you find a whole list of issues and problems that Agile by itself can not solve.  Issues can range from off-shoring, to quality and performance.  Issues that no one approach can solve.

Agile software development seems to be a way of describing everything that is good in software process today.  It combines techniques and team practices with ways of changing organizations.  Thus, when people talk about agile they are talking about so many different things it is really hard to get a handle on them all - it is like saying I like European food and everyone knows what that means.  Do I mean Spanish, Swedish, French...?  Of course I can not mean English.  There is a lot of great stuff that has the label Agile, but the area that I will discuss today are the team techniques that Agile has brought to the table.  In particular SCRUM. 

First let's define a structure that SCRUM fits into.  SCRUM provides a fantastic set of very simple project management processes that help teams better function, but it does not provide great guidance in the areas of actually building software or making sure that project fits into something much larger.  Thus, I always position SCRUM in the context of three other processes.  Firstly, on top is an organizational process.  These are typically described in SDLC milestones, gates or phases.  They represent the key stages a project MUST go through from a funding and control point of view.  The second view of process is the team.  How does the team function to deliver software in support of this lifecycle?  This I label as team.  The third is the techniques that an individual must employ to actually build software.  Techniques such as OOAD or Test Driven Development - Also this is where techniques such as Use Case Driven development fit. SCRUM fits nicely into the middle layer - It is a set of team techniques that really help the team become a team.  But without the top and bottom views of process being in place SCRUM by itself would not work. 

Firstly SCRUM provides a very simple set of roles.  Product Owner (the go to person who owns the problem you are solving), the SCRUM Master (the person who runs the SCRUM meetings and protects the team) and the Development Team (the people that do the work).  I also often add a fourth role that of the Technical Owner, the person who owns the system from a technical point of view and is the go to person about all things about the architecture.  Secondly it provides three meetings.  A kick off meeting at the start of a sprint.  Oh, I had better introduce the idea of a sprint - A sprint is like an iteration.  A small chunk of time when stuff happens - has a clear set of goals and delivers stuff.  In terms of what it delivers they are classically stories, but can also be features or other units of stuff.  What is important about a sprint is that real work is done, that means working software is produced, tested and deployed (maybe to a pseudo production environment).  The second type of meeting, and the one that gets all the press is the daily Scrum - this meeting is short, say 15 minutes, and focused on three questions.  How did you do, what are you doing and what is stopping you?  These three questions enable the project to move along at a rapid, focused rate.  The third meeting is all about retrospective and proving you did what you were promising in your sprint.   So, SCRUM provides a great way of organizing the team, but without the lifecycle the team can not decide on what sprints they will do and the management can not appreciate the value of this sprint in the context of something bigger.  Without some techniques for building software individuals do not know what to do. 

So SCRUM must always be part of something bigger to really make an impact.  I would therefore argue that agile on its own is like a single food item; really to enjoy it, it needs to be put into the context of a meal.

Welcome to my new blog!

February 2nd, 2008

For many years I have written postcards to those of you who have subscribed to them on our web sites.  I will discontinue writing the postcards, but instead talk about the many different aspects on software development on this blog.  And I am not the only one who will write for you.  I am so lucky because I am surrounded by world-class experts coming from around the world.  These people will publish notes here.  Thus, you will here from Kurt Bittner, Ian Spence, Svante Lidman, Pan Wei Ng and several other people.  I will introduce them as they come forward.We will use this blog to talk about our visions and our results.

I like to dream about the future, but I also would like to reach that future at some point in time.  I believe that when it comes to software, dreams only come true if the dreamer drives the realization of the dream.  Too many people have dreams.  Very few actually have the guts to make their dreams a reality.

In 1992 I wrote an editorial in IEEE Software that started like this:  “We all have dreams. Just a little over 25 years ago, I had a dream. I dreamed that the technology known today as "object- oriented" could be used to design telecommunications systems. I dreamed that this technology would be generalized and given a wealth of wholly different applications. I dreamed that the computer scientists who - in those days - ridiculed this technology and saw it as a dead-end street would one day accept it. I dreamed that this technology would in time become an indispensible element in the mainstream of computer science.”

In my summary I wrote: “Our long-term goal (which I hope we can achieve within another 25-year period) is to become as industrial in our handling of computer techniques as engineers are within their various disciplines. This requires, among other things, development of the architecture in large- scale object-oriented systems; it requires that the development process support project management as well as product management; and it requires that we find other, more sophisticated languages - specification languages as well as programming languages - compared with those we have today.”

Now 15 years after I wrote this article, it is interesting to see how far we have gotten.  We have got UML which is a standard modeling language for software development.  Before UML we had lots of different notations with almost no semantics.  UML basically took all of these notations out of the picture and replaced them with one single notation with reasonable well-defined semantics.  It is not perfect but it is practical.  It is being applied around the world for all kinds of software and for all kinds of models including business, requirements, architecture, analysis and design. 

When it comes to process and methodology, we have also made substantial progress.  When I wrote the IEEE article there were more than 26 published methods identified by the OMG (Object Management Group).  The most successful one of these methods (and tragically basically the only one that really survived) was Objectory which became the Unified Process. 

In parallel with the success of the Unified Process, new kinds of methods have come into play.  The most successful of these other methods are labeled agile.  However, the meaning of agile has been confused by its proponents.  What really is unique about agile is that it cares about people.  Thus it is about social engineering.  No process has ever developed software.  Only people develop software.  They may get help by tools.  They may get advices from process or methodology, but eventually we must make our people motivated and competent.  Nothing is as effective in getting software quickly and at low cost as having motivated and competent people.

Given that no methodology has everything you need, and even if it has most of what you need, it won’t be the methodology you want to use in ten years time.  New ideas and new technologies will come into play.  These will come from individuals and companies throughout the world.  If you try to extend your methodology with these ideas, it will grow until it collapses under its own weight. Instead of publishing branded complete processes or methodologies we should focus our skills on single practices.  Usually a methodologist is good on one or a few practices, but not on all practices that a team needs when developing software. 

Now it is time to formulate one of my new dreams.  My dream is that great people around the world will contribute their best practices to the world maybe in the form of a community.  These practices can be of different kinds.  They can be technical such as those from the Unified Process camp, or they can be social engineering practices from the Agile camp.  A team can then select from these practices and compose the selected practices into a way of working that helps the team develop good software, quickly and at low cost.  This is my dream.  Now, we are not just dreaming.  We have come a long way in realizing this dream.  We have developed a practice platform EssWork where you can find a whole set of practices, you can author your own practices and you can select the practices you like and compose them.  There is still a lot to be done before we are finished, but with your help the dream will come true.



Let's be Smart

December 12th, 2007

One of the most popular movements in software development in recent years is the move toward agility. Today everyone wants to be agile. That is good! However, the essence of being agile is being smart. I have for several years expressed that the most important character you need to have to be a great software developer is to be smart. In several of my columns I have summarized what you need to be successful by saying: You need to be smart! What does that mean? Most people know intuitively what “being smart” means in everyday language, but what does it mean for software.      

It is not smart to model everything in UML for instance when building software. It is not smart to model nothing and go straight to code. It is however smart to find exactly that something that is of importance to model and code.  

Advice: What is that something? It is about the most essential use cases and in particular about the most essential scenarios through these use cases. It is about the components and in particular the parts of those components that realize these essential scenarios. Thus, it is about the essentials. Now you may ask what makes a scenario essential. An essential scenario is the response to the question: “what are the most important scenarios the system is supposed to do”. Which scenarios exercise the critical parts of the architecture? Which scenarios need to work in order for us to say that the highest technical risks have been eliminated? It is not smart to write requirements without caring that these requirements are testable. It is smart to make sure the requirements also are test cases. 

Advice: try use cases since they are also great test cases. It is not smart to work in a waterfall manner, first specify all requirements, then do the design and the code, and finally test it all. If you do, you will discover serious problems with performance, architecture, usability too late. It is smart to first build a small skinny system and then build a little bit more, and a little bit more, before you release the system to customers. Each time you build something, you must be able to run, validate and test it. 

Advice: use a controlled iterative approach such as the iteration practice in the Unified Process or sprints in scrum.It is not smart to run off and build a lot of “stuff” before first assessing if you can source the whole or parts of the application (Open Source or commercial offerings). It is not smart to develop software with a process that cannot scale if your system is successful and customers want much more. It is smart to use a way of working that is no more than what you really need, but that can grow as you succeed with the product. 

Advice: make sure your process has a dial for each interesting process parameter so you can turn the dial to the proper position for your project.  It is not smart to throw out your existing process and adopt a completely new process. That is doomed to fail in most cases. Not everything you did in the past was wrong so why should you start all over. It is smart to improve your process in small manageable steps.  

Advice: add one practice at the time. It is not smart to find a shortcut that in reality becomes a detour, such as skipping requirements and going straight to code.  It is smart to do enough requirements to find what to use to build your first increment, your skinny system. In general it is not smart to be extreme in what you do such as: model everything or model nothing, follow a strict waterfall process or an unstructured iterative approach, throw out what you have and start all over. It is smart to be balanced to do what is needed right now but with an eye to the future.  Above I have given a number of examples. In each case, there are some ideas on how to think about being smart. And each case can be expanded further. You will become smart with experience, but experience on the other hand is not a guarantee for being smart.  Of course eventually, it comes back to you. Smart is not the same thing as being intelligent. You can be intelligent without being smart. And you can be very smart without being very intelligent. Smart is not the same as having common sense. You can have common sense without being smart, but if you are smart you must have common sense.  

Smart is to do exactly right, not to find a broad solution that is just about right 

Executable SOA

November 5th, 2007

Today everyone talks about Service-Oriented Architecture. And as with all buzz words there are many interpretations. As the name suggests SOA was originally about architecture, but as time goes by its proponents put more and more meaning into it in the same way as proponents of EA put more and more into it. This is why some people have nicknamed SOA to Service-Oriented Ambiguity.      

What is SOA at its core? It is a kind of software architecture which allows you to connect course granular components called services through well-defined interfaces. These components can reside on any kind of platform (mainframe, client/server, etc.). They can be web applications, java or .NET applications, ERP systems or your own legacy systems.  

What is the business case for SOA? Since, services are essentially reusable components, you can efficiently support new business processes by using old services and interconnecting them. Since, they have well-defined interfaces they can be replaced by any other component complying to the same interface. 

Is this really new? Yes and no. No, because SOA has existed almost forever in the telecom space. The most successful commercial product ever built in Sweden was the Ericsson AXE switch. AXE’s success came to some extent from that its architecture was service-oriented. However, Ericsson had to build its own infrastructure - a unique computer architecture and operating system. My Software Reuse book (Software Reuse: Architecture, Process and Organization for Business Success, ACM Press) from 1997 is essentially a book on SOA. Yes, it is new because big vendors like IBM provide an infrastructure that allows services to reside basically anywhere in a network. 

How to succeed with SOA? It goes without saying that you need good people – leaders as well as champions. And they need to be smart – have common sense, practical, do what is needed not more, and of course be competent. Having said what is obvious let’s move to a more techie perspective:  

Design top down, bottom up and then find the balance as a trade-off between value and cost. Top down means you start from some business process ideas and deduct a solution as a set of interconnected components. If you just go top down it will usually be unreasonably expensive. Bottom up means that you start from what is available/reusable, such as legacy systems, web services, ERP systems, whatever, and build something close to what you need by interconnecting these components. This may not be exactly what your business modeling people were dreaming about, but something you can build quickly with low costs. Finally, you balance the top down and the bottom up and get what you may call a good enough solution. Your solution reuses a lot, but you may also have to build something new. This is smart! However, it is not enough. 

Do this very light (don’t major in modeling or post-its), develop quickly a road-map, build in small executable steps (even if the enterprise system is huge). And beware of the service companies that want to sell you big solutions – lots of billable hours without being accountable for delivering executable software!Don’t fail with SOA. It is so incredible expensive, and there is no reason you should fail. You just have to be smart ;-)  

EA Failed Big Way!

October 18th, 2007

Enterprise Architecture failed big way!

Around the world introducing an Enterprise Architecture EA has been an initiative for most financial institutions (banks, insurance companies, government, etc.) for the last five years or so, and it is not over. I have been working with such companies and helped some of them to avoid making the worst mistakes. Most EA initiatives failed. My guess is that more than 90% never really resulted in anything useful.


Why did people fail? There are many reasons, but they can all be summarized by the word smart. They were not smart when they selected solution. They were not smart when they selected way of working. They were not smart when they organized their business and IT resources. Building an EA is not rocket science. 

There are two common reasons specific to EA failure:

  1. Focus on paper-ware instead of executable software.  When enterprise architects work in an ivory tower without caring about what can be implemented, they produce too much models and documentation without executable solutions.  Enterprise architectures should be implemented incrementally, starting as early as possible. We call such architectures for executable EAs. 
  2. Big gaps between layers instead of seamless relationships.  Usually there are several layers such as a business layer, an application layer, a data layer and a technical layer. There are huge gaps between these layers which results in very brittle architectures. It is like trying to stand on a skateboard which is on top of another skateboard which in its turn is on top of yet another skateboard, etc. To have a chance these skateboards need to behave like one which is hard enough.  Thus the relationship between the business layer end the application and data layers are not straightforward and the relationships between the application layer and the data layer is as hard to manage as it was 20-30 years when we used methods like functional decomposition, or structured analysis and design. It is amazing that people haven’t learnt anything from component based development (with or without objects).

There are many other mistakes that people have made, many of which are related to organizational change in general. Examples include lack of business support for EA, not communicating the scope and purpose of EA, no strong IT leadership etc. In addition to these common challenges, all it takes to succeed at EA is to use best practices for modern software development, avoid upfront academic modeling, build both top down and bottom up, look upon the whole enterprise system as a system of interconnected systems.

Many of the companies that failed are now looking for the next silver bullet – Service Oriented Architecture SOA. To me SOA is what EA should have become. SOA can be described as EA++ -- it is Enterprise Architecture made better. SOA is clearly on the right path, but again adopting it requires that you work smart!  

Who Loves Process?

March 19th, 2007

Right now I am in my home in Switzerland to take a break and ski. I ski during the days and I work mornings and nights. Sleep takes too much time so I do as little as possible of that.:) One of the questions I often get nowadays is “Does Process come from the top or from the bottom?” Before answering I will start with two questions that I usually ask during my presentations. “How many of you love process?” In a group of 750 people, I usually see 3-5 raised hands. These are of course the people who work with process implementation, and they don’t have to follow a process themselves. Then I ask provocatively “How many of you hate process”. Now I get a very different answer. Usually 60-70% of the audience raise their hands!

Why is it like that? I believe that in many larger companies process has been pushed down on the people who know how to develop software. This won’t work anymore. The competent will speak up to get the process they want, which is good. What worries me though is that the pendulum will swing over to the other extreme. Instead of process push-down we will get process pop–up.

People do as they want. If every team gets their own process, they will not be able to learn from other teams, they will not get common set of tools and they will not harvest reusable assets. What we really need is a balance between push-down and pop-up.

For a very long time (since 1987) I have told my clients that one of the most important success factors for process adoption is to identify both a sponsor and a champion for the process improvement. The champion is a developer with many years of experience, great competence and with high credibility among his comrades. The sponsor is a higher level manager who wants to improve the way his organization develops software to support the company’s business. Without support from both don’t bother to adopt any new process.

Another of success factor is to use experienced people to coach the team adopting the new process. However in many cases management wants process but doesn’t want to invest in coaching. They just hope that people will adopt the process offered to them without proper coaching. As a consequence a lot of organizations have failed in process adoption. Management tried a shortcut, but it became a detour. The money an organization spends on product development is a factor of ten higher than the money it spends on proper coaching. For twenty years I have told my clients that without adequate training and coaching don’t bother to adopt any new process.

A third success factor is to balance the need for process from the top (the management) and from the bottom (the developers). This is true for any process whether it is a software development process or a business process. Only working from the bottom is a very slow process, and working only from the top creates a lot of resistance. Unfortunately, in many organizations management pushed down process on people counter to our recommendations. The developers were not enthusiastic but they tried to adopt the new process. That was not easy. Management believed their people were using the process but the developers did what they wanted to do, usually what they always had done. We got a gap, sometimes a huge gap, between what the process said and what people actually did. We call it the process-project gap. Thus we got failures and people started to hate process.

We need to change this. My recipe contains four ingredients:

  1. Reduce complexity. Instead of adopting a big process consisting of many practices, adopt a practice at a time. This will significantly reduce the complexity of process improvement.
  2. Balance the push-down from the top and the pop-up from the bottom, instead of just doing one or the other.
  3. Allow teams to change the practices so they work in their projects. The way you really work must be fed back as a change to the practices. We want a process that is alive and not just a dead book. This allows us to close the gap. One challenge here is of course that traditional process hard-heads will demand that all deviations from the described practices must be documented and perhaps even be given formal exemptions. If that is necessary then people will feel that it is easier to do what is demanded than to do what is right. What we need is a utilitarian perspective on process rather than dogma. :)
  4. Activate the practices and bring them alive in a way traditional process has never been able to do. While activated, a practice can help with all kinds of things. It can review what the developers are doing and suggest corrections. It can automate mundane tasks transparent to the developers. It can train the developers on the job reducing the need for classroom training and traditional human mentors. The selected practices should give context-sensitive help to the developers. This means that they should only give the developers help when they want it and only the help they really want and no more. The goal is to give the developers more time to think and create and to remove a substantial part of the no-brain work that developers suffer from today.

We have had a period of process push-down. The pendulum is now swinging to the other extreme – process pop-up. After a few oscillations the pendulum will hopefully swing to a new position where we will get processes that balance both the need for governance and for the need for creativity.  

Software is International

March 12th, 2007

Since my last postcard I have been traveling around the world talking about the Next Generation Process -- which actually is not about process but about practices.

I have been meeting customers and given seminars in San Francisco, Seoul, Beijing, Taipei, Tokyo, Boston, New York, Jacksonville, London and now I am in Stockholm for a few days. When I see that people need a break from technical stuff in my sessions, I usually give them a break.

Sometimes I ask the audience if they know where object-orientation and components were born. Which was the very first object-oriented programming language ever designed? Usually, I get some mumbling as an answer so then I go on saying: OK, I will teach you some geography and history :-).

Up there in the north (I point in some direction which is not very precise), there is a large, beautiful country that looks like a rectangle called Sweden. To the west of Sweden is a tiny, totally uninteresting country called Norway :-). It is unbelievable but the first OO language was born in Norway (created by Ole-Johan Dahl and Kristen Nygaard). It was in 1967 and the language was Simula. Simula had everything important about OO that you find today in for example Java.

At the very same time, in Sweden, components were born. Everything important about components and component-based development was used in the Ericsson AXE system. The software component idea was my first baby. Thus it is not true that Microsoft and Bill Gates invented software components.

The problem was that we in Sweden didn’t know what our neighbors were doing. Even if the geographical distance between Sweden and Norway is small, the intellectual distance is huge :-). In the minds of Swedes, nothing of intellectual value can come out of Norway. Swedes are even surprised that Norwegians have the capacity to have the reverse opinion :-).

South of Norway is another tiny, totally uninteresting country called … and I hope for an answer…yes, Denmark. Do you know what came out of Denmark? Do you know who the father of C++ was? Yes, Bjarne Stroustrop. Bjarne had studied Simula and when he joined ATT in the US he made C look like Simula and called it C++ :-).

Then, we have another country to the east of Sweden called … yes, Finland. Do you know what came out of Finland? Nokia? More? Linux, yes. Do you know who the father of Linux was. No? Linus Torvalds. In Finland about 5% of the habitants have Swedish as their mother-tongue. Linus is Finnish-Swedish :-).

Anyway, you may wonder why I tell you all this? Well, I want to point out that everything important about software was not born in the US :-). However, if you want to make money from it, you have to make the leap over the pond to the US! :-) 

Software Engineering meets Social Engineering

December 23rd, 2006

There is a conflict between people who support Software Engineering approaches (such as Rational Unified Process RUP) and people who support Agile methods (such as SCRUM or XP). This is understandable since the two approaches are described in an incompatible way. However that is unnecessary, since the ideas behind the two are in all essentials complementary. The trick is how to combine them in a way fair to both of them.

In the software world we like swinging. We are swinging from one way of working to another way of working -- from one extreme to the other. Some people have been swinging for their whole life and have had enough of it.

Once, about five years ago, when I spoke in Singapore for a small group of 50 people, a senior guy stepped up, very frustrated, and told me that UML and RUP would be dead in five years and I would be gone as well. I like provocations so I asked him calmly what he based that upon. He told us that he had worked with software his whole life. “In the 60s we all worked with assemblers, then we got Cobol and Fortran, then we got database design” and he continued describing a zigzag path with new methodologies in every step: “structured programming, structured analysis and design, object-orientation, components and so on.” He felt it was awkward hearing about a new standard modeling language. I told him that there was another path: Assembler, components with assembler, component with a programming language, components with structured programming, components with objects and now components with objects and aspects. And we have all the time used a modeling language that can be described as UML.

Well, I am still here and I intend to be here for many more years…knock on wood. The guy who was so unhappy became even unhappier next day when he was fired by his boss. However, it is true that UML and RUP have lost some momentum, but they are far from dead. They are just taking a break while the world is swinging.

We are swinging from the extreme of doing everything with UML following what many believe is a prescriptive Software Engineering process. And we are swinging to another extreme called Agile. Very young people are having their first swing and now they are Agile. Middle-aged people, what a cute label, are right now swinging from the extreme of working with Software Engineering to another extreme called Agile. We swing and our companies swing with us.

Today everyone is agile. Of course. Everything else would be silly. Let me say it loud and clearly: I am a big fan of agile. My teams at Ericsson were extremely agile.

However, most people I talk to have a fuzzy understanding of what agile really is about. I participated in a panel discussion with agile evangelists in the UK last March. The organizers had hoped I would be against agile. They were surprised. The audience asked us to define agile. The panelists claiming they were agilists started to ramble. They repeatedly said that the most important property of agile was that it was iterative. That is dead wrong. Iterative development is one of the key practices in RUP and it has been around long before RUP. It was previously called spiral development and created by Barry Boehm in the late 70s. So I had to help explain what agile is:

Agile is about three things:

  1. Most important, agile is about social engineering. This is what actually made agile different. It is about how to work as a team, how to make people motivated, how to collaborate, etc.
  2. Agile is about lightweight. Instead of relying on explicit knowledge like in RUP, agile relies on tacit knowledge. In RUP we tried to write down what we know are good practices. However, since people don’t read process books, it doesn’t make sense to have them. Instead agile assumes that people have knowledge in their heads that is enough to develop good software. This can of course be debated but that is what it is.
  3. Agile contributes some technical practices. This is the weakest part of agile. Very little is really new. Iterative and incremental development is as I already have said an old idea. User stories are a special kind of simplified use cases. Most interesting new idea is test-driven development. I am not claiming the agile technical practices are uninteresting, just saying that if it were just for these ideas we wouldn’t have been excited about agile.

As you see Software Engineering and Agile tackle different aspects of software development. Software Engineering’s strengths are about technical practices and Agile’s strength is about social engineering. Thus here the two are very complementary.

That Software Engineering is straightjacket and Agile is light is harder to tackle. The question is can we get the best from both worlds here. Yes, we can!

Finally, the Software Engineering camp has a set of technical practices, the different agile methods have other sets of partly overlapping practices. Can we find a way to live with both? Yes, we can!

To do that we have had to invent a new concept Practices. We don’t talk so much about process anymore; instead practices are first class citizens. Process is just a composition of practices. Instead of talking about second generation processes that are passive and monolithic (big process), we talk about a third generation process which is active and practice-centric. If you already now want to have a peek at a third generation process, please go to my web site