Beyond Agile by Ivar Jacobson

September 25, 2004

Coming back after many years to a place where you previously have lived is a special feeling. I spent a year at MIT 1983-84 as a visiting scientist. I was awarded a grant from the Marcus Wallenberg Foundation in Sweden to study abroad for one year, and I spent this year with my whole family in Boston. This was one of my most important years from a career point of view. I learnt to know a lot of people, some of them were professors at MIT. They were important references for me when I started my company Objectory in 1987.

Thus, this week when I attended the Software Development Best Practices conference in Boston, I also decided to visit MIT and see if I would find any of my old colleagues. I went to the new computer science labs which are located in a new building with a very impressive and advanced architecture. I will not attempt to describe it. I looked for names from my year at MIT. And I went to their offices to say hello, but most of them were out. I found Hal Abelson, who was one of the two teachers of the famous class: Structure and Interpretation of Computer Programs. That was and still is the best class I ever have taken. It was so great that I actually brought it back to Sweden as VCR tapes and it was taught at two universities in Sweden for some years. We were chatting for a while, he recognized me very well. “You were working with usability, right”, probably thinking about use cases. Anyway, I didn’t dig deeper into that.

At the SD Best Practices conference, I gave a keynote on Active Software. The last couple of years I have promoted this new idea as one of the biggest mega-trends on the horizon. A book is underway, and I have also devoted some previous postcards to this idea.

However, I am not going to talk about my keynote in this postcard. Instead I will devote this postcard to agile methods. These have become so popular for interesting reasons which I can’t resist reflect on.

Many talks had the term agile in the heading. Many old talks have been modified to motivate the term. Every author and presenter needs to make clear they are agile. So they join the club. I have joined the club too. I wrote a paper at the end of 2001 with the title: A resounding Yes to agile processes – but also to more. Of course we should be agile, there is absolutely no conflict. For me this has always been the goal of everything I have been fighting for. However, we have different ways of getting there.

Before getting there, let us look at the values of the Agile Alliance:

  1. Individuals and interactions over process
  2. Working software over comprehensive documentation
  3. Customer Collaboration over contract negotiation
  4. Responding to change over following a plan

Who can disagree with that?

  1. Of course, people are more important than any book, even if it is a big book on the web. Books don’t produce software. However, knowledge is important to make people successful.
  2. Of course, with working code your customer is happy, even if the design and architecture is expressed in UML. However, working code needs to be understood and maintained after the initial team developed it.
  3. Of course, software requirements can’t be specified upfront. They grow as people work on growing and experiencing the software iteration by iteration.
  4. Of course, detailed planning at the outset of the project is cheating the customer. Software development is a change process and the project should be smart about accommodating changes.

Of course, we all agree with that. So what is then the difference? There is a huge difference, but before answering that question I will give my opinion on why agile methods have become so popular? There are several reasons:

  1. UP (in particular RUP which is the IBM Rational version of Unified Process) has become too large and complex. This is true, but it also leads to a misunderstanding. It is true that UP is large and comprehensive, but the intention was never that everyone should learn everything about UP. Instead, a small subset of UP relevant for a project is selected, and this is what the project members will be trained in. Thanks to a new class of tools based on intelligent agents, the problem of UP being overwhelming will soon be history. WayPointer is such a tool.
  2. No really good tools from major vendors. We tend to forget that the modern software tool industry is less than ten years old. We have not really yet seen a tool environment with seamlessly integrated tools. My old Objectory tool from 1994 had a lot of features and great integration between different models not yet seen in any other tools. The problem was that the tool was developed in Smalltalk and that never became a commercial platform. However, we will very soon see a new generation of tools coming to our market place.

This was of course a very natural basis to grow discontent and there was room for something new. That new had to be something else than technology, because UP had that pretty well covered. The truth is that Agile methods have not contributed anything exciting and original from a technology point of view. Test first design is maybe the most interesting new idea, but it is similar to use case driven tests. From a technology perspective most ideas in agile methods are 20-30 years old.

So again, the new had to be about something else than technology for it to be missing in RUP (or rather perceived as missing as much of the debate is a matter of perception).

  1. Agile methods have added something new.
    They have added best practices on the so called soft factors in software development, such as pair programming, on-site customer, 40 hours work week and so on. We programmers have been drowning in technology and our work environment has had low priority. We like this new interest for the soft factors.

However, this in it self wouldn’t have made agile methods popular. I believe that the most important reasons for the popularity of agile methods are two:

  1. Many experienced developers felt side-stepped by the success of UP. They felt that their hard earned experience became less valued by management. This is of course a management mistake as a successful process adoption requires both knowledge (UP) and experience (people). Successful adoption typically happens when the most experienced architects in an organization are part of the change effort. Leaving them on the side-line is begging for problems.
  2. Another reason that these methods have become so popular is that they don’t require any specific or defined knowledge. Agile methods stand for "tacit knowledge". For experienced people this is of course exactly what they want, no one can tell them what to do. They can accept some general principles but no detailed suggestions are popular. For the inexperienced people, this is of course wonderful. They can also do it with whatever knowledge they have. One uses what one knows, quite simple.

In reality agile methods require very competent programmers to become successful. This is clearly expressed by its originators. In fact they require more knowledge than what a project using the UP requires. The problem is of course that there are not enough good programmers around the world. And those who are that good are very attractive, so they move around quite extensively. This is very different with UP. Usually once people have used UP for a project, the project is less dependent on any individuals (still individuals are important). The UP provides a lot of explicit knowledge, and it doesn’t rely on tacit knowledge only. We can educate in UP to a deep degree, but you can’t educate in tacit knowledge.

In a way I think these methods that now have introduced a concept for tacit, unknown, unstructured or undefined knowledge, takes us back 25-30 years. Let me explain what I mean more explicitly.

  1. In the beginning there was chaos.
  2. UP was meant to solve/control 1. Unfortunately it has had some negative side-effects as we discussed earlier.
  3. Agile methods are removing the side effects by moving back towards 1.
  4. We think that we instead should remove the side-effects by moving forward to third generation processes/active processes., i.e., instead of going back to generation 1.5.

Thus so far the software industry can be described as based on tacit knowledge. As a result we have got a poor reputation for quality of software. This distinguishes the software industry from other modern engineering disciplines. And those are not based on tacit knowledge! We too can’t continue base our future on tacit knowledge.

By now you know why I think agile methods have become popular. It has happened because we didn’t do a good job with UP and tools. This created discontent and was a good ground for making excellent programmers raise there voice and for young people to feel that they could do it without a lot of knowledge. Thus I view agile methods as populist methods. In my next postcard, I will describe how these approaches if not changed dramatically will again loose in popularity because a new class of tools will make them anti-agile in comparison with what will come. What will come will be agility in many more dimensions than today. I will talk about what will come next. Beyond Agile!