Being Agile by Ivar Jacobson

Rational’s Software Development Conference 2005 is over. It has been a very exciting week. I must admit I am a bit exhausted. The days were packed meeting customers or giving presentations. The nights became very late having fun meeting friends – both old and new.

Jaczone and Ivar Jacobson Consulting shared a booth and it is no exaggeration to say that we got a lot of attention. “I was really taken with the crowd at the spotlight theatre presentation by Dr. Ivar Jacobson and the demo of WayPointer. Standing room only and it wasn't even lunch time!" said Buell Duncan (General Manager, IBM ISV & Developer Relations). At this theatre presentation Tata Consultancy Services announced their work with WayPointer to make their own process active. TCS reported a more than 20% productivity gain (!), increased quality and ease of learning using WayPointer.

Jaczone and IJ Consulting gave a total of 5 presentations, all very well received. My own presentation was titled “Beyond Agile: Smart” – a thought-provoking title. This is a talk that has been cooking for more than a year. I still was not sure about how it would be received.

- o -

Today everyone wants to be agile. It is pathetic to see how basically every speaker has added the word agile to the title of their talks. I haven’t done it. I haven’t done it, for two fundamental reasons.

All my work over the years has, as I soon will explain, strived to make software development more agile.

  1. Agile is not enough, we need more. I want agile+++.
  2. What I actually want is to get smart software development. That is why the title of my talk was “Beyond Agile: Smart”.

Thus we all agree that we need agile software processes. We all agree with the four statements in the agile manifesto. We agree on many agile principles such as iterative development, continuous integration & testing, use only what you need, etc.

However, we have different ways to get there. The so called “agile methods” primarily rely on tacit knowledge. Tacit means implicit knowledge (achieved ad hoc and undocumented). The Unified Process relies primarily on explicit, structured knowledge. This is a big difference that has not come through in the debate. In previous postcards I have discussed this difference, see for instance my postcard from 2004, September 30, Minneapolis.

This postcard is to some extent a repetition of what I wrote then but I will also talk about some new stuff. I will clarify the difference between being just agile and being smart with four new manifesto-principles. I will extend the concepts of pair programming, as in XP, to many other kinds of pairs such as pair architect. However these pairs will be virtual pairs assisting you with knowledge and making you smarter. We will get there soon.

A process or a method being agile to many means that its definition (description, book, web site, etc.) is light, i.e., it is sketchy. Thus when working according to it in a project, you have to use tacit knowledge and therefore the project is agile. This may be true in many cases, but in larger organizations you usually have a lot of people with different tacit knowledge. This creates a lot of difficulties working together. People need guidelines in the form of explicit knowledge to work consistently and create good software. Thus, a very important insight is that: 

   A light process may make a project heavy.

We also know that too much explicit knowledge such as in the Unified Process (UP), can make projects heavy, if they have to select what to learn, learn it, apply it and update it with new explicit knowledge as they learn more.

However, if we in some “magical” or say smart way dramatically can reduce the work to select, learn, apply and update the knowledge in UP, the situation will be different. If we can deliver the knowledge you need, and only that knowledge, and exactly when you need it and not before, then the size of the process doesn’t matter. Whether the process “book” is 100 pages, 1,000 pages or for that matter 100,000 pages will be irrelevant for ease of use. Thus, the more the better! You will only get just what you need and when you need it. And the bigger book, the more real on-line mentoring you will get.

This is what we can do with intelligent agents. Every developer has an online agent, or as we say, a virtual mentor. The virtual mentor is able to select what you need know, teach you exactly that, help you apply what you learnt and learn itself from your experience. With WayPointer we have proven that this technology really works. Much more will happen in the years to come, but already today, as experienced by TCS, substantial increases in productivity, quality, user experience, etc. can be made.

XP talks about pair programming. With these virtual mentors we talk about:

  • Virtual Pair Programmers
  • Virtual Pair Analysts
  • Virtual Pair Designer
  • Virtual Pair Tester
  • Virtual Pair Project Managers
  • Etc.

A process supported by intelligent agents, such as the Unified Process with WayPointer, is much more than an agile process. It is a next generation process. It is beyond both agile methods and traditional processes such as UP. I call such a process a smart process.

A smart process is agile (as defined by the agile manifesto) but it is also more. I have formulated four manifesto-principles to define a smart process and I call them Manifesto for Smart Software Development:

  • Make well-known knowledge explicit over keeping knowledge tacit
  • Active process over passive process
  • Let models be code over letting code be models
  • Team capability over dependency on individuals

I am not sure that these four principles will be the most important, but they work for now. I can’t see that methods or processes that primarily rely on tacit knowledge ever can compete with smart processes. Together I hope we can make the software world smarter and not just agile.

If you want to keep up with how our work on smart methods evolves, sign up on www.ivarjacobson.com and we will keep you informed. 

A Long Trip by Ivar Jacobson

I often get the question if I have any jetlag travelling as I do. It is understandable that people ask about this. After the Rational Software Development Conference 2005 a month ago my schedule became hectic. I will give you a résumé of my travelling. Unfortunately, I can’t be specific when it comes to business meetings.

May 26, Thursday evening I left RSDC’05 in Las Vegas and went to San Francisco.

I spent the weekend in SF to work and meet with friends.

May 30, Monday Redmond, Seattle

May 31, Tuesday Minneapolis

June 1, Wednesday I went to Chicago to board a flight to Stockholm, Sweden.

June 2, Thursday I made visa arrangements for India and I met my family in the evening.

June 3, Friday I left for Mumbai, India. I arrived very early Saturday morning and spent the weekend sightseeing and swimming in the pool. I love Indian food so it was relaxing. June 6, Monday Agneta Jacobson (CEO of Jaczone) arrived late Sunday night. Together, we had customer meetings the whole week. Monday and Tuesday in Pune, Wednesday in Mumbai, Thursday in Bangalore and Friday in Chennai. Late Friday night I went to Singapore and then continued directly to Seoul. I arrived late Saturday afternoon. I spent the Sunday in Seoul trying to recover (by reading emails).

June 13, Monday IJC Korea participated in the KCIC conference, a kind of component consortium. I gave a keynote on Unifying Foundation and one of our Korean consultants Jung-Nam Kim gave a talk about our approach to components. We also had a booth which was very visible. The conference was quite successful for us, so now our team has a lot to follow up on. In the evening we had a great dinner at an Italian restaurant. We all felt a little upbeat.

June 14, Tuesday I went to Beijing and I had a great dinner with a few people from a customer of ours. They have tried to adopt RUP on their own so now they are ready for our help!

June 15, Wednesday we participated in a large software conference in Beijing. I gave a keynote on the topic “What you don’t get from CMMI: Good Software”. The talk was very well received and we got many leads to work with. Before my talk I had two interviews with Chinese weekly magazines.

After the talk I went to the airport to fly to Austin, Texas via Chicago. A very long trip.

June 16, Thursday I attended the UML & Design World conference and I gave a keynote on the subject “Beyond Agile: Smart”. Before I started my presentation I said that today I will tell you about the greatest mega-trend you ever have seen. Last time the world saw something similar was in the mid 50s when we got (passive) software. All my work on Component Based Development, use cases, UML, RUP and aspect-oriented software development is important but just micro-trends within the passive software mega-trend. Active software is a coming mega-trend. After my presentation I asked the audience if they agreed with me that what we are talking about is a mega-trend. I was surprised to see that as many as 50% raised their hands!!! I also asked how many didn’t agree and about 10% raised their hands. At the least my talk created an interesting discussion. A lot of people came up to me afterwards and expressed their gratitude.

After the keynote, I participated in a panel on UML, specifically why UML adoption rate is so low. Only 10% of the developer community has adopted UML. My explanation is that we don’t have really good tools and that UML is too complex for most people to learn…we need smart tools. The most interesting comment came from Steve Cook, now with Microsoft, earlier with IBM. He suggested that UML was very good but UML 2.0 was bad. He meant that the core ideas of UML are excellent. He went on and suggested that we restart the work and build a small subset of UML that we make available. The rest can be added in domain specific languages. Steve, I hope I got you right. Then one of the other panel members (Randy Miller from Microsoft) said that the Objectory tool was many years ahead of the tools when it was available in 1995. [In fact, I think, Objectory supported this small subset of UML features that we need, but ignored much of the nitty-gritty details.] Randy said in public that no tool has yet become as good as the Objectory tool in 1995. I have heard similar things many times. Anyway, forming a core of UML features could be a great thing. I believe the same is true for the Unified Process. Make it simple to understand the core. We can add more with technology such as WayPointer.

In the evening I went via Denver to San Francisco. I was working on emails and business stuff the whole weekend, but Monday I took a day off to go to Napa Valley with friends.

June 21, Tuesday I went to London to meet customers and to give presentations.

June 23, Thursday night I went to Sweden to celebrate a Swedish Midsummer.

Today Friday June 24, is Midsummer Eve. This is a day we celebrate with friends and family.

Next week on Tuesday I will go to Beijing and on it goes.

Do I have jetlag? No, because my body never really knows what it means not be jetlagged!  

CMMI by Ivar Jacobson

May 19, 2005

CMMI

This year has been very exciting so far. I have traveled around the world four times and done lots of additional local flying (such as flying around in the US, in Europe or in Asia). There is nothing as rewarding as making other persons successful, regardless of whether these are customers or individuals in my companies.

Today we launched Ivar Jacobson Software (IJS) in China. This took place in Beijing where we have our Chinese head office. In the morning we held a high profile press conference. More than twenty reporters, more than ten Chinese government officials, and many locally renowned scholars and researchers attended. I held a talk titled “From Vision to Impact: Transforming Software Development in China”. I outlined a vision for making Chinese software development more competitive internationally, analyzed the reality of change and demonstrated how we can help.

Then an exciting moment came. I was asked to remove a cover from some hidden object, and boom: a balloon exploded and thousands of colored paper chips filled the air. The mood in the room rose to a climax and out from below came the company plate, see picture below: 

During the press conference, representatives from Chinese government and our partner IBM Rational delivered their welcome notes to our opening in China. Last but not least, IJS and CVIC SE (a local Chinese software company) signed a memorandum of understanding on building a business partnership.

This shows the Chinese software community that we want to build mutually beneficial partnerships in China.

In the afternoon, we held a public seminar. More than 150 software individuals, from different parts of the software community attended. I talked about “What CMMI cannot give you: Good Software” – a thought-provoking title.

- o -

The whole CMMI thing is based on this so called “law of quality”: The quality of a product is largely determined by the quality of the process that is used to develop and maintain it.

I think Deming discovered this law around 1950. And he had great success in Japan by improving the quality of manufacturing processes. There are a few things that are vague in the above law when it comes to software development.

First, what is quality in software? The quality of software is a much broader concept than that of common goods, such as automobiles. The quality of software has gone way beyond defect rates. (It seems though that many still haven’t got it.)

Second, manufacturing and software development are of different nature. There have been lots of debate on this topic and no common understanding has been reached so far.

Third, how large is largely? In what dimension? What if it doesn’t help the dimension (getting good software) I care about at all? That is exactly the case with CMM & CMMI.

The Chinese government has in a very direct way supported its software industry to adopt the Capability Maturity Model CMM/CMMI. They have done so inspired by the success of the Indian software industry. The Chinese software industry hopes to attract foreign companies to outsource software development to China by being able to offer high quality software and skills to low cost. As an effect many Chinese companies have made substantial investments in moving their people to higher levels of CMM, some companies even to CMM level 5. However, this has not resulted in western companies significantly moving their business to China. Why not?

CMM/CMMI focuses on describing the way you develop software and it helps you measure it. However, it basically doesn’t care about what kind of software you get from your process. Why should any company - from the west or from the east - care at all about how you internally develop software for them? Well, maybe they should care a little bit, but not much. However, what they really should care about is if they get good software or not. Good software is many things. Some of the most important things are that the software must meet business requirements, that it gives great user experiences and that it can gracefully change for the life-span of the business (that is for ever). CMM/CMMI is silent about what to do to get good software in these respects. This explains my title of the talk: “What CMMI cannot give you: Good Software”.

In fact CMM/CMMI can have an opposite effect on a software development organization than the desired one. It can reinforce a process that results in bad software and thus make it harder to introduce a process that results in good software. Jokingly, I sometimes say that this is analogous to paving cow-paths. You get better cow-paths, but you will still have cow-paths instead of roads. Therefore the right way is to first identify a process that is proven to produce good software and then introduce CMM/CMMI. If you do it this way you will get good software and you will produce the good software in a way that can be improved and measured.

My final words were (please remember that a good process is a process that results in good software):

It is easy to make a good process measurable,
but it is hard to make a measurable process good. 
 

Being Smart by Ivar Jacobson

December 8, 2004

Being Smart
After three intensive days in China I make a summary. We have had numerous interviews both the old-fashioned way – meeting journalists – and through on-line chatting. It was truly amazing but the chat had more 2000 participants! I gave a keynote at a Rational partner seminar, and I was invited to meet several of Rational’s partners. We were very well taken care of, in particular we were taken to JiNan (a one hour flight south of Beijing) to give a seminar to a leading software house and outsourcing company. As a bonus, we got a nice guided tour in the old city now being preserved for the future.

Since my consulting company Ivar Jacobson Consulting now is setting up an office in Beijing we got a lot of attention from the Chinese software industry. We are already discussing interesting partnerships with several companies, the constraint is our time. Such partnerships will also involve Jaczone since we view WayPointer as absolutely essential to succeed with UP (or RUP) implementations.

This was a very nice trip, I am very grateful for the help we got from our old Rational friends, in particular to Steve Chen (general manager for the IBM Rational brand). We will soon be back and start growing our presence in China. China is an unbelievable opportunity for us. However to be successful we need to be smart.

- o -

Being smart is thinking for yourself and taking your knowledge and putting it into context and thereby delivering business value. It is not about slavishly following a set of rigid principles, or for that matter a rigid process. However, totally rejecting proven principles and well-defined processes and instead basing your work on nothing but whatever happens to cross your mind when you start to work is of course extremely non-smart.

Being smart is about making the right trade-offs to make sure that you get the results you are after in the most cost-efficient way. This is often not the same as what might be perceived as the quickest. For example if you are creating a suite of applications it is important to have a well designed architecture that can allow the suite of applications to grow and evolve. Building this architectural foundation will not pay off until perhaps the second or third application of the suite is built. Also, making the right trade-offs is often not the same as doing it exactly by the book. New problems and new technologies sometimes call for creative thinking. This takes an open-minded attitude and an unassuming culture. The kind of self-righteousness that sometimes is displayed by technically gifted engineers is a direct contradiction to this. Having predefined conceptions about how things should be done (like use light tools, or modelling is not helpful) is simply a very conservative mind-set that predefines the notion that tools are not helpful and will never be helpful either. Truth is that no tool can do all of the job, or even make all parts of the job easier. But then, most proven tools can offer value in some part of the process. Smart people recognize this and make a conscious decision on whether the tool should be used or not based on cost-benefit in the actual case, not based on some preconceived notion of whether such and such tools are useful or not. When I do work around the house and have a traditional screw-driver with me and have two screws to screw-out I will do with the screw-driver. If there are 100 screws to deal with, well then I go get a power screw-driver. Different problems require different solutions.

Being smart means also to understand your role in the organization you are part of, and making sure that you as best you can contribute to the aggregate value delivered by your organization. Sometimes this means that you must blow the whistle on “bad practices” or break a few “rules”. For the most part, however, it means to cooperate with your team members in an agreed upon way. In a commercial software development environment this typically means that it is smart to accept and use common terminology, commercial tools, and a defined process, even if these not necessarily are 100% aligned with your personal preferences.

As you have understood by know I believe that to be smart you must know many things and you must know how these things correlate to each other and how they work in real-life situations. So to be smart you need knowledge and experience.

Thus the question is then, can one get access to this knowledge and experience some other way than learning the hard way? I think so. A prerequisite to this is of course that you first find ways to describe and formalize such knowledge and experience. UP is just that, an attempt to capture and systemize knowledge and experience. There is however a problem with the form here. UP and RUP (for that matter) are just static texts. This sometimes makes people perceive them as “all or nothing propositions”. Even if there are tools to customize the static presentation of RUP very few have the luxury to do upfront process customization before commencing on the real project. Also, this is a very non-agile approach to process customization as the description still is static and can’t adapt. The way is of course to capture experience in the form of an active process. I.e., Active process makes it possible to access and assimilate more information allowing you to make smarter decisions. This is due to that an active process is sensitive to your defined goals, your personal preferences, and your responsibilities, and of course the context of your organization and the system under development. And, as you all know, this is just one of the advantages of an active process.

And finally, SMART people think.  

Active Process by Ivar Jacobson

October 30, 2004

Shenzhen is in China about a one hour drive from Hong Kong. It takes more time to get there though since you have to go through immigrations and customs. Yesterday I gave a one day seminar for a telecom company. Before my seminar, one of my colleagues in Ivar Jacobson Consulting had given a week of training in software architecture. Our approach to software architecture is a major improvement to the one in RUP: it is simpler, more systematic, broader and better prepared for the future (such as aspect-orientation). Thus the training was very well received.

In Shenzhen people can have fun. This time we went to a night club area. In Minneapolis we had a hard time to find any place to go to. This time there were almost too many night clubs, everywhere there were night clubs. We were four persons including my female host, and we went to the largest night club I have ever seen. It had everything and it was very civilized: big bar, show after show with performances by famous persons, disco. We had a good time watching the shows and talking. We talked about the future of software development and about life in China and in the Western world. I also discussed what will follow further down in this postcard. It was a very nice night in company of very nice friends.

- o -

I have some interesting quotes for you:

"Imagine a software development project where everything works well together -- the tools, the teams, the platforms, and the processes. You no longer have to imagine." This quote is from the marketing of the new IBM software development platform.

Also consider this one: "With Visual Studio [2005] Team System, process is not just documentation. It also manifests itself as actual tool behavior changes. When you chose the process at project inception, you are also choosing the workflow and work products, which then drive how the system behaves."

These quotes are interesting as they show that the major vendors now are beginning to see the necessity of integrating tools and process. I also like them because I discussed these concepts 10 years ago.

- o -

Back in 1995, I wrote a series of papers about the software engineering process (SEP) and its support environment (SEPSE)*, the vision then was similar in many ways to the vision that Microsoft and IBM is talking about now. To use a bit more modern terminology, let's call it "development platform" instead of SEPSE from now on.

Such a development platform should cover of all the tools required to support a software development process. The tools should be integrated in the sense that they work together to achieve a common overall goal - to build software according to a process - and communicate with each other as necessary. And, there are many tools that must be included, e.g., tools for capturing requirements, modeling, coding, debugging, testing, change/bug tracking, version control, and what have you.

Anyway, a key feature of such a development platform that I discussed is the capability to support a process, including workflow support and modeling support. Quoting: "The SEP enactment tool represents a unique feature as it coordinates the right set of collaborating modeling tools and artifacts for any given task, i.e., it drives the integrated environment for the user."

Now, ten years later, and with ten more years of experience my thinking has matured. Today I am not talking about a "SEP enactment tool" today I talk about Active Process.

There is a huge difference here and I believe that eventually Microsoft and IBM will realize this too and that they need to move beyond what they currently are promoting as process features in their (upcoming) software development platforms.

The difference is that a process enactment tool only provides guidance on the macro level in that it will control which activities that can be carried out by a certain person (given her assigned roles), when the activities can be carried out, e.g., checking that required input models and documents are available and so on. This kind of control is typically based on that users enter meta-data into the system that describes the state of models. An example of such meta-data is the level of stability; a model could be classified as preliminary, ready for review, or approved. This will however, not help you with the real work such as specifying clear requirements, creating good models, writing good code and so.

An Active Process covers all of this but goes beyond it in three major ways. An active process should:

  • Provide advice that is specific to the system under construction as opposed to just feeding generic descriptions and help texts. This means that the active process must have access to and be able to reason about the content of deliverables and intermediate results of the software development process, not only meta-data entered by users.
  • Provide advice just-in-time when needed and in an unobtrusive fashion. The developer should never be stopped or hindered by the active process. The choice should be with the individual at all times.
  • Be self-configuring to become as agile as possible, but no more agile than that. I.e., it should be self-configuring based on the characteristics of the organization (distributed, outsourcing, etc.) and characteristics of the system to be developed (size, criticality, etc.).

Even if IBM and Microsoft are only starting their journey as I started it 10 years ago it is still noteworthy that these companies now are starting to see process and tool support for process as an integral part of their environments. Going forward I think we will see more of this and other vendors will need to follow suit as their offerings will be considered incomplete else wise.

As you may know, Jaczone has been developed and evolved the Active Process concept over the last five years. What we have learned is that this is quite a challenge to get right; however we have come pretty far down the road. And this challenge is what makes it fun! 

Complexity and Unified Process by Ivar Jacobson

September 30, 2004 

Minneapolis was the very first city on the US main land that I visited. In 1981 a delegation from Ericsson went here to evaluate some technology from Sperry-Univac. I don’t remember more than that I got very bad flu on the flight over, and that I stayed in bed for most of my visit. Well, not entirely, because one evening our host took us out to a dancing-place. For a couple of hours my flu was gone, and I had fun.

This time I came to Minneapolis to meet a customer that is deploying UP broadly. I was also invited to speak at the Rational User Group in the Twin Cities. This was a nice event with quite a large number of people. After my presentation, some people took me out for dinner. During the dinner I told them about my dancing experience 23 years ago. I asked: “How is it today, do you still have good dancing places?” “Sure, let’s show you.”

Thus we left the restaurant, and spent the next hour walking from place to place to check out the dancing scene. Place after place was either closed or there was a big guy in front of the entrance, saying: “You are not going to like it!” We were not young enough.

However, eventually we found a place. The dancing hadn’t started yet, so we watched a show with pretty girls. A tall, big woman with a mike came up to us and interviewed us. At the end she asked “Are you all straight? Or, do you want to try something new?” We were at a gay bar. The pretty girls were boys. The only place open for dancing was a gay bar. Well, we didn’t feel uncomfortable. People had fun, and everyone respected each other so we enjoyed the music and the drinks. However, it is no overstatement that we didn’t dance much.

- o -

In my latest postcard from Boston, I explained why I think agile methods have become popular. It has happened because we didn’t do a good job with UP (or other defined processes) and tools. This created discontent and we got agile methods -- or populistic methods. Agile methods don’t provide any defined knowledge. Instead they rely on "tacit knowledge". You can do it with whatever knowledge you have. This was of course also very popular.

However, the basis for this discontent will soon be gone. It is just a question of time, at the most two years, before we will have substantially better tools from leading tool vendors, and they will be better integrated. On top of them we will get intelligent tools (WayPointer is such a tool) that remove a lot of the no-brain work that programmers spend their time on today. These tools will make it impossible for ‘tacit knowledge’-based methods (read agile methods) to compete in terms of agility. They will be even less competitive when it comes to quality in measures such as productivity, quality, time-to-market. Quality includes good architecture and design.

Moreover, the complexity of the UP will be eliminated through these intelligent tools. Let me take you through this in smaller steps:

  • Why is UP complex today?
    One reason is that UP is a rich knowledgebase. This knowledgebase defines a set of best practices, and their interrelation. The goal is to have all the best practices that a team may need available easily.
  • What is the problem with this knowledgebase?
    The problem is that it can be overwhelming. This was never a surprise to me. I have known for many years that this is how people perceive the UP. I have been crystal clear for all these years that the only way to really implement UP is through substantial training and mentoring. My assumption was that only a small fraction of the software developers would be able to adopt UP properly. I have been clear when talking to people interested in UP that if they are NOT willing to do these investments, they shouldn’t even bother to think about UP.

Another problem is that even if UP is rich it is not rich enough. This may sound as a paradox, but it is not. I believe that one of the major problems with UP is that it is generic. Since UP is a book, it can’t be specific to a particular application or domain. Special cases or exceptions have to be treated very superficially. Otherwise no one would read it. This means that the process is explained generically and that examples and such probably come from a problem space of no interest to a particular reader. Examples and advice must be particular and context sensitive to be useful. However, this problem can also be solved.

  • Why did we then develop this rich knowledgebase?
    The goal has all the time been to achieve agility + more. My belief is that the more people know about software best practices, the more agile they will be able to work. However, I also wanted more as I will tell in a minute.
  • How can we help people adopt and apply this rich knowledgebase easily?
    Through a completely new class of tools, based on intelligent agents, the knowledgebase in UP can be served in very small chunks. The minimum knowledge needed in a particular detailed context will be served. Not only so, it is presented in context of the actual system that the user works with, instead of generically. The rules in UP may trigger and defects may be found, reuse may be suggested, next steps proposed, solutions carried out. The tools will serve as a motor for each individual and for the team as a whole. WayPointer is for the time being the only such tool. With such tools, the programmer can use his or her time to what is creative instead of doing "no-brain work". Today 80% of the programmer’s time is spent on doing no-brain work. This number is based on projects using agile-like (or code centric) methods. With WayPointer this time is reduced dramatically.

We all want agility. The goal is the same for all of us. However we have different ways of getting there. The biggest difference (and maybe the only really important reason) is that agile methods assume tacit knowledge, whereas UP assumes explicit (defined) knowledge served through intelligent tools to make the software process active.

Why is explicit knowledge so important?
Moreover, I strongly believe that good software is developed by people being smart, and that you can only be smart by having critical knowledge – and of course having this knowledge turned into experience and skills. In our world the critical knowledge is on best practices in software development. This requires explicit knowledge available to the whole team; tacit, unstructured knowledge which only is in the heads of individuals is not sufficient. This is why UP serves a very important role; it includes much of the critical knowledge you need and intelligent tools can serve this knowledge when you need it.

I believe that it is not enough to be agile. The term agile doesn’t imply anything about whether the software is good or not. I have always worked to get good software (good in terms of all measures including good architecture, good test cases, good interfaces, good components, etc.). Furthermore, it has all the time been my belief that if you know how to make good software, you can always figure out how to make it fast or agile. This is what we have been doing with UP + intelligent tools. Thus instead of talking about being agile or about agile methods, I would rather talk about being smart and smart methods. Smart means agile, but also much more. It means all the things we need to get good software. UP with intelligent tools helps you to become smart.

Next year, the world will be looking for a new buzzword. Maybe I should launch the term Smart Methods. Maybe I should give a keynote with the subject: "Beyond Agile: Smart". Trust that Smart Methods are building on top of defined knowledge with smart agents.

What do you think?  

Beyond Agile by Ivar Jacobson

September 25, 2004

Coming back after many years to a place where you previously have lived is a special feeling. I spent a year at MIT 1983-84 as a visiting scientist. I was awarded a grant from the Marcus Wallenberg Foundation in Sweden to study abroad for one year, and I spent this year with my whole family in Boston. This was one of my most important years from a career point of view. I learnt to know a lot of people, some of them were professors at MIT. They were important references for me when I started my company Objectory in 1987.

Thus, this week when I attended the Software Development Best Practices conference in Boston, I also decided to visit MIT and see if I would find any of my old colleagues. I went to the new computer science labs which are located in a new building with a very impressive and advanced architecture. I will not attempt to describe it. I looked for names from my year at MIT. And I went to their offices to say hello, but most of them were out. I found Hal Abelson, who was one of the two teachers of the famous class: Structure and Interpretation of Computer Programs. That was and still is the best class I ever have taken. It was so great that I actually brought it back to Sweden as VCR tapes and it was taught at two universities in Sweden for some years. We were chatting for a while, he recognized me very well. “You were working with usability, right”, probably thinking about use cases. Anyway, I didn’t dig deeper into that.

At the SD Best Practices conference, I gave a keynote on Active Software. The last couple of years I have promoted this new idea as one of the biggest mega-trends on the horizon. A book is underway, and I have also devoted some previous postcards to this idea.

However, I am not going to talk about my keynote in this postcard. Instead I will devote this postcard to agile methods. These have become so popular for interesting reasons which I can’t resist reflect on.

Many talks had the term agile in the heading. Many old talks have been modified to motivate the term. Every author and presenter needs to make clear they are agile. So they join the club. I have joined the club too. I wrote a paper at the end of 2001 with the title: A resounding Yes to agile processes – but also to more. Of course we should be agile, there is absolutely no conflict. For me this has always been the goal of everything I have been fighting for. However, we have different ways of getting there.

Before getting there, let us look at the values of the Agile Alliance:

  1. Individuals and interactions over process
  2. Working software over comprehensive documentation
  3. Customer Collaboration over contract negotiation
  4. Responding to change over following a plan

Who can disagree with that?

  1. Of course, people are more important than any book, even if it is a big book on the web. Books don’t produce software. However, knowledge is important to make people successful.
  2. Of course, with working code your customer is happy, even if the design and architecture is expressed in UML. However, working code needs to be understood and maintained after the initial team developed it.
  3. Of course, software requirements can’t be specified upfront. They grow as people work on growing and experiencing the software iteration by iteration.
  4. Of course, detailed planning at the outset of the project is cheating the customer. Software development is a change process and the project should be smart about accommodating changes.

Of course, we all agree with that. So what is then the difference? There is a huge difference, but before answering that question I will give my opinion on why agile methods have become so popular? There are several reasons:

  1. UP (in particular RUP which is the IBM Rational version of Unified Process) has become too large and complex. This is true, but it also leads to a misunderstanding. It is true that UP is large and comprehensive, but the intention was never that everyone should learn everything about UP. Instead, a small subset of UP relevant for a project is selected, and this is what the project members will be trained in. Thanks to a new class of tools based on intelligent agents, the problem of UP being overwhelming will soon be history. WayPointer is such a tool.
  2. No really good tools from major vendors. We tend to forget that the modern software tool industry is less than ten years old. We have not really yet seen a tool environment with seamlessly integrated tools. My old Objectory tool from 1994 had a lot of features and great integration between different models not yet seen in any other tools. The problem was that the tool was developed in Smalltalk and that never became a commercial platform. However, we will very soon see a new generation of tools coming to our market place.

This was of course a very natural basis to grow discontent and there was room for something new. That new had to be something else than technology, because UP had that pretty well covered. The truth is that Agile methods have not contributed anything exciting and original from a technology point of view. Test first design is maybe the most interesting new idea, but it is similar to use case driven tests. From a technology perspective most ideas in agile methods are 20-30 years old.

So again, the new had to be about something else than technology for it to be missing in RUP (or rather perceived as missing as much of the debate is a matter of perception).

  1. Agile methods have added something new.
    They have added best practices on the so called soft factors in software development, such as pair programming, on-site customer, 40 hours work week and so on. We programmers have been drowning in technology and our work environment has had low priority. We like this new interest for the soft factors.

However, this in it self wouldn’t have made agile methods popular. I believe that the most important reasons for the popularity of agile methods are two:

  1. Many experienced developers felt side-stepped by the success of UP. They felt that their hard earned experience became less valued by management. This is of course a management mistake as a successful process adoption requires both knowledge (UP) and experience (people). Successful adoption typically happens when the most experienced architects in an organization are part of the change effort. Leaving them on the side-line is begging for problems.
  2. Another reason that these methods have become so popular is that they don’t require any specific or defined knowledge. Agile methods stand for "tacit knowledge". For experienced people this is of course exactly what they want, no one can tell them what to do. They can accept some general principles but no detailed suggestions are popular. For the inexperienced people, this is of course wonderful. They can also do it with whatever knowledge they have. One uses what one knows, quite simple.

In reality agile methods require very competent programmers to become successful. This is clearly expressed by its originators. In fact they require more knowledge than what a project using the UP requires. The problem is of course that there are not enough good programmers around the world. And those who are that good are very attractive, so they move around quite extensively. This is very different with UP. Usually once people have used UP for a project, the project is less dependent on any individuals (still individuals are important). The UP provides a lot of explicit knowledge, and it doesn’t rely on tacit knowledge only. We can educate in UP to a deep degree, but you can’t educate in tacit knowledge.

In a way I think these methods that now have introduced a concept for tacit, unknown, unstructured or undefined knowledge, takes us back 25-30 years. Let me explain what I mean more explicitly.

  1. In the beginning there was chaos.
  2. UP was meant to solve/control 1. Unfortunately it has had some negative side-effects as we discussed earlier.
  3. Agile methods are removing the side effects by moving back towards 1.
  4. We think that we instead should remove the side-effects by moving forward to third generation processes/active processes., i.e., instead of going back to generation 1.5.

Thus so far the software industry can be described as based on tacit knowledge. As a result we have got a poor reputation for quality of software. This distinguishes the software industry from other modern engineering disciplines. And those are not based on tacit knowledge! We too can’t continue base our future on tacit knowledge.

By now you know why I think agile methods have become popular. It has happened because we didn’t do a good job with UP and tools. This created discontent and was a good ground for making excellent programmers raise there voice and for young people to feel that they could do it without a lot of knowledge. Thus I view agile methods as populist methods. In my next postcard, I will describe how these approaches if not changed dramatically will again loose in popularity because a new class of tools will make them anti-agile in comparison with what will come. What will come will be agility in many more dimensions than today. I will talk about what will come next. Beyond Agile!  

RUP and Agility by Ivar Jacobson

July 23, 2004

Another Rational User Conference has just been concluded. Since 1997 I have attended every one of them, now eight conferences. There were many good talks, but I had no time to attend more than the opening keynote by Mike Devlin. Instead my days were filled by meeting people. Many had heard about my recent adventures in Singapore and Korea and wanted to understand where I am heading. Basically, I see a huge need to help people around the world to apply best practices of software development. Even if we have developed the RUP, we have not been successful in getting people to adopt it and apply it. One of the consequences of this “failure” is that agile methods like XP have become popular. I want to change that, and since RUP not yet has penetrated East Asia, this is a good place to start.

My keynote this year was titled “What you didn’t know about RUP”. So this talk was about the secrets of RUP. Actually a lot of people, including RUP evangelists, don’t know about these secrets. One of the secrets is that RUP or the Unified Process in general, is agile and is in fact becoming increasingly agile as time goes by. I feel that we have not succeeded in explaining why it is so. I took it as my mission to do so.

First I developed some vocabulary to explain why RUP already is agile. When people talk about a process being light or heavy, they actually mean that the process definition (the “book”) is light or heavy. They don’t talk about the process in the meaning of what actually is done and created when you run a project. So we actually have two concepts to discuss, the process definition (PD) and the executing process (P). What we really care about is that the P is agile and not really that the PD is agile.

Thus we can have a light PD (a book) but since it doesn’t explain much it is not of much support to the team that runs a project. They will have to invent the particulars of the process as they go. This may slow down the project substantially, something I have experienced in many projects. Thus a light PD frequently result in a heavy P.

Also, although a rich PD may require that people spend time upfront to learn what to do; it may result in that the P works very smoothly. Everyone in the team knows what to do, when to do it and how to do it. Thus a rich PD frequently results in an agile P. I have seen this happen many times in serious commercial software development.

So far most people agree with me. So let us now talk about the executing P and how to make it agile. Today, to get a PD like RUP to be used in a project, it takes three things:

  1. A subset of the PD has to be chosen and maybe some additional process steps will be added. There are good tools to do so, but still this slows down the project. To be able to select what to use you need to know all that can be used. Most companies don’t have the luxury of time and competence to do that. Makes the P heavy!
  2. Once a proper PD has been created, it has to be adopted. This means that people have to learn it, they need to get it into their heads. This requires training which takes time. Makes the P heavy.
  3. Finally, while running a project, the PD must be applied consistently and effectively. This means that people need to know what to do, when and how. Usually, people can’t do this by themselves, it is a far too big step to go directly from the classroom to your office and do useful things in a productive manner. Thus projects need mentors. Makes the P heavy.

Why would people take on all this work (to define and learn a PD) to develop software? There is only one good reason. We know that people will develop better software, better software from all perspectives.

What have we done to make RUP implementations overcome these difficulties?

  1. Now and even more in the years to come, I believe we will see a number of predefined variants of RUP become available. We will have ready-made light PDs, and we will have heavy PDs for product-line developments. This makes RUP more agile. You start using the PD that works for you.
  2. With RUP comes a lot of training material. RUP comes with a set of very systematic models, enforcing traceability and consistency between them. This makes it very systematic compared to the light methods, and consequently we can train developers in a systematic way to understand good software development practices. We can train people to become competent. Still, yet training needs to be done in a more agile way.
  3. Good software and software developed in an agile way can only be created by smart people. You can only be smart if you are competent. Since RUP helps you to become competent, you also have the preconditions to become smart which helps you to become more agile in applying best practices.

However, we won’t stop here. Already when I started the development of what became RUP back in 1987, I knew we need to take one step more. We need to make the process active and executable, not just a dead book. We need to support the developer with “intelligent” tools. This is why we started Jaczone and developed WayPointer. WayPointer will help us make a big step forward in agility:

  1. One day I believe WayPointer will help configure the PD while you are working. WayPointer will recognize what the team is doing and present a PD that is just enough of process. For a small project it will be a light version of RUP. For a larger project a richer version of RUP will be provided. This will happen as you go, without having a heavy upfront configuration activity. Isn’t this agile?
  2. With WayPointer, training doesn’t happen in classrooms but while you work. Thus no heavy training before starting, instead “train as you work”. Very agile, right?
  3. Finally, WayPointer will help you to also apply the PD. It is very context sensitive, so it can give proposals of what to do in every micro step as you work. It can facilitate the doing of many activities which today have to be done manually, but which should follow one another in a natural way. It speeds up the whole project in a substantial way. This is what I would call agile, and it far exceeds what you can do with what is called agile methods.

To summarize, agility can only be delivered by people, people that are smart. People can only be smart if they have knowledge. Knowledge needs to be shared between the members of the team; otherwise it will slow down the team. Most of the knowledge they need are rules about best practices and implementation technologies. Most of these rules can be captured in a “tool” like RUP, but to achieve substantial agility, these rules should be manifest in another more intelligent tool, making the process active. In that way there is no limit to how agile we will be developing software in the years to come. 

Going Back Home by Ivar Jacobson

July 3, 2004

After three days in St. Petersburg I am now on my way to the city where I grew up and graduated from high school. The city is Ystad located in the very south of Sweden at the Baltic Sea. Every year I spend a week in Ystad to meet old friends, friends I have had since I was 7-18 years old.

To my surprise there are not yet any major software conferences in Russia. Conferences are organized by universities or by international companies like SUN, Microsoft, etc. I gave a talk at the 5th international scientific and technical conference and got a good impression of what people worked with and what was presented. Then I met with academic leaders with a strong network in the software industry. I spent a day at St. Petersburg State University as a guest of Professor Andrey Terekhov.

That day turned out to be very interesting. For many years I have been wanting to develop process and tools to reengineer old software systems to become modern component based software, modeled with UML and implemented on modern platforms such as J2EE or .NET. Andrey told me that he had been working with a company in the US that sells tools and services to reengineer from the bottom and up. I know the company very well. In fact, a few years ago, they offered me to become a member of their advisory board. And now I met the guy who developed their tools.

The reason I didn’t accept the offer was that I didn’t see that their design would be generic enough. They had their own modelling technique, their own process and their own tools. My approach is the one I wrote about 1991 in my OOPSLA’91 paper on Reengineering Old Systems to an Object-Oriented Architecture. However, I liked their low level reverse engineering tools. And these tools were developed by Andrey and his company. I think there is a huge demand to reengineer legacy systems, particularly those written in COBOL or PL/1. I have many customers that would be willing to invest in such tools. I am now thinking about if and how to do this. Thus, the meeting with Professor Andrey Terekhov and his team was really refreshing.

During my stay in St. Petersburg, I enjoyed sightseeing and good restaurants. Elena Ivanova arranged my whole visit and she and her son were great guides. Of course, the Ermitage was very interesting. Also just walking around in the city during the beautiful White Nights and along the Neva River was very relaxing and romantic. And I love Beluga caviar. I had caviar with small vodka every night, and carried home as much as you are allowed. The price for 213 g top quality caviar was just 50 USD. At home I would have had to pay more than 1,000 USD. Maybe we are in the wrong business?  

Looking Back on Aspects by Ivar Jacobson

June 6, 2004

Often I am asked which my favourite country is. In addition to Sweden, where I grew up and have family, I have many favourite countries. Most countries have something that makes me feel happy and at home. Singapore is certainly one of them.

Last Thursday I held a seminar in Singapore for more than 200 persons and launched my latest adventure – the forming of a new company, Ivar Jacobson Pte Ltd. We will help organizations to implement the best practices of the unified process. We will also expand the set of best practices, not just help with the existing ones. For instance we will train and mentor organizations in aspect-oriented software development and active software development (using intelligent agents).

One of the best practices is about aspects. The first time I heard the term aspect-oriented programming was back in 1997. I immediately saw it as an interesting technology, but at the time I couldn’t take a serious look at. I was working on the first version of UML, I was working on getting RUP right and I was initiating an effort on active software processes – actually what now has resulted in Jaczone WayPointer. When I finally had time to take a good look at aspects it was in September 2002. I downloaded a lot of papers and studied them for a couple of days. After that I contacted Harold Ossher at IBM Research and Karl Lieberherr at North Eastern University in Boston. They are two of the leaders in this space. The most well-known guy on aspects is Gregor Kizcales. I tried to get his attention as well, but he was too busy at the moment.

In November 2002, I visited the IBM folks and spent a day with them understanding their work. After the meeting I was very impressed and excited about what they had done. I left their office, rushed to Newark airport; I had to run to the gate. This is normal. I was on my way to Stockholm. When I was seated in the plane, I ordered some champagne and began to relax and think a little. Suddenly, it struck me. Didn’t I do something similar before? Didn’t I write a paper on the same subject at OOPSLA’86 – the very first OOPSLA conference?

When I came to Stockholm, I started to hunt for the paper. It was a paper that discussed a topic that I mentioned as future work in my Ph. D. thesis from 1985. However, I got no interest for the ideas in the paper, so I decided to leave the subject. I felt it was too early to push those ideas. So I just forgot about it. My work on component-based development with objects and use cases was so successful so there was no room for new inventions. However, now I wanted to find the paper, I went to the publisher’s web site. I found the paper. I had to pay $95 to download it! My own paper!!!

The title of the paper is “Language Support for Changeable Large Real Time Systems”. In that paper I introduced several concepts – existion which represents a base, extension which represents separate functionality which you want to add to the existion. Instead of modifying the existion to invoke an extension, we used a mechanism to allow the extension to add behaviour into the existion. Thus from the perspective of the existion, no modification was needed. This means that you can add extensions after extensions without breaking the existion. The key idea is this: by keeping extensions separate from the base from the developer’s perspective, the system is much easier to understand, maintain and extend.

The basic idea sounded very much like what aspect orientation research is trying to achieve. But I needed confirmation. Two hours after I downloaded the paper I sent it to Karl Lieberherr. He responded: “Wow Ivar, this is an early paper on aspect-orientation”. He asked me if I had anything more. Since, I throw away everything I don’t work with; my first thought was that there was nothing more. However, I was excited, and my thoughts went back to the time before the paper. My memory asked me, “Didn’t you file a patent for a similar work?”

The patent was filed in 1981 and I made it as an employee of Ericsson. I called the Ericsson patent department and asked if they had saved the application. After a week they came back. I got a copy of the application – in Swedish. The application used typical patent language, so I had actually never understood it. It was written by a patent engineer. However, attached to the application was an internal Ericsson paper that described the whole idea in a couple of pages. It was a quite detailed paper with a practical example. This paper was also in Swedish. I had both documents translated by a professional translator into English, and you can find them on www.ivarjacobson.com (look for published papers and aspect-oriented software development).

The patent was about what we called a sequence variator. It works at the micro-program instruction level. The highlight of the design is this: A program has a list of instructions. To each instruction, I added a flag. If this flag is turned on, it means that there is an extension that needs to be executed at this point. The sequence variator will fetch instructions from the extension, and thereafter resume at the next instruction. This branching is taken care of by the sequence variator. The developer of the original instructions do not need to code branch statements.

Alright, to make a long story short, Karl Lieberherr and Harold Ossher liked my early work. Karl wrote an email where he compared my early work with modern aspect-orientation: Extension ˜ Aspect, extension point ˜ join point, etc. I then wrote a couple of papers on aspects and use cases (see www.jaczone.com/papers) , and I was a result invited to give keynote talks at international conferences on aspect orientation. I was very happy being recognized for my early work. Now I am working on a book on aspects and use cases together with my colleague Pan-Wei Ng from Singapore. You will soon see this book in bookstores. I will be very happy if you buy it :-). I will be even happier if you read it (because this is not the same) :-). You can also ask me for a presentation. After all, I am in the business of promoting best practices in software development.

Page 14 of 16« First...78910111213141516