In need of a theory for software engineering by Ivar Jacobson

To an external observer it would appear that the consensus about the way software should be developed changes dramatically every second or third year, more frequently than the whims of fashion. Trends seem to come and go with no rhyme or reason, and it seems that the label you adopt is more important than the results that you get or the things that you actually do.

Are we working in engineering or in a fashion industry?

Have you ever taken the time to investigate a new method or practice only to find that it is just the re-branding and re-gurgitation of ideas that you have seen many times before?

Have you ever got frustrated that every new idea about software development that comes along seems to be at the expense and in aggressive competition with everything that has gone before?

Does it seem to you that following that latest software development fashion has become more important than producing great software?

In their hurry to become fashionable people seem to throw away the good with the bad. Instead of learning from their own experience and building on top of the good things that they do, they heedlessly throw everything away and start with something they believe is fundamentally new. It is as though they have no solid knowledge to stand upon. Thus they swing easily to a new trend without being able to preserve what they have learnt and experienced.

Big companies around the world carelessly discard expensive process and tool investments, almost before they have even tried them. Every project adopts a new method which new team members must first learn and master before they can begin to work.  Every time someone changes their job they have to learn a new approach before they can get on with the real task at hand.  This is not effective; we cannot learn from experience as we are forever starting over.

Nothing new ever really becomes established. Everything is marketed to the trend setters and early adopters, those who most desire to be seen as different and fashionable. People who have moved on even before the new ideas have truly established themselves or shown their real worth. We cannot get anything to stick.

In reality very little seems to change.  As in the fashion world, there is much ado about next to nothing. In something as trivial as clothing this may be acceptable, but with the size of our investment in software this is wasteful, expensive and absurd.

We are all deadly tired of all the buzz that we suffer from.

What about agile?

The latest fashion to sweep the industry is “being agile”.  Now let’s be quite clear here, the “agile” movement has made a very positive contribution to the software industry. It has reminded us that people matter first and foremost when developing software.  This is not really new but it is important, and it did seem to have been neglected by the previous, more technically oriented, fashions such as object orientation, and programming in java. By presenting a set of values the agile manifesto created something robust and resilient that can withstand the waves of change that will be bought on by the next fashion.

It’s a shame that the same cannot be said for the many agile methods that have promoted themselves as supporting the agile philosophy. For a movement that values people over process and tools it has certainly given us a lot of “new” processes and tools. The majority of these have been effective in promoting agile values by putting the team back at the heart of what is done to develop software but in bringing these things back into focus, much has been lost or obscured as new terms are introduced for old things, creating the illusion of something completely new.

The end result of this constant re-packaging and re-branding of old ideas is a constant churn in the way that software development teams work. Rather than re-focusing of peoples’ efforts away from wasted work onto the production of quality software all that seems to happen is an arbitrary renaming of the things they do and produce.

Even with something as correct and beneficial as the agile philosophy it can start to get lost in the churn and hype. Already we are starting to see a backlash against agile and our fear is that the benefits will be lost as early adopters move onto the next bandwagon and the late majority re-assert their right not to change to something that has obviously gone out of fashion.

Your current way-of-working - It’s a soup of ideas with no clear ingredients

Regardless of whether you are using a commercially developed or bespoke software development process, regardless of how it is documented or communicated, regardless of whether you use the latest buzzwords or the language of twenty years ago I am sure that it is really just a combination of bits and pieces found in other published processes with a little bit of local knowledge related to your specific domain or business.

Each new method may be presented as completely new, but in reality they are all (even mine) just re-combinations of old ideas with the occasional new in-sight thrown in for good measure.

This results in a lot of wasted effort as old truths are rediscovered but cloaked in apparently new clothing. New clothing that can prove hard to resist. Younger and less experienced coworkers promote new trends, following new gurus, supported by the hype of a media always hungry for “news”.  Managers who have lost touch with actual development find themselves in a hopeless situation: resist the newest fashion and they brand themselves as out of touch. Pilot projects are run to force proof of merit for the new approach, but motivated developers can make anything work on a small scale.  As a result the new approach overtakes the old, and all that was working with the old approach is thrown out along with the parts that were not.  Only too late do they discover that the new approach itself has parts that don’t work along with the parts that do. It is not survival of the fittest but survival of the most fashionable.

If we can separate the ingredients from the soup we can empower people to build the way-of-working that they need and to be able to adapt the method to cope with a changing industry and incorporate new ideas that complement and build on their own hard won experience.

Addressing the challenge

We need to stop forever chasing after fads and easy answers that forever disappoint without discouraging innovation and new ideas.  We need to stop constantly re-packaging and re-branding old ideas just for the sake of it. We need to help people understand how to build great software.  But how?  This is a problem I have thought about for at the least ten years and of course I have a concrete idea on how we can get there.

At the root of our problems is a deep misunderstanding of the nature of software development.  Researchers have tried to attack this problem with new theories like formalism to prove correctness of programs, or through formal languages which never have been adopted outside the academic world.  Industry efforts have spent years standardizing swollen meta-models that defy easy understanding.

There are many ways for people to learn software engineering but the only way to get good at it is through practice and experience.  Universities, technical institutes and commercial training courses teach us a particular way of working, such as UP, XP, Scrum (or whatever happens to be popular at the time) but they give us very little practical experience. When we later meet the challenges of the real world we find that the way-of-working has changed again and we have no solid knowledge to stand upon.  Thus we swing easily to a new trend without being able to preserve what we have learnt and experienced.  This is not effective; we cannot learn from experience as we are forever starting over.   Thus we need a theory.

In myopinion, this theory is right in front of us - we just need to grab it. We can start with all these methods, processes, and practices out there and find the "truth" of software engineering.

The solution looks somewhat like this:

1.   Find the Kernel – the Mother of all Methods

All methods have a lot in common.  After all, they are all used to develop software, and they all acknowledge that there are certain things that we always need to do when we develop software.  For instance, we always write code, we always test it (in one way or the other), we always think about requirements (documented or undocumented), we always have a backlog (explicit or implicit), and we always have a plan on a paper or in our heads. To borrow an overused metaphor, we want to find the DNA for software development.

We need to find the “essential core” or “kernel” of software development, that which cannot be made simpler.

By studying about 50 methods, including XP and Scrum, my team has identified a kernel with some 20+ elements, things we always do or always produce. On the surface, there may appear to be large differences in these studied methods and the ways we work with them. As an example, you can capture requirements with features or with use cases.  But there is a common basis for the many methods, which we capture in our kernel elements.

2.  Understand the nature of methods or processes

An important insight is that every method is a soup of practices, whether or not the practices are explicitly called-out.  For example, RUP consists of many integrated practices, while Scrum essentially consists of only one practice, project management, and a few agile work patterns.

3.  Describe each interesting method using the kernel.

With the kernel in place all methods can be described in a uniform way, as specializations or extensions to this kernel.  We can harvest the implicit practices from all widely used and proven methods or processes, such as architecture, components, iterations, etc.  Some practices will overlap, e.g. risk driven iterations from RUP and backlog driven sprints from Scrum.  Some practices will complement one another, e.g. use cases and project management.

The kernel clears away the cosmetic differences between methods, such as the same thing being called by different names in different methods.  For example RUP talks about iterations whereas Scrum talks about sprints, but they really mean pretty much the same thing. By doing this we can clearly see the real differences between different methods.

This is not just a theory but something that my team has already started to work on. Today around 15 such practices have been developed using our initial, prototype kernel. Since the kernel is agnostic in relation to any specific practice, we can simply find out what is the actual difference between different practices, not just on the surface but in depth. This decreases the element of religion in which every method is embedded.

It would be excellent if our technical institutes or universities would educate students in the basics of software engineering, followed up by training the students in a set of good practices using that base. Education will become more logical since it focuses on individual ideas instead of the particular soup of ideas that forms every method, process, or methodology. I believe students will love it. There is also space for a lot of relevant research here.

Remember Kurt Lewin's words: "there is nothing as practical as a good theory." A good theory makes it easy to learn and develop your knowledge without going overly religious.  That would be smart.

The first step is to build on the experience of our initial kernel and build a new kernel. It is clear that we in my team made a few mistakes during our initial work but the end result is that we know that this is not an abstract theory but a pragmatic approach that is proven to work. Now we need to build an improved kernel together. One that is more accessible, more intuitive, and more generally applicable. To do this we need your help. To find out more about how you can get involved see further down in this paper.

Realizing the Benefits

This isn’t just something that affects methodologists, process geeks and academics; it is something that will benefit everybody involved in software development.

What is in it for the industry?

Most large companies have their own home-grown method or process, one which is a soup of standard ideas complemented by some of their own more business specific ideas.  Usually these processes are described in a thick book or a website, and lots of money has been invested to document them. Sometimes people are trained on the process, sometimes they are just pointed toward it.

In reality the process is often ignored; the only parts that are actually used are the parts that form the “oral tradition” of the organization.  This is explained by the re-discovered law of nature: people don’t read process books.  New ideas are introduced into the organization, and the old processes fall out of fashion and their books become shelf-ware.

In these large companies there may even be more than one process.  For example the big system integrators may have ten to twenty different processes.  Sometimes they are quite similar, but the similarities are hidden behind all the cosmetic differences.

If your company was to adopt the practice idea, you wouldn’t need to throw out your entire way of working because something new and sexy is becoming popular.  Instead you would improve your existing way of working one practice at a time.  You could even adopt practices used by other companies without throwing out those existing practices that seem to be working well.

To get started you would look at your existing way of working as a set of practices. Then you would look for your pain points and complement your existing way of working by removing any practices that are not working and replacing them with practices that address the weaknesses.  This is easy to do once you have understood the kernel and its usage.

In a large organization with many different ways of working, you can use this approach to successively improve each way of working without forcing everyone to use the same method or process.

This approach will make it easier to adopt new practices without having to change the other practices you have.  Imagine that you have already introduced the kernel and described your practices a couple of years ago.  Then you would be able to introduce Scrum easily by replacing your existing practice for project management by Scrum without having to make any major modifications to your other practices.  Looking into the future, Scrum will most likely be replaced by a new practice, and you will be able to do that easily, too.

What’s in it for the academic and educational community?

Most university professors have made an academic career and have never really had a chance to practice software development on a large scale.  Still they have to teach software engineering which of course is not easy or stimulating.  They just do it because it is in the curriculum and not because they have something to contribute.  They have no theory to teach, just a set of ideas or one specific approach.  When challenged, one professor, who is successful in computer science and who teaches software engineering, said: “Amazingly, the students like to bathe in the mess we ship to them”.  I realize this was not serious, but for sure this teacher was not proud of what he was doing.

A theory will fundamentally change this situation.  Students will learn the basics of software.  They will get a language to communicate about software process, practices, patterns, etc.  I can imagine that they will get a language with a grammar that represents the kernel and language constructs to describe the practices being constituents of the process.  The language needs to be executable so that the practices can come alive.  By this I mean that practices are not just specifications but also executables.  When running a project, the practices will start to run and instances of activities, work products, roles with competencies will be created and populated by real things.  Aspects seem to fit well to model practices.  There are very interesting semantic rules that need to be identified and defined.  There is whole new world that opens up to the students to understand the fundamentals of software engineering.  Not to mention the whole new world for practically and theoretically interested researchers.

What’s in it for the methodologists?

Thinking back on my own career from 1987, I was asked by many people to write a book on the methodology I created.  At that time Objectory had a number of new ideas such as use cases, use case driven development which is a kind of test-driven design, collaborations, sequence diagrams, components and component based development.  Most of the rest was not special.  Implementation, unit testing, system test, performance test, configurations, planning was quite conventional.  Of course, I had experience from the whole lifecycle, but I was not a world-class expert on everything.  However, to write a book I had to cover the whole lifecycle, even if much of it was not my expertise.

With the new theory we are looking for, there will be no need to describe anything else than what is new.  You won’t need to write a book to publish your new ideas and put them in the context of all the other things a software development team needs to do.  You will just describe your new practice or your new pattern and it will be available to the whole world the next day.  Anybody in the whole world with great ideas can contribute and become successful.

What’s in it for the teams developing the software?

Finally we will all be able to escape from endless churn caused by slavishly following fashions and become proper software engineering teams. Teams that build and extend their knowledge by practicing good software development based on a solid foundation, one that is not constantly changing and forcing you to learn the same things over and over again. One that lets you demonstrates your expertise by the results you’ve achieved rather than the courses you’ve attended. One that lets you bring in new ideas and new team mates easily and seamlessly without performance dips or wasted effort.

Final Words

Our understanding of software engineering lacks a basic theory.  As a result, we keep reinventing old approaches with slightly different words, obscuring the real innovations and making it harder to use the good parts of the new while discarding the bad parts of the old.  The theory will help us to substantially improve our education in software engineering.  It will help us to be less naïve in how we react to new ideas popping up around us.  Finally, it will also help us to adopt new ideas faster than we can today.

At its core the theory relies on practices.  Now there are good and bad practice concepts.  The idea of practices has been around for fifty years, but these practices are loosely defined.  We need a practice concept that is more precise.

  1. Our practices need to be designed for the developers and not as we described process historically, for the process engineers.
  2. We need to be able to keep the practices separate, so we can compose them and in that way create our own way of working (our process).
  3. Practices need to be accessible and extensible – providing lightweight practical advice that is easily put into practice rather than comprehensive academic texts. Practices should focus on the essentials – the minimal path rather than the maximal path.
  4. Finally, we need practices that can come alive and be executable. Practices must be more than descriptions.

The real beneficiary of this theory will be the software industry, as has already been proven in many companies.  We will be able to easily educate our people, get them up to speed, improve the way we work with our products, reengineer (a stronger word than refactoring) our products systematically, and continually improve the way we work.

The result will be better software, significantly faster and at dramatically reduced costs. As mentioned above this is something we need to work on together. To find out more about how you can get involved visit www.ivarblog.com and give your feedback to my blog “In need of a theory for software engineering” or e-mail us at a_theory@ivarjacobson.com.

87 Comments
  1. Matthew | May 9, 2012 at 10:12 am Reply

    avatar

    Can you validate this problem with scholarly research?

  2. Vili Forsell | December 30, 2009 at 5:30 pm Reply

    avatar

    So. How to make professionalism pay off?

    What will make the everyday programmers even consider trying to develop their skills further?

    In music, the audience will boo you out, if you’re not clearly good.

    In software, if it’s “good enough”, no-one cares.

    The client will not be interested in the code nor be capable of evaluating it.

    Peer review outside companies won’t work because of trade secrets.

    Peer review inside companies will remain inside companies and divert from others… causing conflicts when work-force changes.

    With much change in work-force and plenty of time, the practices will probably consolidate to something common across all companies… the know-how becomes tradition.

    Maybe mapping the skills and professional tendencies of the programmers and directing them to same teams?

    Theory for that?

    Just a few toughts, if someone wants to pick them up.

  3. Kerstin Jonsson | December 28, 2009 at 9:56 pm Reply

    avatar

    Ivar,
    There are loads of threads here which deserve attention, and I look very much forward to see them in a comprehensive analysis, maybe as part of your SEMAT initiative. I especially appreciate the discussions around the subject of theoretical or scientific training versus professional experience. And I do think there lies a danger in mixing up the two aspects.

    The discipline of Software Development rests (or should preferrably rest) on distinguished and established sciences such as Computer Science, Mathematics, Linguistics, System Theory, Social Psychology, Cognitive Sciences, Motivation Theory and probably a few more. The span is rather large and covers a broad range of university faculties. I don´t think adding something like “A Science of Software Engineering” to the list solves any real problem. The scientific part is there already.

    In my view, what we need most in order to move on is Professionalism. A Professional stance is built on skill, experience and craftsmanship. This in turn rests on theory and subject knowledge, but it cannot be “taught” in the way you can teach theoretical knowledge. A scientific theory can be understood and “learned” if you have a good tutor or by reading a book. But the real insights that lead to a change in behaviour is based on personal experience, preferrably in active company with good role models or mentors. Without any theory the road to excellence will be very long and full of time consuming mistakes. With a good theoretical foundation the professional training will proceed faster and with a more predictable result. What I would like to see in this initiative is an effort to examine the theoretical foundation for different aspects of our profession and then discuss how we can best support the development of professionalism throughout the practitioning community.

    Following an impulse from a previous posting, I would like to encourage a comparison to the discipline of music. No serious person would take a seat in the violin section of a professional orchestra after a three weeks summer class in swedish folkmusic. OK – you did take a few years lessons in school, but you didn´t really grasp the musical notation back then, so that´s why you go for a summer class. Do you really think of yourself as a professional musician after that endeavour? Thought not. But how come so many software developers lack the most basic skills to make them able to work as team members in a software development project? And I bet they get better payed from the start than most of our philharmonic members. If the solo flute is out of tune everybody suffers and she will hopefully be exchanged. If a cellist repeatedly ignores the conductor´s demand for a change in phraseing, he will not be in the next production of that conductor. Play down bow when the rest of your section plays up bow and your colleagues stop talking to you. There is a wealth of learned theory and internalised behaviour connected to the role of musician in the orchestral teamwork.

    Why can´t we claim the same for our “profession”? At the moment I am trying – in the role as software architect – to implement use case driven iterative development in a new team with rather senior developers. I can see a whole bunch of reasons why this doesn´t work. Probably some of them relate to my own incompetence, but the main obstacle seems ro be the lack of a common understanding of how software is developed. Iterative development is how I have learned and prefer to work. These guys obviously prefer to work in other ways (but they fail to tell me how) and the project delivery is in danger. In the orchestra I as a conductor would know how to instruct a colleague based on our common understanding of “music making”. In this software project, in an organisation lacking support for a development process, I can only pray and hope these guys will ask before they do something really creative and original…. As you, Ivar, would say: “This is unsmart”.

  4. David Pletcher | December 22, 2009 at 11:24 pm Reply

    avatar

    I spend many years at Ericsson designing (and later managing the design) of APZ systems. My view on S/W engineering is still heavily influenced by those practices. Since then I been involved in Systems and software design in Defence, Automotive and lately Maritime.

    I’m disturbed by the tendency of current SW practices, e.g. Agile, that appropriate past practices, wrap them up in new terminology without proper attribution, and repackage those practices as the answer to “everything”. And if you are sceptical, the standard response to reply with the epithet, “waterfall”. The latest fad is kanban – an utterly spurious adaptation from the Toyota Production System that treats requirements/work as if they were standardized units (with well-defined means and variances) that can be picked off a work queue. I wonder if the perpetrators of this movement would push the comparison as hard if they truly knew how it supported TPS objectives. (And the key word is ‘system’ in this context; not collection of processes that can be cherry-picked at whim.)

    It seems to me that in the chase for the next new process, we forget that engineering takes place in the context of what is to be designed, e.g. PC software does not need to be as rigourous as Telecoms software and that should influence our choice of processes, tools and languages.

    It’s a bit of a rant, but I have come to realize that software development is subject to more fads than management is. I think of Peter Drucker’s throwaway line about being referred to as a ‘guru’ – it’s only because ‘charlatan’ is difficult to spell.

  5. Philippe Back | December 3, 2009 at 1:19 pm Reply

    avatar

    … and the point is that a good theory like the one in the kernel is tremendously useful indeed.

  6. Philippe Back | December 3, 2009 at 1:16 pm Reply

    avatar

    On the kernel.

    Is EssUP/EssWork, the best part for me is not in the Practices themselves.

    The best part lies into the kernel and the alphas (+lifecyles), the activity spaces, the areas (green, yellow, blue) and the gameboards.

    Practices are secondary to that when you look at is as achieving outcomes. The kernel addresses outcomes, the Practices do address means.

    I would rather focus on pushing the kernel concepts before doing any Practice evangelization.

    We can coach almost anyone about a practice but we need to align teams and companies on the kernel principles and structures.

    These days people are talking lean, Kanban etc. The Kanban thing at Corbis is just a pattern (in some project management practice) to use to manage the workload. The alpha is the backlog and the states of it are really more centered around moving the state of the alpha forward. Putting Kanban in place is perfectly coherent with the backlog concept in EssUP.

  7. James King | November 22, 2009 at 7:52 am Reply

    avatar

    I found this article and the related conversation very interesting.

    It aligns perfectly with a discussion I had last week when quoting my grandmother while running a project management course.

    I quoted my grandmother saying “never mistake activity for progress” one of the participants put me on the spot by noting that I had quoted my grandmother 3 times in response to questions. He then noted that I was allegedly an agile type project guru and asked if we had learned much about running projects since my grandmother’s time.

    I responded that we have spent a lot of effort learning (and sometimes relearning) many lessons. But we still haven’t gotten around to mastering the simple process of finding out what people need and then delivering a solution that meets their needs.

    I would love to have a better response than that so I am very interested to see what comes next through spending the time to understand the core of the problem (of software development) rather than marketing a partial new solution which, if followed with discipline, will probably result in success.

  8. Ivar Jacobson | November 21, 2009 at 5:39 pm Reply

    avatar

    I want to thank all of you who have contributed to my blog “In need of a theory in software engineering”. You have really helped me understand what concerns and challenges we will face if we try to move forward.

    As an effect we now have reached a community effort which I will tell you more about within a week or so, but I want to quickly let you know this.

    SOFTWARE ENGINEERING METHOD AND THEORY (SEMAT) is the initiative and the community. Please go to our web site http://www.semat.org.

    On this site you will find the Call for Action statement and the people that already have signed up in support of the problem. Note that so far we have intentionally just identified a serious problem in the software industry, we have not provided any solution. This much harder work will start soon.

    For your convenience we (and this letter is signed by me, Bertrand Meyer and Richard Soley) have here extracted the impressive list of signatories:
    Pekka Abrahamsson, Scott Ambler, Jean Bézivin, Dines Bjorner, Barry Boehm, Alistair Coburn, Larry Constantine, Erich Gamma, Tom Gilb, Ellen Gottesdiener, Sam Guckenheimer, Brian Henderson-Sellers, Watts Humphrey, Martin Griss, Ivar Jacobson, Philippe Kruchten, Stephen Mellor, Bertrand Meyer, James Odell, Meilir Page-Jones, Ken Schwaber, Alec Sharp, Richard Soley.

    On our web site we have also have a Question and Answer section. Here you will find answers to some questions such as Why did we go public without having a proposed solution.

    You can all support the initiative by pressing the reply button on the web site.

    Let us know what you think.

  9. Bilal H. Butt | November 4, 2009 at 6:12 pm Reply

    avatar

    Hi Ivar,
    I attended your talk today at MdH. I was just thinking over the evening, about how you believed a single kernel approach will help merge all the existing methodologies. Correct me if I am wrong, but what I understood was that company having 20-30 processes now can have one kernel, with different practices, so as to provide a unified way of modeling. This should certainly help the developer to switch from one department to another, within a company.

    Since you are dealing with generalizing this, then two competitor companies using your kernel solution will have more or less very less difficulty understanding each others process’.

    So even though it seems highly effective for developers to learn this, but my question is does companies really seem to accept this “generic” thing, as they did UP, UML or others, and lose their competetive edge?

  10. David Drell | October 26, 2009 at 4:30 pm Reply

    avatar

    I agree with the need for a theory of software engineering. This has nothing to do with languages and tools. I am not an architect (that is of concrete and steel) but I bet if you look back on the history of creating buildings you will find a common thread of how people conceive, share vision, pool resources, generate motivation, create the conceptual designs, the detailed designs, the execution plans, and finally the execution. This is a well known discipline that when followed yields success at a very high rate. The trick will be forming a software culture which requires the same self discipline and standards of craft to which builders of buildings are constrained to simply by virtue the nature of their building materials. But first, software people need to be educated on the standard ‘theory’ of how humans work together to build complex things. Instead of working on the new software theories, if we simply look at what people have been doing for thousands of years we will realize that software building should be done in the same way that humans have done everything else. The only new aspect will be replacing the natural process limits imposed by hard materials with unnatural, cultural limits to imposed by people on people.

« Older Comments