In need of a theory for software engineering by Ivar Jacobson

To an external observer it would appear that the consensus about the way software should be developed changes dramatically every second or third year, more frequently than the whims of fashion. Trends seem to come and go with no rhyme or reason, and it seems that the label you adopt is more important than the results that you get or the things that you actually do.

Are we working in engineering or in a fashion industry?

Have you ever taken the time to investigate a new method or practice only to find that it is just the re-branding and re-gurgitation of ideas that you have seen many times before?

Have you ever got frustrated that every new idea about software development that comes along seems to be at the expense and in aggressive competition with everything that has gone before?

Does it seem to you that following that latest software development fashion has become more important than producing great software?

In their hurry to become fashionable people seem to throw away the good with the bad. Instead of learning from their own experience and building on top of the good things that they do, they heedlessly throw everything away and start with something they believe is fundamentally new. It is as though they have no solid knowledge to stand upon. Thus they swing easily to a new trend without being able to preserve what they have learnt and experienced.

Big companies around the world carelessly discard expensive process and tool investments, almost before they have even tried them. Every project adopts a new method which new team members must first learn and master before they can begin to work.  Every time someone changes their job they have to learn a new approach before they can get on with the real task at hand.  This is not effective; we cannot learn from experience as we are forever starting over.

Nothing new ever really becomes established. Everything is marketed to the trend setters and early adopters, those who most desire to be seen as different and fashionable. People who have moved on even before the new ideas have truly established themselves or shown their real worth. We cannot get anything to stick.

In reality very little seems to change.  As in the fashion world, there is much ado about next to nothing. In something as trivial as clothing this may be acceptable, but with the size of our investment in software this is wasteful, expensive and absurd.

We are all deadly tired of all the buzz that we suffer from.

What about agile?

The latest fashion to sweep the industry is “being agile”.  Now let’s be quite clear here, the “agile” movement has made a very positive contribution to the software industry. It has reminded us that people matter first and foremost when developing software.  This is not really new but it is important, and it did seem to have been neglected by the previous, more technically oriented, fashions such as object orientation, and programming in java. By presenting a set of values the agile manifesto created something robust and resilient that can withstand the waves of change that will be bought on by the next fashion.

It’s a shame that the same cannot be said for the many agile methods that have promoted themselves as supporting the agile philosophy. For a movement that values people over process and tools it has certainly given us a lot of “new” processes and tools. The majority of these have been effective in promoting agile values by putting the team back at the heart of what is done to develop software but in bringing these things back into focus, much has been lost or obscured as new terms are introduced for old things, creating the illusion of something completely new.

The end result of this constant re-packaging and re-branding of old ideas is a constant churn in the way that software development teams work. Rather than re-focusing of peoples’ efforts away from wasted work onto the production of quality software all that seems to happen is an arbitrary renaming of the things they do and produce.

Even with something as correct and beneficial as the agile philosophy it can start to get lost in the churn and hype. Already we are starting to see a backlash against agile and our fear is that the benefits will be lost as early adopters move onto the next bandwagon and the late majority re-assert their right not to change to something that has obviously gone out of fashion.

Your current way-of-working - It’s a soup of ideas with no clear ingredients

Regardless of whether you are using a commercially developed or bespoke software development process, regardless of how it is documented or communicated, regardless of whether you use the latest buzzwords or the language of twenty years ago I am sure that it is really just a combination of bits and pieces found in other published processes with a little bit of local knowledge related to your specific domain or business.

Each new method may be presented as completely new, but in reality they are all (even mine) just re-combinations of old ideas with the occasional new in-sight thrown in for good measure.

This results in a lot of wasted effort as old truths are rediscovered but cloaked in apparently new clothing. New clothing that can prove hard to resist. Younger and less experienced coworkers promote new trends, following new gurus, supported by the hype of a media always hungry for “news”.  Managers who have lost touch with actual development find themselves in a hopeless situation: resist the newest fashion and they brand themselves as out of touch. Pilot projects are run to force proof of merit for the new approach, but motivated developers can make anything work on a small scale.  As a result the new approach overtakes the old, and all that was working with the old approach is thrown out along with the parts that were not.  Only too late do they discover that the new approach itself has parts that don’t work along with the parts that do. It is not survival of the fittest but survival of the most fashionable.

If we can separate the ingredients from the soup we can empower people to build the way-of-working that they need and to be able to adapt the method to cope with a changing industry and incorporate new ideas that complement and build on their own hard won experience.

Addressing the challenge

We need to stop forever chasing after fads and easy answers that forever disappoint without discouraging innovation and new ideas.  We need to stop constantly re-packaging and re-branding old ideas just for the sake of it. We need to help people understand how to build great software.  But how?  This is a problem I have thought about for at the least ten years and of course I have a concrete idea on how we can get there.

At the root of our problems is a deep misunderstanding of the nature of software development.  Researchers have tried to attack this problem with new theories like formalism to prove correctness of programs, or through formal languages which never have been adopted outside the academic world.  Industry efforts have spent years standardizing swollen meta-models that defy easy understanding.

There are many ways for people to learn software engineering but the only way to get good at it is through practice and experience.  Universities, technical institutes and commercial training courses teach us a particular way of working, such as UP, XP, Scrum (or whatever happens to be popular at the time) but they give us very little practical experience. When we later meet the challenges of the real world we find that the way-of-working has changed again and we have no solid knowledge to stand upon.  Thus we swing easily to a new trend without being able to preserve what we have learnt and experienced.  This is not effective; we cannot learn from experience as we are forever starting over.   Thus we need a theory.

In myopinion, this theory is right in front of us - we just need to grab it. We can start with all these methods, processes, and practices out there and find the "truth" of software engineering.

The solution looks somewhat like this:

1.   Find the Kernel – the Mother of all Methods

All methods have a lot in common.  After all, they are all used to develop software, and they all acknowledge that there are certain things that we always need to do when we develop software.  For instance, we always write code, we always test it (in one way or the other), we always think about requirements (documented or undocumented), we always have a backlog (explicit or implicit), and we always have a plan on a paper or in our heads. To borrow an overused metaphor, we want to find the DNA for software development.

We need to find the “essential core” or “kernel” of software development, that which cannot be made simpler.

By studying about 50 methods, including XP and Scrum, my team has identified a kernel with some 20+ elements, things we always do or always produce. On the surface, there may appear to be large differences in these studied methods and the ways we work with them. As an example, you can capture requirements with features or with use cases.  But there is a common basis for the many methods, which we capture in our kernel elements.

2.  Understand the nature of methods or processes

An important insight is that every method is a soup of practices, whether or not the practices are explicitly called-out.  For example, RUP consists of many integrated practices, while Scrum essentially consists of only one practice, project management, and a few agile work patterns.

3.  Describe each interesting method using the kernel.

With the kernel in place all methods can be described in a uniform way, as specializations or extensions to this kernel.  We can harvest the implicit practices from all widely used and proven methods or processes, such as architecture, components, iterations, etc.  Some practices will overlap, e.g. risk driven iterations from RUP and backlog driven sprints from Scrum.  Some practices will complement one another, e.g. use cases and project management.

The kernel clears away the cosmetic differences between methods, such as the same thing being called by different names in different methods.  For example RUP talks about iterations whereas Scrum talks about sprints, but they really mean pretty much the same thing. By doing this we can clearly see the real differences between different methods.

This is not just a theory but something that my team has already started to work on. Today around 15 such practices have been developed using our initial, prototype kernel. Since the kernel is agnostic in relation to any specific practice, we can simply find out what is the actual difference between different practices, not just on the surface but in depth. This decreases the element of religion in which every method is embedded.

It would be excellent if our technical institutes or universities would educate students in the basics of software engineering, followed up by training the students in a set of good practices using that base. Education will become more logical since it focuses on individual ideas instead of the particular soup of ideas that forms every method, process, or methodology. I believe students will love it. There is also space for a lot of relevant research here.

Remember Kurt Lewin's words: "there is nothing as practical as a good theory." A good theory makes it easy to learn and develop your knowledge without going overly religious.  That would be smart.

The first step is to build on the experience of our initial kernel and build a new kernel. It is clear that we in my team made a few mistakes during our initial work but the end result is that we know that this is not an abstract theory but a pragmatic approach that is proven to work. Now we need to build an improved kernel together. One that is more accessible, more intuitive, and more generally applicable. To do this we need your help. To find out more about how you can get involved see further down in this paper.

Realizing the Benefits

This isn’t just something that affects methodologists, process geeks and academics; it is something that will benefit everybody involved in software development.

What is in it for the industry?

Most large companies have their own home-grown method or process, one which is a soup of standard ideas complemented by some of their own more business specific ideas.  Usually these processes are described in a thick book or a website, and lots of money has been invested to document them. Sometimes people are trained on the process, sometimes they are just pointed toward it.

In reality the process is often ignored; the only parts that are actually used are the parts that form the “oral tradition” of the organization.  This is explained by the re-discovered law of nature: people don’t read process books.  New ideas are introduced into the organization, and the old processes fall out of fashion and their books become shelf-ware.

In these large companies there may even be more than one process.  For example the big system integrators may have ten to twenty different processes.  Sometimes they are quite similar, but the similarities are hidden behind all the cosmetic differences.

If your company was to adopt the practice idea, you wouldn’t need to throw out your entire way of working because something new and sexy is becoming popular.  Instead you would improve your existing way of working one practice at a time.  You could even adopt practices used by other companies without throwing out those existing practices that seem to be working well.

To get started you would look at your existing way of working as a set of practices. Then you would look for your pain points and complement your existing way of working by removing any practices that are not working and replacing them with practices that address the weaknesses.  This is easy to do once you have understood the kernel and its usage.

In a large organization with many different ways of working, you can use this approach to successively improve each way of working without forcing everyone to use the same method or process.

This approach will make it easier to adopt new practices without having to change the other practices you have.  Imagine that you have already introduced the kernel and described your practices a couple of years ago.  Then you would be able to introduce Scrum easily by replacing your existing practice for project management by Scrum without having to make any major modifications to your other practices.  Looking into the future, Scrum will most likely be replaced by a new practice, and you will be able to do that easily, too.

What’s in it for the academic and educational community?

Most university professors have made an academic career and have never really had a chance to practice software development on a large scale.  Still they have to teach software engineering which of course is not easy or stimulating.  They just do it because it is in the curriculum and not because they have something to contribute.  They have no theory to teach, just a set of ideas or one specific approach.  When challenged, one professor, who is successful in computer science and who teaches software engineering, said: “Amazingly, the students like to bathe in the mess we ship to them”.  I realize this was not serious, but for sure this teacher was not proud of what he was doing.

A theory will fundamentally change this situation.  Students will learn the basics of software.  They will get a language to communicate about software process, practices, patterns, etc.  I can imagine that they will get a language with a grammar that represents the kernel and language constructs to describe the practices being constituents of the process.  The language needs to be executable so that the practices can come alive.  By this I mean that practices are not just specifications but also executables.  When running a project, the practices will start to run and instances of activities, work products, roles with competencies will be created and populated by real things.  Aspects seem to fit well to model practices.  There are very interesting semantic rules that need to be identified and defined.  There is whole new world that opens up to the students to understand the fundamentals of software engineering.  Not to mention the whole new world for practically and theoretically interested researchers.

What’s in it for the methodologists?

Thinking back on my own career from 1987, I was asked by many people to write a book on the methodology I created.  At that time Objectory had a number of new ideas such as use cases, use case driven development which is a kind of test-driven design, collaborations, sequence diagrams, components and component based development.  Most of the rest was not special.  Implementation, unit testing, system test, performance test, configurations, planning was quite conventional.  Of course, I had experience from the whole lifecycle, but I was not a world-class expert on everything.  However, to write a book I had to cover the whole lifecycle, even if much of it was not my expertise.

With the new theory we are looking for, there will be no need to describe anything else than what is new.  You won’t need to write a book to publish your new ideas and put them in the context of all the other things a software development team needs to do.  You will just describe your new practice or your new pattern and it will be available to the whole world the next day.  Anybody in the whole world with great ideas can contribute and become successful.

What’s in it for the teams developing the software?

Finally we will all be able to escape from endless churn caused by slavishly following fashions and become proper software engineering teams. Teams that build and extend their knowledge by practicing good software development based on a solid foundation, one that is not constantly changing and forcing you to learn the same things over and over again. One that lets you demonstrates your expertise by the results you’ve achieved rather than the courses you’ve attended. One that lets you bring in new ideas and new team mates easily and seamlessly without performance dips or wasted effort.

Final Words

Our understanding of software engineering lacks a basic theory.  As a result, we keep reinventing old approaches with slightly different words, obscuring the real innovations and making it harder to use the good parts of the new while discarding the bad parts of the old.  The theory will help us to substantially improve our education in software engineering.  It will help us to be less naïve in how we react to new ideas popping up around us.  Finally, it will also help us to adopt new ideas faster than we can today.

At its core the theory relies on practices.  Now there are good and bad practice concepts.  The idea of practices has been around for fifty years, but these practices are loosely defined.  We need a practice concept that is more precise.

  1. Our practices need to be designed for the developers and not as we described process historically, for the process engineers.
  2. We need to be able to keep the practices separate, so we can compose them and in that way create our own way of working (our process).
  3. Practices need to be accessible and extensible – providing lightweight practical advice that is easily put into practice rather than comprehensive academic texts. Practices should focus on the essentials – the minimal path rather than the maximal path.
  4. Finally, we need practices that can come alive and be executable. Practices must be more than descriptions.

The real beneficiary of this theory will be the software industry, as has already been proven in many companies.  We will be able to easily educate our people, get them up to speed, improve the way we work with our products, reengineer (a stronger word than refactoring) our products systematically, and continually improve the way we work.

The result will be better software, significantly faster and at dramatically reduced costs. As mentioned above this is something we need to work on together. To find out more about how you can get involved visit www.ivarblog.com and give your feedback to my blog “In need of a theory for software engineering” or e-mail us at [email protected].

87 Comments
  1. Jim Maher | October 20, 2009 at 8:00 pm Reply

    avatar

    Like many commenters, I’m not crazy about the word “theory” in this context. Is there a succinct “theory” of engineering? If so, I don’t know it. How about chemistry, math or physics? Not really; there are many theories. How about building? No; there are sets of best practices and many approaches that are applicable to different types and scales of what is to be built.

    I like better your re-statement “something that explains in a clear and understandable way what software development in all its variations is all about”. To me, that’s a “definition” of software development. WHAT we do, differentiated from HOW we do it.

    Yet the activity you’ve described conducting is a generalization and abstraction of various methodologies and practices – which is HOW we do it. Seems a bit muddled to me.

    I, for one, do not need another software development methodology. PLEASE! I BEG YOU!!!

    I do sometimes enjoy thinking about WHAT we do. And question whether that’s the best way to expend our efforts. Questioning such a “definition” of software development might be fruitful, but after 35 years I refuse to waste any more development time debating HOW we do it.

  2. Vsevolod Vlaskine | October 16, 2009 at 7:27 am Reply

    avatar

    Dear Ivar,

    There is be a way to software engineering theory, which does not seem to be articulated in your article and blog.

    It is hard to express it in a few words, but the theory should not be just a collection of industry experiences or even generalization of them. On the other hand, many engineers are afraid of another grand narrative.

    The theory should enable a software engineer or a team to theorize his own actions, i.e. reflect upon them and make sense of them. The unique quality of software engineering is that it essentially a semantic activity standing as close to the language (natural or artificial) with its expressiveness and meaning-making facilities as literature or philosophy does.

    The other unique quality of software engineering, in which it exceeds philosophy and literature, is it’s doubleness: it should connect two planes of meaning (e.g. requirements and implementation, client and server logic, code and naming conventions, management and team aspects, etc) in a performative artifact (which compiles, runs, and produces results).

    Over the last few decades, there have been great breakthroughs in this semantic approach, leading to intellectual hyper-productivity. In software, it is the theoretical basis of Pattern Languages. More generally, though, works of Deleuze (Bergsonism, A Thousand Plateaus, etc) introduce most important and powerful concepts like molecular vs molar, machinic assemblages, rhisoma, plane of consistency, nomadic science, etc. Without even mentioning software engineering or process, his theory in fact waits to be re-written as a textbook on hyper-productive software engineering and organization.

  3. Bob Harwood | October 15, 2009 at 9:35 pm Reply

    avatar

    I’ve worked in engineering departments, next to “real” engineers. There’s not much different between the fields — except “real” engineers regularly use the disciplines taught in school in their work. Sadly, it’s rare that I see CS grads do the same. Our peers notice that and hence treat our field with disdain. (When was your last discussion on a formal measure of complexity to evaluate two alternatives, the order of your algorithm, solving a problem using known solutions, or the net present value of two possible technical options based on capital investment, time to market, estimated cash flow, etc..?)

    The main difference between our field and other engineering areas is their reliance on the practices in an accepted engineering handbook and our total lack of one. GoF is probably the closest thing we have, and it only addresses one aspect of our work. SWEBOK just isn’t practical enough to be useful for the average practitioner. The Civil Engineering Handbook would be a good reference model (ISBN 0-8493-0958) for what our field needs.

    Making this more concrete by way of an example:
    We all know we need to perform testing. We have a general consensus on what constitutes good unit testing, whether expressed as “BICEP” or in Meyer’s “Art of Software Testing” as a list of rules. What we need less of is religion on TDD vs in-process. As Larry the Cableguy says, just “Getter done”. So, a handbook should describe the practice of testing and leave the process definition as an exercise to the reader, with perhaps some links to common approaches. The SWEBOK is inverted in this respect, spending pages on process and embedding links on the practice of testing.

    Well, that’s my two cents.. I may be wrong.

  4. Ivar Jacobson | October 14, 2009 at 2:03 pm Reply

    avatar

    Kristen,
    No we don’t have “the theory inhouse.

    We have developed a kernel that worked to describe around 50 very different methodologies. We use it in customer engagements. And it works well. We have presented it at numerous occasions, in keynotes, in workshops. It has also been presented briefly in a few papers with Dr Dobb’s journal. You can download it on our web site. Of course, more should be published, but we have focused on getting working software EssWork. This helps you to work with the kernel and practices on top of it

    However, if we want something universal then we need the participation of many people and many organizations. A “standard” kernel is not developed by a single company. In this aspect, our kernel makes it credible that a “standard” can be developed and used as a basis for research and development.

  5. Kristen Kjoberg | October 14, 2009 at 11:48 am Reply

    avatar

    Ivar, I agree with you that it would be usefull with a theory for software engineering.

    As I understand, you have “the theory” inhouse, what are your plans for presenting it public?

  6. Bob Corrick | October 12, 2009 at 1:05 pm Reply

    avatar

    Did I miss a link to the kernel of the theory? As a working developer, I am very interested.

    I found the contributions here so far to be of a high standard, for which: thank you.

    (PS I arrived here via a link from a Dr Dobb’s email.)

  7. Hans Ewetz | October 10, 2009 at 9:59 am Reply

    avatar

    I do agree with your comments comparing Software Engineering with fashion and politics. However I have some objects against the idea of your view of a single Theory.

    It seems like you are looking for a general (or at least some properties of a) method that is capable of solving what is often called ‘complex’ problems. That is, problems that contain many parts and where the parts do not all have the same characteristics. There exist general methods for solving problems where there are lots identical parts (statistical methods) and methods for solving problems containing few parts where the parts have different characteristic (analytical methods). Unfortunately there are no general methods for solving problems having medium to large number of non-identical parts. And, unfortunately, large software systems are exactly such systems. The methods used for solving ‘complex’ problems are usually heuristic methods that promise nothing but tilts the odds in our favour.

    Viewing software solutions from a different perspective shows why the ‘Single Theory’ is most likely a false start. If we view a software solution in a digital space (being it a ‘human readable’ design or a binary computer program) there are only a finite number of solutions. OK, there is a VERY, VERY, I mean a REALLY large number of solutions, but the number is still finite. The process of creating a software solution can then be seen as ‘finding’ one pr more points in a solution space. Now, how do I find a point in such a vast solution space? Typically some sort of heuristic method must be used. Various IT paradigms (possibly temporal ones) are heuristic methods. There are of course other heuristic methods such as hacking. Yet other methods exist. As PC Bridgeman in ‘The Way Things Are’ points out, ‘… that there are other methods of dealing with the future than by projecting the past. Eating hashish is one, or opening the pages of the Bible at random, or paying attention to dreams.’ (Here, projecting the future refers to inductive methods, which are usually the basis for heuristic methods).

    In any case, different parts of the solution space have different characteristic and will (most likely) require different methods during a search. It’s difficult to imagine that a single theory would have the necessary characteristic for searching through any and every corner of a solution space, which is very likely not to be uniform, but to have an unpredictable and surprisingly complex topology.

    By viewing software development as a search through a finite solution space I believe it’s easier to focus and possibly formalize issues that come up during development of different complex systems. For example, how can we ensure that once a point has been found for some type of problem we can easily move to a ‘nearby’ point when requirements change? Currently re-factoring is often used when moving between nearby points. This is not to say that re-factoring will work in all parts of a solution space. It would also be interesting to investigate and possibly formalize how to avoid ending up in points that effectively are cul-de-sacs. I.e., a complete re-write would be required if new requirements were to appear. Yet another interesting issue is to understand how (changing) requirements would take us to different parts of the ‘solution space. This is exactly what Coplien describes through variability and commonality analysis in his book ‘Multi-Paradigm Design for C++’.

    Based on my ‘discussion’, I can’t see that there is nor, nor will there ever be a single ‘Theory’ for doing software Engineering in the same way as there is no and probably never will be a theory for building machines, a Theory for doing biological engineering or creating new medicines. The one and single Theory that such sciences have in common is the principle that guides all empirically based sciences: a theory must be falsifiable. If the software is the theory, then Software Engineering already has a Theory.

    I apologize for the rather long comment. However, your article addresses a large and complex topic and it’s difficult to make a meaningful response in only a few lines. Of course, I might be completely wrong … and as usual, I learn the most from the people I agree with the least … :-)

  8. David W. Wright | October 7, 2009 at 3:30 am Reply

    avatar

    OK, I did a search on software engineering curriculum, and McGill University came up first, with:

    http://www.mcgill.ca/ece/undergrad/information/se/2009-10/

    There is no magic here. It looks like my 1970′s Computer Science degree overlaid with electrical engineering, extra math, and a shot of commercial IT Methodologies (Requirements Analysis!).

    Perhaps it is true that engineering is the accumulation of experiences,good and bad. The cliche (but probably true) is how may bridges fell down in ancient times before people figured how to do it successfully most of the time. Looks like we still have a lot of experiences ahead of us in software development before anything truly repeatable emerges.

  9. David W. Wright | October 7, 2009 at 3:10 am Reply

    avatar

    My (perhaps limited) understanding of the meaning of a theory is that a good theory explains things. Newton’s Theory of Gravity explains why things fall and the speed they fall at; I don’t know how much this actually helped the gentleman who invented elevators.

    So, I think a theory of software would explain why software behaves the way it does, within the known environment of hardware and such. That will be a good theory… probably a ways off.

    On the other hand, engineering solves problems, like making an elevator go up without falling down.

    So combining the concepts of theory and engineering is probably not a good foundation foe what we want, as many others have pointed out.

    So, back to software engineering. What is the foundation for any engineering? Mechanical? Civil? Electrical? etc? I don’t know (although I could probably look it up), but Discipline comes to mind, a la “The Discipline of Software Engineering”. But as Dennis Miller used to say, that’s just my opinion, I could be wrong…

    Two things, however, that I do know:
    1) Back in my student days (way back), any attempt to categorize software programming as engineering was sneered at by real engineers, all those electrical and civil and mechanical folks. It just wasn’t disciplined enough.
    2) While supporting my offspring’s more recent student days, I saw that Software Engineering is now a topic/discipline/subject taught at Engineering Schools. Obviously something had changed…

    So, my question goes out to Software Engineering students and grads and their professors: what is being taught? Can it help us non-credited IT people do a better job developing software? I want to know…

  10. Ivar Jacobson | October 7, 2009 at 12:51 am Reply

    avatar

    … we have developed software for more than 50 years. We ALL agree on a lot of things, even if there are lots of other things (maybe even more) that we see differently. Today, we have not yet identified that part we all agree on. We need to do that. If we call that a theory, a kernel or something else is not the point.
    The point is that we need to identify what is the basics. That core needs to be extensible to allow variations due to different ideas and new technologies.
    With a better theoretical basis we would be able to better compare new ideas, new methods, new practices, you name it. And, we wouldn’t so easily buy into something that is to 90% just old renamed stuff presented under a new brand. This is what so many of us do today. Personally, I love fashion when it comes to clothes, but when it comes to software it is just too expensive.

  11. Alex Tolley | October 4, 2009 at 5:32 pm Reply

    avatar

    The long run of different programming approaches and development methodologies strike me as very akin to different media and schools of art.

    This might be a heretical idea, but what if there is no more a theory of software engineering than there is of any of the arts, e.g. graphic design?

  12. Warren Porter | October 2, 2009 at 11:57 pm Reply

    avatar

    Here is a quote from a reddit comment on a submission of an article by Joel Spolsky. I think it is very insightful and it may help you.

    http://www.reddit.com/r/programming/comments/9nipa/joel_on_software_the_duct_tape_programmer/c0dj48h

    There comes a point in your career where you realise that most of what people say in our industry is complete shit. There’s dogma, buried in fashion with a light dusting of truth.
    Software devleopment today is much like architecture was before Newton. People traded rules of thumb and pontificated on nonsense.
    Then Newton came along and gave us the tools do the job properly.
    At the moment, there is very little science in software development. There’s a lot of people who think they know how to develop software yet if you stick a handful of these people in a room, they’ll disagree on even the most fundamental points.
    The best thing to do is to ignore what everyone else is doing and just focus on making your customers happy.
    I’m lucky to have discovered this so young (26). Having a pleased customer and just being happy with myself about them liking my work put the joy back in coding. The endless worry of “is my code acceptable?” was replaced by “is my customer happy?”
    I feel that fashion has always dictated the answer to the first question whereas the second question was not only much easier to address, but easier to measure too!
    Happiness comes from having happy customers – not building a cathedral to the god of clean code. I think that was what Joel was trying to say.

  13. Marco Dorantes | September 4, 2009 at 9:36 pm Reply

    avatar

    I wonder if the search for an underlying theory that explain and predict the phenomena (observed facts) in our trade should be, in fact, a research for a meta-theory. Have you, Mr. Jacobson, reflected on the paper “Programming as Theory Building” published by Peter Naur in 1985?
    Alistair Cockburn reproduced it in his Agile Software Development: The Cooperative Game (2nd Edition) and also here: http://tinyurl.com/mwnc2x

  14. Scott Ambler | September 3, 2009 at 11:53 am Reply

    avatar

    I’ve been sitting back and watching this conversation evolve with great interest, and just thought I’d share some thoughts:
    1. The choice of the word “theory” was unfortunate. There’s been several postings, which were interesting and made good points, from people who went off on a bit of a tangent because of this word. Apparently it’s a word that is far more loaded within this community than I thought.
    2. A lot of people are couching their responses in context of the agile/lean movement currently popular today. Some people seem to realize that this is mostly a rehash of older techniques, some apparently have “drunk the kool-aid” of the marketing rhetoric. I suspect that the latter is a symptom of a serious problem within the industry.
    3. In the end it’s contextually situational, or in other words “it depends”. Some people are convinced that agile/lean works much better than traditional/waterfall and I have no doubt that for them it’s true. Yet I am also sure, having actually seen it, that traditional/waterfall works well for other people, mostly because of the culture of their organizations. What works well for you might not work well for others, and vice versa. Hence the need for some sort of body of knowledge where people can pull techniques from and hopefully build something which meets their actual needs.
    4. I suspect that fewer people would have fallen for the Scrum silver bullet had there been an accepted body of knowledge in place.
    5. The SWEBoK clearly isn’t well accepted in the industry, and unlikely ever will be from what I can tell.

    - Scott Ambler
    Chief Methodologist/Agile, IBM Rational

  15. Marco Dorantes | September 3, 2009 at 12:32 am Reply

    avatar

    I am also researching about possible theories upon which software engineering could be based. Findings will help to evaluate the truth of the idea that software engineering is a branch of engineering discipline. So far, it is most likely that software endeavors could be characterized as instances of a social science. See one of my posts about the subject here: http://blogs.msdn.com/marcod/archive/2006/01/11/SeriousSoftwareEngineering.aspx
    Best regards.

  16. John Bailo | August 17, 2009 at 5:26 pm Reply

    avatar

    As a participant and practitioner of SCRUM and Agile all my career (they just never called it that when I wrote my dBase III programs), the only thing I can say is: SCRUM/Agile works, and almost every other “methodology” is completely useless. Sorry, but as an efficient programmer, who tries to deliver on time, using the least materials for a fair price, I twist in pain every time I read about some “methodology drive” multi-million dollar software project that takes 3 years, spends all the budget on “plans” and either gets scrapped, or delivers garbage that ends up being shelved, only to start another multi-million dollar project.

    I like the kernel! You definitely honed it down to what’s required. But as far as building it — iteration, hard work and insight with a drive to always use the least “material” seems like the way to go.

  17. 4johnny | August 14, 2009 at 7:34 am Reply

    avatar

    How is this “kernel” any different than previous attempts? The creators of previous “new” methodologies likely also believed they were doing the same kind of analysis and synthesis of the knowledge space at the time.

    I am not so sure that our methodologies are what is holding back Software Engineering anyway. I would proffer that it will never be a rigorous enough discipline until practitioners are held legally liable, like in other Engineering disciplines.

    • Scott Ambler | June 16, 2009 at 12:11 am Reply

      avatar

      Just wanted to add to Ivar’s response. When the results of your retrospective, or whatever reflection activity you follow, result in ideas which go against the advice in the industry then you should stop and think about it. What is different about your situation? What made you come to a different conclusion than the theory folks? What context is their advice applicable for?

      Realistically, no matter what you identify as a potential improvement it will likely agree with the advice given by one or more pundits and also disagree with the advice of others.

      You’ll need to think for yourselves and act accordingly.

      - Scott

  18. Vili Forsell | August 3, 2009 at 7:29 pm Reply

    avatar

    I wrote the contents of this already in more detail.

    Then I lost it all to bit-space, as usual by a stroke of forgetting to copy before initiating something that refreshes the browser… :)

    Anyway, I think you are correct about the unstoppability of the marketing machine.

    There’s nothing wrong or curious about the rephrasing of old stuff and presenting it as new and we can’t do anything about it.

    It’s how human creativity works in my opinion, copy old stuff and associate it in a new way.

    It’s not in our interests to fight reality, I suppose?

    The problem why not many respect the good practices of our field is because people don’t either need them, or have any interest to understand them.

    And that’s an educational problem.

    In my opinion young people, or why not children, are still enthusiastic about things until life, or bad examples, or something like that make them feel otherwise.

    So that’s the weak point we should strike at.

    We would achieve our better engineering practice through teaching young people general problem solving (which ties into software engineering quite well) and critical thinking.

    There’s nothing new with that.

    The thing which seems to be forgotten often is that we should do it on their terms and skill level.

    Otherwise they wouldn’t listen or learn.

    The aim would be to make them motivated and interested in understanding even difficult concepts.

    And that might be better achieved by supervised learning. ;)

    Anyway, from my viewpoint that is not happening at the moment, the constant education cuts and increasing bureaucracy make sure of that.

    It’s not a business that has a good and iterative quality assurance. It is improved only when people complain, if even then. Politics.

    For example I’m not a naturally gifted person, but my attitude to learning seems to be good; I’m somewhat curious, focused and like understanding, sometimes even obscure, concepts.

    By time I have learned to do that better.

    Why can’t other people learn that too and improve our industry?

    I don’t think we need people who come for a larger pay and just freeride in development teams…
    because that’s exactly what happens when we have a watered-down education system and large intakes.

    And by the way, lower quality standards for freshmen is not all bad, it just means the industry is becoming a more common part of the society.

    And that is a challenge in terms of education to make things simple and easy enough to teach, to maintain at least a shred of the old quality.

    By getting a uniform and general outline for the methods of the industry we would ease the teaching process and ease the work load on the people who actually work in it too.

    I don’t think we can keep up with the amount of “new” information coming up all the time, nobody will have any time to actually work.

    Anyway, sorry for the longish rant, but maybe it will give some ideas to you, or a nice long laugh. :)

    Either way is good.

    I hope I said everything I wanted to say.

  19. David Blake | August 3, 2009 at 5:52 pm Reply

    avatar

    An interdisciplinary approach would be welcome. This seems more like the areas of anthropology and business than the kinds of theory that determine whether the engineering in a bridge is sound.
    I commend you efforts and your thinking.
    I don’t think it will stop the marketing machine for two reasons. First, marketing itself has a kernel and a well-founded theory to draw upon, but that doesn’t keep people from writing books that present old ideas (with different names this time around) as new and remarkable. Second, so many C.S. and C.E. grads from top schools forget all of their formal training and couldn’t tell you the difference between a depth first search and a breadth first search if their lives depended on it. A few of us do retain such ideas after graduation and use these formal ideas in what we do every day. For those of us who enjoy applying formal theories in practice, and who chuckle every time we here someone call themselves a “scrum master,” an intellectual framework like the kernel you seek would be a welcome system for relating development approaches past, present, and future.

  20. Vili Forsell | August 3, 2009 at 4:22 pm Reply

    avatar

    If I am not wrong, software engineering is a trade which involves people in how it works, how it is interpreted and for what it is used.

    People are not static in time or even measurable in many relevant aspects and thus we have on our hands an interdisciplinary field with qualitative information.

    If we are to gather up a theory, then we need to formalize problem solving in general and connect it in a simple way to people and their cultures.
    If it’s not simple, it won’t be used or understood.

    Why?

    People order software, people use it, people read the code and the comments, etc. The fashion -like behaviour of the industry just reinforces this assumption.

    Formal books are rarely read for fun or without something external, forcing motivation.
    As awful as it sounds, we’d have to make it popular culture.

    So what we’re talking about is basically a nice summarum of every scientific discipline involving and connecting people, mathematics, problem solving in general and just plain software engineering.
    It can be done, but it demands a lot of change in attitudes to be even considered seriously.

    So I’m not really expecting that any one of you will react to this much else than a laugh.

    And discipline is something almost as rare as the people who actually are interested in reading theoretical literature; it’s an issue of motivating and inspirational leadership.
    And inspiring and motivating leaders are also rare…
    we need “proper” software engineering to become a habit and part of the culture. Culture and general habit force people to do its bidding much better than hoping for a generation of disciplined and enthusiastic students.

    I could continue onwards to tell about how one of the major limiting factors will be the educational system, which in turn is limited by the governments and their “wise” decisions, etc. But that would be overkill and probably common knowledge to most of you.

    On a positive note, software engineering is a young culture and that makes it much easier to affect than 50 years from now. So it’s not impossible, just incredibly hard and it demands unified action.
    Latter of which is in my opinion already extinct, but what do I know? Sometimes people surprise. Though mainly when cornered, hungry and when survival depends on it. :)

    Anyway, I have for many years been interested in problem solving in general and I know finnish educational system doesn’t teach it in any form. All I know about it, I have had to derive by abstraction from specific problems.

    For example AI, neural networks and analysis of algorithms have been fun stuff in that sense, but to know how people behave I have had to turn to social psychology (the social kind), basic stuff about social anthropology, and stuff like that.

    One good introductory book with different approaches to the areas would probably suffice on that, just to get an idea on what the discussion is usually missing and then diverts into some specific issue and nothing will be solved.

  21. Skip Pletcher | July 29, 2009 at 7:17 pm Reply

    avatar

    If we are offering a theory, we need to describe it in a way which can be tested, that is to say, as something measurable. If observations match expectations (within tolerance), the theory remains unchanged; if measures exceed tolerance, the theory must be revised/supplemented/rejected.

  22. Luca Minudel | July 29, 2009 at 6:10 pm Reply

    avatar

    @Skip Pletcher

    > The more I read in this blog, the more I sense a need to find and offer measures

    An experienced scrum master / agile coach working together with the team should find the answer

    In you are looking for an external answer … maybe this is a smell an experienced scrum master / agile coach is missing, the retrospective is not working or simply you just need to be more patient and give time to let solutuons to mature their results :)

  23. Skip Pletcher | July 29, 2009 at 4:30 pm Reply

    avatar

    The more I read in this blog, the more I sense a need to find and offer measures (my Maslowian ‘Golden Hammer’) in the description of any suggested practice and to apply them in its application. Processes (when described as SIPOC) can be measured in terms of efficiency. But how do we measure a practice within a process?

    What benefit does this practice offer? _or_ What problem does the practice solve? What method(s) might I use to measure how well it is (not) providing the expected benefit or solution? Not only would measures allow us to better select appropriate practices, such a convention would enable us to compare results (within a given context) among differing practices.

    For example: if we get recurrent build errors, ought we to try Fagan inspections, pair programming, automated profiling on compile, or ladle some other choice chunk from among the soups? We are smart enough to know that there is no single practice which is always better than all the rest, are we not also smart enough to suggest a measure which is appropriate for the purpose of that practice? Is change in build error the right thing to count for measuring the effectiveness of these practices, or are these errors symptomatic of something which may be measured closer to the causative act?

    Once discrete measures are considered, the cumulative, synergistic, or conflicting effects of practice sets may be assessed by Work In Process metrics.

    Ultimately, these measures will affect the bottom line, but if the measures of practice sets are rolled to CIO-level metrics, we will remain a fashion industry, offering novelty because it makes a difference (a very low calorie soup of quite untasty chunks) rather than efficiency by doing things better.

    “Taking the temperature” is a measurement.

  24. Ivar Jacobson | July 23, 2009 at 8:02 am Reply

    avatar

    You are right about that most new methodologies have never really been verified. Let me first tell you my own approach. In 1988 we had in my company developed a first version of the Objectory process which ten years later became RUP. It was developed based on my work at Ericsson on a component-based methodology.

    Two observations:
    1) Objectory was based on solid experience over 20 years in projects within a distributed organization with totally more than 5,000 developers involved. Large scale was there. Components, architecture and modeling were there. We could claim that we used UML 30 years before UML was designed. Component diagrams, sequence diagrams, collaboration diagrams, state charts, activity diagrams were all used from 1968. Testing was there.

    2) However, a lot was new. Use cases and their design, implementation and test were new. Object-oriented analysis was new. Implementation using an object-oriented programming language was new. Etc.

    The dilemma is that everything new has to be verified, and that is not easy. It has to be done on many projects, for many products, large and small, in distributed organizations, etc. That takes time. Still it is the only serious way of verification of a new methodology.

    This is why I didn’t publish Objectory until 1992, even if a first version existed in 1988. Substantial changes were made between 1988 and 1992 based on experience in using Objectory in 7 organizations, different product types and for several projects. And all customers at that time were Swedish so we could fix all upcoming problems very easy. Since 1992 the change frequency has been quite low. For instance use cases are almost the same thing today as then. However, between 1988 and 1992 they changed a lot.

    Our approach was probably as serious as you can be given that you still need to make money on what you are doing.

    However, let me tell you an anectode from 1988. I wrote about it in my blog from July 2006: Meeting with Dave Thomas. Dave suggested: “You should write a book about use cases and objects – no more that 150 pages. Then that book will become a market leader. He saw my hesitation. I said something like software development is serious work, you can’t describe that in 150 pages. He responded, of course you can’t, but people want simple solutions, they don’t want complex books. And, then you can come out with an improved version every year, and people who bought the first version want to by every other version. :-) This is how you create market leadership. Of course, I should have done as he suggested.”

    Finally and now back to your comment on the kernel. The kernel we have developed has been in place for four years by now. It is the result of studying around 50 different methodologies and it has been used in 10-20 large organizations. It is still improving, but with less frequency than before. I have no doubt that we (as an industry) can find such a kernel. However, we will need to collaborate to make it happen.

  25. Payman Hodaie | July 23, 2009 at 12:35 am Reply

    avatar

    In my opinion for any theory, or methodology in this case, to truly gain credibility it first has to be verified, preferably in some empirical maner. Lack of verification is one of the problems with existing methodologies. They are extremely subjective.

    Empirical verification makes methodology development objective, and consequently goes a long way in silencing the methodology wars. Distilling the superset of methodologies into a new kernel structure would not do so.

    You also write: “Most university professors have made an academic career and have never really had a chance to practice software development on a large scale.” In my experience, the same could be said about many methodologies.

  26. Skip Pletcher | July 7, 2009 at 5:58 pm Reply

    avatar

    Letting efficiency = effectiveness divided by cost; might we say that, at its most basic, the goal of software engineering is efficiently matching (digital computing) machine behaviour to human needs/expectations? Ultimately, perhaps “complementing human behaviour with machine behaviour” in the service of some business goal?

  27. ivar2 | July 7, 2009 at 12:04 am Reply

    avatar

    Hmmm, it depends on in which context you use the term contextual practice. :-)
    All practices are contextual, so it becomes redundant in most cases.

    Better practices as David Wright suggests seems OK, but why not be happy with just practices or good practices and avoid talking about best practices.

    By discussing terminology we avoid the real problem, to understand what software engineering fundamentally is. Let’s go back to that, please.

  28. Scott Ambler | July 6, 2009 at 2:25 pm Reply

    avatar

    I prefer the term “contextual practice” because the “goodness” depends on the context in which you apply the practice and how you apply it.

  29. David Wright | July 4, 2009 at 4:24 am Reply

    avatar

    Hmmm, I have been using “Better Practices…”.

    http://www.amazon.com/gp/product/1435718534/ref=cm_arms_als_dp

    • Karl Wiegers | November 29, 2009 at 5:26 pm Reply

      avatar

      Meilir, a term I’ve heard used for this phenomenon is “management by BusinessWeek”. Pick a fad, any fad….

  30. ivar2 | July 3, 2009 at 12:44 am Reply

    avatar

    Ellen, there are no universal ‘best’ practices, since we have no objective measures. We can talk about good practices. Within a company you may talk about our best practices.

    I like your expression ‘good practices applied practically’.

  31. Ellen Gottesdiener | July 2, 2009 at 6:11 pm Reply

    avatar

    Useful conversation, and thanks to Ivar for leading it.

    Ivar’s suggestion will help to make adapting more acceptable in the community.

    in the end, adapting practices/processes/techniques – and continual reflection on those adaptations — is what works and helps teams collaborate to deliver value.

    as for me: i don’t believe in ‘best’ practices, rather good practices applied practically.

    ~ ellen

  32. Ivar Jacobson | June 15, 2009 at 11:32 pm Reply

    avatar

    Luca,
    In direct response to your questions:

    - The theory will give everyone a more robust understanding of what software development is all about, so people can better understand different new ideas, practices and avoid silver bullets.

    - The theory will be applicable in everything you do. Of course, the way you apply it will vary dependent on all kinds of circumstances: what kind of software you build, your competence level, your organization, which practices you adopt, etc.

    - If what you have done is healthy, there shouldn’t be a conflict.

    Ivar

    • Scott Ambler | June 10, 2009 at 11:06 am Reply

      avatar

      I think that what we’re seeing in the agile community is a perfect example of reinvention, renaming, and rebranding run amok. To be fair, many of the thought leaders in the agile community are clear that their ideas trace back to SW concepts from the 60s and 70s. However, there are several orders of magnitude more practitioners out there that don’t seem to get it. I’m often amazed to see people on mailing lists spout off about the evils of the Unified Process, only to see them a few postings later describe a “new” technique that they’ve come up with that has been in the UP for over a decade.

  33. Luca Minudel | June 15, 2009 at 10:16 pm Reply

    avatar

    Ivar, Scott,

    - Who is it for the new theory for sw eng and agile CMMI, the scrum master, the team, the product owner, the management ?

    - When should be used, during the project inception, at every retrospective meeting, after every release, at the project end ?

    - When findings from the team during a retrospective meeting conflict with results provided by new theory for sw eng or agile CMMI how should the team manage the conflict ?

  34. Ivar Jacobson | June 15, 2009 at 5:37 am Reply

    avatar

    To all of you that provide comments to my blog “In need of a theory for software engineering”:

    Many thanks, I truly appreciate the interest you have taken in my blog.

    If you don’t mind, please send me an email to [email protected]. Then I can provide you with updates on what I will be doing.

    Cheers

    Ivar

    • Duane Johnson | October 2, 2009 at 8:35 pm Reply

      avatar

      @Paulo, I like your perspective of this. Perhaps we are at a stage in human development where we lack data about ourselves. It is very difficult to measure things like effort, distraction, sleep quality, quality of interaction with others, etc. I tend to look to Kevin Kelly’s ideas on the “Quantified Self” as elements contributing to a new theory of software development–one that includes human dimensions as much as technical ones.

  35. Pedro Galvan | June 15, 2009 at 5:26 am Reply

    avatar

    I like this too. I believe that the definition, agreement and communication of this core set of principles of sw engineering is greatly needed.

    It will do a lot for education of sw engineers, since it would result in having more people who understand principles of sw engineering rather than just follow fads (which hasn’t been very productive). We need sw engineers who are able to say “Ok, X practice from Y process is simply their way of implementing Z principle of sw engineering. And we all know that the benefits, consequences and appliability of Z principle are this and that, and therefore we need to do it (or don’t) in this project/organization.”

    It would also make it a lot easier to “map” process/methods to one another. This is a frequent headache in organizations that need to know how their inhouse process maps to X other process, or that want to evaluate their level of compliance of their process compared to another one.

    Experts in our field would also be enabled to make real advancements, rather than wasting efforts on reinventing the wheel and reexplaining it time and again.

    And in terms of SWEBOK. I agree that this is something different. What we are looking for is an essential set of principles, not an extensive description of activities and techniques.

    - Pedro Galvan
    Editor in Chief. Software Guru Magazine

    PS – Being an ex consultant at Rational, I can now sleep peacefully after reading your comment: “However, the way RUP was introduced and adopted could only result in a failure …”. I know I am not alone. Thanks for taking that burden off our back :).

  36. Skip | June 11, 2009 at 12:38 pm Reply

    avatar

    We theorize from observations, and can do so without the benefit of measurement, but only measurement will permit us to refine theory. So a theory must be expressed in measurable terms.

    Some forms of software engineering practiced decades ago are ‘good’ in that the software they created is still in use today and running acceptably. If COBOL written in 1980 executes reliably, unchanged, today then that was ‘good’ engineering. How do we measure, consistently among differing chunks in our soup, which practices result in ‘better’ software engineering? Is it production cost or maintenance cost? Goodness of fit or mean time between failures? Any theory is incomplete unless it provides some measure against which it can be tested (whether the means to conduct such test yet exists or not).

    We may need to shift our level of thinking above practical engineering if we want to encompass our concerns with theory. Is there a theory of civil engineering? No, but there are theories of communication and flow which support highway construction and (air or sea) port design. Each such theory is not singular, independent, or unifying. Instead, the theories form a mesh (like woven fiberglas) which can be applied to form useful artifacts.

    So the idea of dissecting software engineering into a set of practical needs, and associating each need with a set of practices which may meet those needs, seems quite useful. Evenso, it is the collection of measures about the satisfaction of those needs which will guide us in the direction of a useful set of theories.

    And software engineering is a very human exercise, perhaps we are closer to politics or medicine than painting or shipbuilding, for software is built not only to support but to produce desired changes in dynamic human enterprise.

  37. Scott Ambler | June 10, 2009 at 11:36 am Reply

    avatar

    As you can tell from my various replies to other people’s comments I really like this concept. I suspect that it will be difficult to execute on but that it is something this industry desperately needs.

    I particularly like Meilir’s observation that it’s a combination of fashion and ignorance, but I’d also add in arrogance and greed as well. Too many people believe that they are in a unique situation, therefore they need to “invent” their own process solutions (arrogance). Vendors and consultants need something to sell us, so are highly motivated to “invent” something new to sell us (greed). It will be hard to overcome this human behavior, but we should be able to make it harder to justify these behaviors by having a viable resource available to everyone.

    More thoughts to come.

    - Scott

  38. Meilir Page-Jones | June 8, 2009 at 9:20 pm Reply

    avatar

    Folks, here’s a Dr Dobbs article that also addresses the issue Ivar raises:
    http://www.ddj.com/architect/217701907;jsessionid=QFTH3UZP4XWQ0QSNDLPSKH0CJUNN2JVN?_requestid=246061

    I liked it a lot and it shows that others are puzzling this issue too.
    Meilir

    • Ivar Jacobson | June 1, 2009 at 10:18 am Reply

      avatar

      Mitch,
      SWEBOK is a massive attempt to create a body of knowledge, but it lacks the underlying theory to do this being asked for. To describe the difference in a proper way I would need to write another paper, maybe I will do that.

      However, look at the critique form my friend Grady Booch or see Cem Kaner’s blog http://www.satisfice.com/kaner/?p=7
      and you will understand why we need a theory.

      Thanks

    • Scott Ambler | June 10, 2009 at 10:51 am Reply

      avatar

      As I was reading Ivar’s post I was wondering if someone would bring up SWEBoK. Yes, there’s some good material in it, but I think that the marketplace has pretty much spoken when it comes to the SWEBoK’s consumability. For whatever reasons, SWEBoK is yet another resource that’s sitting on the shelf being ignored by the mainstream practitioner.

      - Scott

  39. Ivar Jacobson | June 7, 2009 at 7:05 pm Reply

    avatar

    Hi Meilir,

    Your RUP anecdote is very typical. I think most people would agree with me that RUP contained a number of good practices such as iterations, use cases, components, architecture. However, the way RUP was introduced and adopted could only result in a failure.

    The most common mistake was “throw out you existing methodology and start using RUP”. You couldn’t easily change one practice at a time. And, all the text you had to read to understand something. It was not designed for the people who matters, developers, but for people who never will use it, the process people.

    Please go to my web site, http://www.ivarjacobson.com, and learn about the Essential Unified Process (EssUP) and EssWork. I think you will be surprised!

    • Ivar Jacobson | June 1, 2009 at 10:26 am Reply

      avatar

      Hi Alan,

      See my next comment to Mitch regarding SWEBOK. SWEBOK is not an alternative.

      Eclipse OPenUP has inspired by us introduced a precise practice concept. Problem is that this practice concept is designed to make process engineers happy, but not to make the developer community happy. It still has the flavor of old RUP. Although RUP includes many good ideas, it failed unfortunately dramatically in implementation.

      My blog is intended to create interest that is common for all of us in the software industry. Thus I am not interested in critisizing other people’s work to much. I feel I have the right to critisize my own work. RUP was my baby so I have taken the responsibility to correct it in its basics.

      If you are interested inunderstanding the differences between OpenUP and our work EssUP/EssWork, please visit our web site [email protected].

      Thanks

  40. Meilir Page-Jones | June 7, 2009 at 2:56 pm Reply

    avatar

    The combination of fashion and ignorance.

    Ivar, you’ve hit the nail on the head with this posting. You make some extremely important statements on issues that have bothered me throughout my entire career in the software industry.

    Since you’ve covered the main topics so well, I can’t add much. But that’s not going to stop me! (;>)) Let me perhaps amplify one or two points.

    Preamble:
    My criticisms of the software world are directed more towards the commercial / industrial realm of software practice, rather than towards the academic realm. In fact, if everyone in the software industry actually had a computer science / software engineering degree perhaps the software world would be a better place.
    End of preamble.

    In February 2003 the CIO of a company near where I live (no, not Microsoft!) suddenly mandated that “we will be a RUP shop by the end of the year”. He didn’t actually know what RUP was, but at some CIO breakfast he’d discovered that other CIOs were apparently using the latest methodology – something called RUP.

    So, without further ado, he put every one of his developers, database administrators, testers, business analysts and so on through a series of RUP training courses – or “sheep dips” as they were secretly known. This was tremendously lucrative for the training company that administered the RUP courses.

    In the spring of 2004, I interviewed there for the position of contract business analyst. During my interview, they proudly showed me what it meant to be a RUP shop. As far as I could tell, it meant drawing some poorly named ovals and stick figures, together with some unintelligible diagrams depicting lines connecting boxes. All this, I was proudly told, was UML. (Incidentally, I was ultimately rejected for the position for the heinous sin of “not being RUP enough”.)

    At a “rubber chicken” dinner in late 2004, I ran into a couple of people from the company, who gave me an update on the current situation there.

    People widely resented RUP because it had been stuffed down their throats by the CIO. Compounding this, there were bitter disputes within teams as to what RUP actually was. Fuelling these disputes was a cadre of fundamentalist RUP political commissars (such as the person interviewing me), attached to each team to make sure that everybody was “RUP enough”. Notably, these RUP commissars enforced form but not substance.

    A further moral-breaking war had broken out between the analysts and the developers. The developers had decided en masse to ignore the diagrams and other documents they were getting from the analysts and just to go ahead and code whatever they thought was right. Finally, the testing people were in utter dismay about the whole endeavor.

    The parting shot of the two people at the dinner was: “Do you know anyone who’s hiring?”

    The problem, of course, was the combination of ignorance and fashion. The CIO was ignorant of what the shop’s problems and opportunities for improvement really were – and he certainly didn’t bother soliciting any opinions from underlings. He mandated a solution that his staff still didn’t understand in any depth – even after months of training.

    Nevertheless, the solution was fashionable; it was a Fad of the Decade. No doubt at the next CIO breakfast he could hold his head up high and say with pride: “We’re a RUP shop too.”

    Now, just like a teenager seeking the in-crowd at school, he could fit in because he was wearing the right label.

    Meilir Page-Jones

  41. David Wright | June 3, 2009 at 5:15 am Reply

    avatar

    I have long been a believer of not following fashion; if a company has managed to implement a methodology and gets good results, I always recommend sticking with that methodology for as long as it keeps delivering good results. That is usually a lot longer than fashionistas will admit, and avoids the cost of change to the next fashion, always more than any immediate benefit.

    As to one soup dominating the menu in the future? Anything is possible, but I can function without it….

  42. Luca Minudel | June 2, 2009 at 11:26 am Reply

    avatar

    Hi Ivar and thanks for your reply

    Any breakthrough from the theory is welcome of course

    In agile sw dev and in sw engineering as it is in biology and surgery and social engineering the theory is mostly driven by understanding emerged by the practices from the field.

    In my personal experience and based on my current understanding, what make hard developing and delivering working useful software in a cost-effective way today are
    - missing/partial information
    - new unexplored technology/application areas
    - unpredictable events
    - lack of skills
    - the people factor

    What it looks crucial is the ability to collect the relevant feedback, to understand it in the current point of time and context and to understand how to best react.

    Do you think we already have enough understanding emerged by the practices from the field to build new theories that can help this ?

  43. Ivar Jacobson | June 2, 2009 at 8:15 am Reply

    avatar

    Hi Luca,

    Yes, software engineering is like other engineerings concerned with the practicalities, but there is a lot of knowledge, rules, patterns, practices that we can learn and reuse over time. Thus we need for instance languages to describe these things. That would be part of the theory we need.

    We spend too much time to just reinvent the wheel, when we could use it to bring software engineering to new heights.

    “There is nothing as practical as a good theory” was first said by Kurt Lewin a leader in social sciences. The key ideas in agile came originally from social engineering.

    Thus, this is as agile as it gets.

  44. Luca Minudel | June 2, 2009 at 7:19 am Reply

    avatar

    > we cannot learn from experience as we are forever starting over.

    When you look at fields like biology, surgery or experimental physics this sentence does not work.
    Also when looking at complexity science this sentence does not work.

    While CS is concerned with theory and fundamentals, software engineering is concerned with the practicalities of developing and delivering working useful software in a cost-effective way (Ian Sammerville)

    This is a key point in Agile sw development

  45. Jerry Van Polen | June 2, 2009 at 4:40 am Reply

    avatar

    Ivar,

    Excellent development of the “software engineering as a fashion-driven industry” notion or meme. Your use of the term ‘theory’ seems a bit under-developed or unexamined. Could it be serving as a placeholder, in part, for a strong commitment toward economy of principles, axioms, or core vocabulary? For a visceral sense that “ten might be too few, and a hundred too many”?

    With respect to necessary diversity in recognizable styles, despite economy in principles, I recall this quote as one concerned with residential design: “Schools or styles of design exist to allow average practitioners to do good work.” If there are magnitude differences in programmer productivity, there might be magnitude differences in likely returns from free-thinking.

    As a side-note, while attempting to check if I was perhaps paraphrasing David Pye, I came across this incisive introduction: http://mike.teczno.com/notes/books/nature-and-art-of-workmanship.html . As quoted in the link, this is the kind of observation which makes Pye an exemplar in meditation on method:

    “What Ruskin is inveighing against is not hard labor, but patient work. He did not realize, or so it seems, perhaps because he never had to work for a living, that a fair proportion of patient tedious work is necessary if one is to take any pleasure in any kind of livelihood, whether it be designing or making, for no one can continuously create and no one ever has. He did not realize there is great pleasure in doing highly regulated workmanship.”

    Cheers

  46. Paulo Sérgio Medeiros | June 2, 2009 at 3:36 am Reply

    avatar

    Ivar, first, thanks for this great “reflexive” post!

    I’m an academic and also work within the industry. I share the same view with you, that software engineering needs theories concerned with its kernel. Also, theories that are focused on the software engineering practice.

    The driving force of theory building is a process of generalization and abstraction. In this manner, it is fundamentally a bottom-up endeavor as a mean for the DNA search.

    At the first step of the ladder we need to achieve a higher commitment of software engineers with science (empirical methods) to produce input for theory generalizations, as for example, doctors are in medicine. When i say “empirical methods” i don’t strictly mean quantitative methods, especially considering the social aspect of software engineering practice. However, we clearly must have better educated professionals in this topic to that happen.

    The other steps of the generalization ladder are for scientists and experienced software engineers. As in other disciplines, again like medicine, that use, for instance, systematic reviews.

    But to have legs to climb the ladder we need above all a definition of what is a software engineering theory. In other words, we must all be within the same perspective to put together our efforts. Some academics are beginning to try that: http://www.springerlink.com/content/v75720085mn87205/

  47. Ivar Jacobson | June 1, 2009 at 4:50 pm Reply

    avatar

    Phil,
    Ten years ago I would possibly have agreed with you, that the industry needs discipline. Today, I have learnt that no single process, methodology, you name it will be adopted in all companies, for all products, etc.
    The big system integratorsmay have up to twenty different methodologies. The big telecom vendors may have a similar situation. They seem to be totally different, but they are not. At the least they have a common core, the kernel I talk about. Then they have a bunch of practices. Some are similar, but that is hidden since they use different terminology.
    Please, read my blog one more time.

    Moreover there will always be competing practices. Once, I felt we had overwhelming arguments to move away from feature-driven development (as we did at Ericsson back in the 1970′s) to use-case driven development. I still think so. However, reality is that we cannot prove that some practices are better than others. There are very few practices that can be accepted as the very best practices — if any.
    Thus we need to accept that there will be different soups (processes) as we have had in the past. Now we want to achieve that at least we can identify what ingredients there are (which different practices there are). We have not been able to do that before.
    In order to get there we need to better understand the fundamentals of software engineering.

    Thanks

  48. phil abernathy | June 1, 2009 at 9:19 am Reply

    avatar

    Ivar,
    If we separate the ingredients from the soup we will either have to eat raw ingredients or put them together to make another soup. Just practices will leave us reinventing the soup every time. We need a recipe for the ingredients…a process…call it waterfall or Agile or whatever. The recipe is not the problem nor are the ingredients nor are the number of recipes around. We just cant seem to follow the recipe to the ‘T’ and continuously improve it in small steps.

    I dont see us needing a theory, I think the basic problem is that we have too many and too little discipline to follow any single one consistently till we know enough to improve it in small steps.

    If we look at what Toyota has done in the car industry we can learn a lot…they have a disciplined process with the principle of small, continuous improvements at the heart. That is their theory and they evolve their practices to match.

    IMHO what the IT industry needs is discipline…not a theory…..or is that a theory in itself ;-)

  49. Mitch | June 1, 2009 at 4:41 am Reply

    avatar

    Ivar,

    After I read your article, I was going to post about SWEBOK, but I see Alan beat me to it.

    I have been doing this for maybe as long as you have, but I don’t see us in need of a theory. I would suggest the experienced practitioner has been performing iterative, incremental development long before the fashion show buzzwords came along.

    I really must disagree that, “Our understanding of software engineering lacks a basic theory.” I would say the basic theory has been around for decades, already captured in software engineering text books, has been practiced for decades, and recently distilled into a Software Engineering Book of Knowledge. Have you read SWEBOK? If so, how is it possible to postulate that there is no basic theory?

    I see that people have not generally figured out that any size software project is actually a team sport, and we have not figured out how to play at a pro level as a team. That takes a lot of practice and discipline, something that we do not get enough of in the real world of “virtual” teams. Also, any non-software developer sees our industry as either voo doo or total commodity, we really have only ourselves to blame as we have not educated the very people we work for on what software engineering is.

    Anyway, interesting article, but I disagree with the premise. We have a basic theory, it works and is just fine. Maybe it is a matter of acceptance…

  50. Alan Kelon O. Moraes | June 1, 2009 at 12:22 am Reply

    avatar

    Ivar,

    there is an effort to build the software engineering kernel you are proposing: Software Engineering Body of Knowledge – http://swebok.org. SWEBOK is under revision to release its next version in 3Q2010.

    Eclipse OpenUP, on the other side, already implements the concept of practices you mention.

  51. Ivar Jacobson | June 2, 2009 at 1:50 pm Reply

    avatar

    Luca,

    Yes, I do.
    I and my team have developed a kernel on which we have been able to describe more than 50 diverse processes/methodologies. We have been able to separate out many individual practices in such a way that they can be composed in any desired combination, even with an existing old process/methodology.

    However, we have not made the effort to formalize our work further than to develop a working tool/platform for this. In order to get something widely usable we need to get great people involved. This work has just now started. I am very encouraged by the support.

    All of you that are interested please comment here and send an email to [email protected].

    Thanks

  52. Scott Ambler | June 10, 2009 at 10:58 am Reply

    avatar

    We should also remember that Toyota typically isn’t reinventing practices from scratch. Instead they often adopt an existing manufacturing practice and then evolve it from there.

    In software development we can benefit from the same approach. I’ve lost track of the number of organizations that I’ve gone into where they’ve cobbled together a bespoke process which looks a heck of a lot like all the other bespoke agile processes which I’ve seen. Worse yet, when they ask me basic questions about how to improve on what they’re doing my response is generally “well, if you were to Google that concept you’d find several articles addressing that very issue, articles which also describe the practice(s) which you just spent several months reinventing yourself.”

    I think that we can do better as an industry than what we’re currently doing now. A little more maturity and discipline amongst practitioners, and a consumable resource of process-related material, would go a long way to improving things.

    - Scott

  53. Scott Ambler | June 10, 2009 at 11:27 am Reply

    avatar

    Here’s an analogy which goes beyond the single soup concept: When I go to the grocery store I have a wide variety of options. I can purchase ingredients for soup, I can purchase cans of soup, or even soup-based pre-packaged meals. The ingredients are only useful if I know how the use them, have a recipe to follow, or have someone to show me how to use them. If not, I can still use the ingredients but will likely have a few learning experiences on my way to developing my own version of soup (assuming I don’t poison myself along the way). The canned soup is ok, and marketing aside Vendor A’s Tomato soup is often very similar to Vendor B’s Tomato soup, and I’ve got a pretty good chance of making soup that day. These soups are mass-produced by the ingredients that I could have used myself. I can also use the ingredients to tailor the soup to something that is more of my own taste. The pre-packaged meals are more complex, addressing the more complex needs when I find myself in a situation where soup just isn’t enough. They also use the same basic ingredients that I could purchase myself.

    The analogy is that with some common practices that we can pull off the shelf, the ingredients, we can put together our own processes in a bespoke manner, we can (hopefully) purchase fairly simple base processes off the shelf that are made from those ingredients, and even complex processes also made from those ingredients.

    - Scott

  54. Scott Ambler | July 7, 2009 at 12:19 am Reply

    avatar

    But “best practices” is an incredibly wonderful marketing term that we’ll never get out of our lexicon. A good practice is trumped by a better practice which is in turn trumped by a best practice. Luckily “bestest” isn’t a real word, so things could be worse. ;-)

    Focusing on modifier-free practices is definitely the way to go.

  55. Skip Pletcher | September 3, 2009 at 4:13 pm Reply

    avatar

    Ivar, does the kernel identified by your team differ significantly from the named items diagrammed on pages 13 and 14 of the first report of a NATO Conference on Software Engineering ( http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PDF )?

    Scott is right, a good BOK would help. But I don’t believe we need a BOK which is independent of the practice definitions; rather, the definitions comprise a BOK, contextually relevant to the practices applied.

    To provide the value of a BOK, however, we need more than description for a practice; we need to see its promised value and how to extract practice-specific measures to determine whether or not the application of that practice is succeeding at delivering that value in our own context. I think the recently revised PSP BOK provides a precise set of definitions and measures, albeit for a very restricted and peculiar scope.

    Then we will have a set of practices for process engineering which combine to form a project-specific BOK. “Process engineering” because XP, RUP, TSP, and Waterfall are less software engineering method sets than they are ways to manage software engineering. The software engineering occurs within them (sometimes in spite of them).

  56. Ivar Jacobson | October 7, 2009 at 1:37 am Reply

    avatar

    Remember the phrase: “There is nothing as practical as a good theory”. We have many good theories (such as Newton’s formula or Einstein’s work), but most theoretical work has not given us anything more than new doctors.
    Thus I understand the resistance to using the word theory in the context of engineering. Personally, I use it to stand for something that explains in a clear and understandable way what software development in all its variations is all about.
    I am OK finding a better term if that would help, but I definitely want to make clear to the academic world what is needed and that we want their help to clarify what we mean and how we can develop that into the future with less fashion than today.

  57. Scott Ambler | October 6, 2009 at 2:12 am Reply

    avatar

    Heretic! Thou shalt be burnt at the stake for thy blasphemy!

    Seriously though, that’s a good observation that should be discussed. There’s an assumption that we’ve gotten to the point where we could reasonably develop such a theory and get it right. That assumption may not be true.

  58. Kerstin Jonsson | October 18, 2009 at 5:58 pm Reply

    avatar

    The notion that there is no theory in art strikes me as odd. Think about music, or theatre. How could there be institutions all over the world playing works of masters if there was no common inderstanding how the manuscripts and scores should be interpreted? There are different schools of art of course, but there is definitely a common basis, or theory, shared by all. I think our industry would profit a lot from a similar theoretical basis, but be careful not to throw out the artistic child already sitting in the bathtub designing software.

  59. Ivar Jacobson | October 7, 2009 at 12:38 am Reply

    avatar

    You are not alone feeling the way you do. It is sad that we as professional software developers have not got further.

  60. Ivar Jacobson | October 7, 2009 at 1:26 am Reply

    avatar

    There is no doubt that some work in developing good software requires talents that can be compared by working with art, but that part is very small in proportion. Most work at big companies are far from art work. We do the same things (actually we follow the same patterns) over and over again. In order to reduce this part, we need to know what it is and how we can get it done smarter. Whether we call this a theory or something else, is not the point. However, I am sure, if we want to make this clear and understandable — that means we need some formalism.

  61. Hans Ewetz | October 15, 2009 at 10:00 pm Reply

    avatar

    Ivar, what would be interesting to me is to understand what properties it is in software engineering that you see as different than properties in other engineering disciplines where complex problems (as described in my previous comment) must be solved Surely, you are not attempting to solve general complex problems no matter what type of substance the system parts are made out of.

    I could easily see that software is more malleable than many other materials. After all, we don’t have to deal with chain saws, screwdrivers and other tools when constructing software systems – instead we can sit down and just tap the keyboard with our atrophying fingers to change structural parts of a software system.

    However, the malleability of software is an illusion since it is not the physical properties of the parts that make it difficult to construct complex systems. Instead it’s the number of parts and the interaction between parts that makes it difficult. Because of this, if you have a ‘Kernel’ it should be applicable to the construction of ANY type of complex system. This, to me, seems VERY unlikely.
    I guess I’m asking what it is that you believe that is so special in software engineering (as compared to other engineering disciplines) that would make it possible to create a general strategy for solving problems?

    So my question still remains: is the idea of a Kernel (or Theory) for solving general complex problems realistic? At best, I could see that a Kernel can find solutions to problems having certain characteristic. Maybe there will be many Kernels – each one specialist for problems having specific characteristic.

    Any thoughts?
    Regards,
    Hans Ewetz

  62. David W. Wright | October 19, 2009 at 3:26 pm Reply

    avatar

    Music theory could be a good analogy (although I am always wary of analogies…).

    I know a little about this because I took music in high-school, still play a bit for fun, and have a son in university as music composition and theory double major. Listening to him, music theory covers a lot of ground. For example, he ‘generates’ his own baroque fugues because he says they follow a set of rules…and that’s why JS Bach could write so many, he had a pattern.

    I have also read a little music history, and have been reminded of religious choral music early in the middle ages; they had a simple notation for guiding the monks, but it was not exact. The notation we use today for writing music was still be developed/invented.

    Does this sound like software development? We have various guides, but no common notation.

    And think about written music. You could call it a model. It is used to communicate to all who read it what the music sounds like. It uses a common vocabulary, much like any good methodology tries to do. (Yes, a conductor has some room to interpret a piece of music, with tempo and dynamics variations, but you would still know it is Beethoven’s 5th…)

    So, a common notation, and patterns/rules for different types of software, that would be a good theoretical basis. The thing is, can we figure it out and adopt it faster than the centuries it would take for music theory?

  63. Ivar Jacobson | October 23, 2009 at 8:48 am Reply

    avatar

    Kerstin!
    Of course most arts stand on some basic theory, but I would guess that the amount of creativity is far larger in music creation than in most software development (where according to some studies more than 80% is no brain work, in particular for large software development endeavers. Having said that, of course we need space for creativity as well, but we shouldn’t make everything we do creative.

  64. Skip Pletcher | October 26, 2009 at 8:17 pm Reply

    avatar

    I see significant distinction between project management and software engineering, but “standards of craft” is an interesting phrase, especially as others speak of “software malpractice.”

  65. John Bailo | October 26, 2009 at 8:54 pm Reply

    avatar

    Yes, of coure. I refer you to The Pattern Language of Alexander:

    http://www.patternlanguage.com/

    There are timeless ways of building things…software has not been around for more than a century, but yet we have learned much.

  66. Skip Pletcher | December 28, 2009 at 10:37 pm Reply

    avatar

    Well said! We are, of course, not a profession unless we have something to profess: a theory.
    I am beginning to understand this may less be a theory based on mathematics and verified by experiment than one based on experience and verified by practice — better yet in a team (ensemble) performance!

  67. Kerstin Jonsson | December 30, 2009 at 5:48 pm Reply

    avatar

    I guess this is one of the main challenges with this initiative to build a theory: How to make people use it? What´s the use of an ever som good theory if nobody makes practice of it? I agree that idealism is not the main characteristic of the average programmer today. Without idealism nobody would be a musician today either. It would just not be worth the money. Maybe we are talking about different species here?

  68. Vili Forsell | January 2, 2010 at 7:57 pm Reply

    avatar

    I’d hazard a guess and say that the main characteristic of an average programmer would be laziness… the easiest/fastest/most familiar solution is the one that is done in practice.

    Now how did that saying go… the best theory is the most practical one, or something like that…

    Anyway, I’d think simplicity and familiarity are important for this to succeed.

    People who are unaccustomed to calculating sums and predicate logic when thinking efficiency and correctness of code, won’t use it.

    On the other hand those who are accustomed to those will more likely use them to approach the said problems too.

    To freely quote Sun Tzu… “Know yourself and know your enemy and you won’t lose a hundred battles”.

    To get a change succeed the developers of the theory should take the “client” into account and cater to his/her appetite.

    Just like in software engineering. ;)

    To get complex solutions familiar to people, the education is of prime importance… but even education is not perfect, nor yield fast results.

    So it can’t be counted upon to change the people the theory is made for such that they adopt it when it’s ready.

    In good software development cycle, in my opinion, the client is part of the project from the beginning.

    The same should be done with the developed theory, in my opinion.

    I believe you have gotten the metaphor I’m using by now… you’re developing a theory ~ you’re developing excellent and useful software for a client.

    I don’t think I need continue anymore; you can fill in the rest. :)

    So the question is… how would you make great and useful software for a client?

    And when you’ve answered that question to yourself, apply it to practice.

    To know your client, you can use all the information you can get from those who research our clients (probably social psychology, psychology, anthropology..?) and from your own experience.

    They’ll probably answer, if you ask.

    They’ll feel themselves useful, which I think is what people want to feel. :)

    It’s like team-work… (in my opinion) if everyone in a team is taken equally into consideration, everyone will also be backing up the results of the team.

    And converse will happen if it doesn’t happen.

    People want to feel they’re a part of the decision making process…

    And the team we’re talking about is our industry, of course.

    What can I say?

    Keep your eyes open and if you see someone leaving out of the discussion, try to get them more into the conversation… to make them feel that they’re part of the team.

    This is getting unclear at a fast pace, as is usual when I get tired, so I’ll wrap this up jsut by saying that consider this a software engineering challenge, where the client is part of the team and should feel he/she is part of the team… and apply your expertise.

    You should do just fine and end up in a practical result, which is familiar and easy to the “client” and thus give a nice foundation for later developments. :)

Newer Comments »