Let’s fix the problem: Understanding the nature of software engineering by Ivar Jacobson

In my last blog I described our immaturity when it comes to software development. Companies follow the latest fashion and jump from method to method without standing on what they already have adopted and that is working.  This is expensive and it results in bad software.  When I am upset I say that it is stupid and ridiculous.  How do we change this?

We need a basic theory describing what software development actually is. In my opinion, this theory is right in front of us - we just need to grab it. We can start with all these methods, processes, and practices and find the "truth" of software development. We have already done this in my company and the result has been used by hundreds of companies around the world.

First we started with that core of things that we always do when we build software. For instance, we always write code, we always test it, we always think about requirements (documented or undocumented), we always have a backlog (explicit or implicit), and we always have a plan on a paper or in our heads. It is the “essential core” or kernel of software development, that which cannot be made simpler. To borrow an overused metaphor, this is the DNA for software development.

With my colleagues, I have identified some 20+ such elements by studying about 50 methods, including XP and Scrum. On the surface, there may appear to be large differences in these methods and the ways we work with them. As an example, you can capture requirements with features or with use cases.  But there is a common basis for the two methods, which we capture in our kernel elements.

Then we draw on these kernel elements to describe widely used and proven methods and practices: architecture, Scrum, components, iterations, etc. Today around 15 such practices have been developed. Since the kernel is agnostic in relation to any specific practice, we can simply find out what is the actual difference between different practices, not just on the surface but in depth. This decreases the element of religion in which every method is embedded. Education will become more logical since it focuses on individual ideas instead of the particular soup of ideas that forms every method, process, or methodology. I believe students will love it.

It would be excellent if our technical institutes or universities would educate students in the basics of software engineering, followed up by training the students in a set of good practices using that base. There is also space for a lot of relevant research here.

Remember Kurt Lewin's words: "there is nothing as practical as a good theory." A good theory makes it easy to learn and develop your knowledge without going overly religious.  That would be smart.

8 Comments
  1. Eduardo Vieira | February 22, 2010 at 12:32 pm Reply

    avatar

    Ivar and friends,

    I think that problem is that, as a discipline, we are still in the “pre-industrial” age. The “Agile Cry” is the manifest saying that software is craftwork and it depends more on the people doing in then on recipes or standard of how can anyone build it.

    I believe that the way for reaching the “industrial” age is the use of pre-built components, and I believe that this path is being trailed by some parts of industry (mass market software and games), with the use of engines. These kind of software are built for being sold “pret a porter” and so it can benefit of some kind of scale gains.

    But, how can we solve the problem of the custom tailored systems, which is the majority of the software projects that fail? In these projects, the problem usually is not the implementation itself. It´s in the analisys. Usually, we start building something that we don´t know exactly what it is.

    I think you gave some interesting insights, as the “skinny system”, but I believe that custom development is a intractable problem in itself. I think that today, a custom tailored system project will succeed only when the “businness” understands that analisys is so important to the project that it could not be left for IT alone.

    I am with you when you say that we have to work as a team. The problem is how to make businness understand it.

    Kind Regards (and my apologies for bad english),

    Eduardo

  2. Hassan Diab | December 10, 2009 at 7:39 pm Reply

    avatar

    “Anyone can develop a software at anytime. Few courses in progarmming and/or managment and anyone will be ready for developing software.”!! It’s the feeling we get when talking to people without any background or experience in software engineering. In the other hand, no one can claim to be mechanical, electrical or civil engineer.
    Unfortunatley, this is the reality. Anyone may develop software but not anyone can develop a good software. When civil engineer works on building up a house, there are standards that should be respected. But what about developing software!! This is why I believe a theory is a need and necessary but it’s not enough. Sofwtare standards and normes should also be defined and related to the theory.

    Regards
    Hassan

  3. Ivar Jacobson | May 20, 2009 at 2:40 pm Reply

    avatar

    You don’t give me any clue on why you disagree.

    Of course everyone tests! Of course testing is part of the kernel I describe, the DNA of software engineering.

  4. James Eisenhauer | May 20, 2009 at 2:24 pm Reply

    avatar

    I have to disagree. Everyone does test.

    The difference is how good they test and the techniques they use. Compiling is a form of test. I am sure every developer runs their code once to make sure its working.

    Stating a bunch of methods for better testing kinda misses the point of the article.

    That being when we break it down to the DNA these are things that we naturally have to do, more like the physics of software development.

    Then when we evaluate all of these methodologies we do so from these core areas understanding the natural problems that occur in this space and how a new method may help us deal with it.

  5. Ivar Jacobson | March 18, 2009 at 8:24 pm Reply

    avatar

    Hi Andrew,
    In the software world we have always had two conflicting views of what development actually is. Is it an art or is it an engineering discipline. For sure it has a component of creativity, but most people developing software are solving similar problems over and over. There is a scientific study suggetsing that 80% of the work is no-brain work. We apply the same patterns over and over again. This is probably true in many other disciplines as well.
    I worked at Ericsson at the time when switches were developed using relays. Then we moved to build switches using computer-based systems. The nature of development didn’t change. Personally, I don’t see a fundamental difference between software development and hardware development. Both are engineering. This is true for all kinds of software, such as banking, insurance, defense, automation, robotics, etc.
    Thus you need skills, knowledge and engineering to develop software. However, one of the most serious problems in software is that we really don’t have a theory of what software engineering really. This is why we are so fashion-oriented and run after all these silver bullets.
    Best regards
    Ivar

  6. Andrew Gordon | March 10, 2009 at 8:39 pm Reply

    avatar

    Ivar,

    I think the main issue is the term ‘Software Engineering’. Having originally trained as a Mechanical Engineer then moved over to software, I’m not sure that software can be truly classed as an ‘engineering’ discipline in the same way mechanical, civil and electrical can.

    Most of the engineering disciplines are bound by physics and therefore, to a certain extent, repeatable and deterministic. Software, on the other hand is a virtual construction and does not have the same properties as other engineering systems.

    I think we are possibly locked into this belief that improving engineering processes will deliver higher quality software, on time, on budget. We possibly miss out the major factors that tend to influence success, and that’s the capabilities of the people building the system, how they are organised and the culture they operate in.

    On this subject I’m a fan of Robert Glass and his views that the skills, knowledge and creativity of the individuals involved in developing a software system a the biggest factor in a projects outcome.

    Process does have an influence and having good process embedded in your projects is desirable, but the capabilities of your people has a greater influence on success in my view.

    May be we should be building ‘Capability Models’, comprised of knowledge, skill, culture as well as process, rather than your classic SSADM, DSDM, RUP, SCRUM etc?

    I do agree with your view that at the core, or kernel as you call it, there are the key disciplines that make up every method or process framework that’s every been developed. I think your are right in that we need to equip software professionals with these kernel skills if we’re going to more successful in the future.

  7. Ivar Jacobson | March 8, 2009 at 9:44 am Reply

    avatar

    Dear Daniel,
    Thanks for your long comment. Unfortunately, I cannot comment on everything you brought up, but let me say this regarding your comment that testing is not something everyone does:

    Everyone should test… because if the development team does not test then the users will have to do it. “Not testing” is really just outsourcing testing to the users – which they may not be happy about.

  8. Daniel Wildt | March 6, 2009 at 6:41 pm Reply

    avatar

    Sorry for this big comment…

    “First we started with that core of things that we always do when we build software. For instance, we always write code, we always test it,…”

    What? We always test it?

    That’s not something everybody does… sorry. You cannot generalize this, because lots of people don’t focus even on the core things you are listing. Here I can be generalizing too :-), but the way I see one question we need to ask is: What do people value while developing software projects?

    It is dysfunctional what we find comparing what happens in companies in the market developing software and what people learn inside educational institutions. What are the values? Most of the time they are not the same.

    For example, here is a story…

    I saw teams in the past doing smoke testing and considering a code ready for production. That’s a “flop testing” concept for me.

    Once in production, they need to work overtime to solve defects and they believe their life is tough since they need to work lots of hours during a week.

    They find it normal.

    They blame other colleagues for their own failure, they blame managers, they blame the business analyst that did not gave more time for the developer to work on a feature, and so forth… but, what about working in prevention of defects?

    No, they start checking who works more. It becomes a competition.

    It’s just a way to see things.

    A perspective. :-)

    A bad one, but still… a perspective.

    The problem is to think that defects are something normal. Actually they can be something normal while we are developing and working with testing teams and customers to validate, but they can’t be something normal in production.

    Sometimes the only metric we can use to measure a piece of code is the number of WTF per minute… but if people could understand how to play with code coverage, cyclomatic complexity, code inspection / code review, code audit, unit testing, root cause analysis, functional testing, performance testing, we would be able to stop counting defects and we would start carrying about prevention.

    I’m not saying that all of that should be in place in a software project, but people need to be aware of those possibilities. The biggest problems is that all of those are available for a long time now, but people don’t use! People don’t value that? Why? This is something to research.

    If teams care about testing, requirements, communication with teams and customers, feedback, continuous improvement, we could be in a different stage. But people see things going bad and they think that’s ok.

    Then they decide to change and what you said happens. They “follow the latest fashion and jump from method to method without standing on what they already have adopted and that is working”. Sometimes they don’t have anything working.

    With the right tools and process we can go well, but with an expected behavior from a team, you can have great results. So I think we need to work on the culture, on the mindset.

    Bottom line here for me is: Prevention is good, reaction is bad.

    From your website (http://www.ivarjacobson.com/services/work-packages.cfm): “Define a suitable culture and working environment to enable your team work more collaboratively, efficiently and agilely.”

    That’s it!

    Having an implicit or explicit backlog, or using Use Cases or User Stories to gather requirements. That’s nothing if a team is not able to communicate, if a team is not able to organize and work together, with synergy, and more important with trust.

    So there’s a lot more on software development than a software development life cycle, or tools to buy, or a process to follow.

    We need to work on the culture! I believe that after preparing a team, everything else will be easier.

    That’s something I never heard when I was in College but it is something I teach nowadays.

    When I learned quality and testing in college, more than 10 years ago, it was a class with a lot of humor.

    Now I have the opportunity to teach software quality and test in college, but in a serious way.

    It can be “religious” but I share with my students the need to work on prevention.

    Prevention is something I started to value when I read Toyota Production System (TPS) books and articles. I started that 5 years ago. One example is the work from Shigeo Shingo (see http://en.wikipedia.org/wiki/Shigeo_Shingo).

    I agree with you that we cannot be “religious” when teaching, but we need to share perspectives. Some of them are our beliefs and values we follow.

    It’s like talking about Waterfall without reading the article and not remembering the statement Dr. Winston W. Royce does after the picture presenting what we know now as the Waterfall cycle. Quote: “I believe in this concept, but the implementation described above is risky and invites failure”. More of that on http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

    The problem? I learned in college that waterfall method was the thing to do. It was the standard. There are others, but waterfall is the one. More I study and practice, I see that this is not correct.

    But now, lots of teams in the market thinks that waterfall is the thing to do.

    I value iterative and incremental cycles.

    So, maybe the root cause is education? Do we need to review our education process, how do we teach software engineering?