SEMAT (Software Engineering Method and Theory): A Call for Action by Ivar Jacobson

We are some people who have observed software engineering theory and practice of the past decades and have realized that it is now time to revitalize this discipline. We have been quietly planning a “revolution”.

For those who have been following my columns may know that, for a very long time, I have been talking about that we need a theory of software engineering. See my two blog entries, “A problem to fix: We don’t understand the nature of software engineering” February 2009, and “Someday we must become professionals!” March 2009, which describe my thoughts on this issue when it all started over a year ago.

Since then I have advanced gradually. First, I explored this idea by publishing a paper, entitled “In need of a theory in software engineering”, first in China (in Chinese) and then in Australia. Later on I joined forces with two colleagues of mine, Bertrand Meyer, the creator of Eiffel, and Richard Solely, the chairman of OMG. We have formed a community called SEMAT (Software Engineering Method and Theory). Its goal is to make software engineering a more systematic discipline than it has been in the past.

This effort started with a Call for Action a few weeks ago. To put it in a sharply simplified way, we state: 1) software engineering is not a mature discipline, and 2) this situation has to be changed fundamentally ”based on a solid theory, proven principles and best practices that include a kernel of widely-agreed elements,” etc.

This “Call for Action” has generated huge responses. Its statement is signed thus far by 31 experts in the software development arena who are from around the world and who come from many different disciplines, such as computer science, Agile (Scrum, XP, Lean) CMMI, Unified Process, Eclipse & Jazz, design patterns, testing, metrics, empirical study, languages, etc. It is quite significant that so many influential individuals have endorsed the same initiative and shared the same vision where each of them has different views and approaches to software development.

Beyond that, in less than two weeks, there are over 700 people from around the world who have registered as supporters. The community is growing very quickly. Take a look at our Website (www.semat.org) to get the latest update.

Now the hard work has began. Many of those who signed to support the “Call for Action”, of course, have some ideas or solutions to the problem. But no one seems to have what we would like to find, namely, a kernel that is widely accepted. Thus, instead of now starting with seeking the solution, we would like to start to develop a specification of the solution. If all goes well as expected, as a community, we will jointly present such a specification in three months. By then we will host a workshop, at which different ideas for solutions are supposed to be presented.

The big challenge of this initiative is to achieve consensus among so many strong and driving forces. SEMAT differs from other initiatives in that we are not trying to unify the world around a method, but rather, focusing on the creation of a small kernel that contains elements that we always have when we develop software. We call these kernel elements universals. Furthermore, we insist that industry and research communities should be harmonized. Both of them have their strengths and have their unique roles to play, but they need to work together as a team resulting in a synergy for us to be successful.

We are well aware that what we intend to achieve is not an easy task, and in fact could be quite difficult. Therefore, we discuss it openly within the community. To achieve the goal of SEMAT initiative we need a community effort, which requires all of us to work collegially, constructively, and to put our personal interests aside, Each of us should strive to make a contribution to the software world as a whole. The longer we wait, the longer we will continue presenting the society with the image that software is fragile, full of defects and poorly done. Let us stop wasting time and generate some positive impacts. It is smart!

4 Comments
  1. Garold Johnson | January 17, 2010 at 1:28 am Reply

    avatar

    I want to keep this brief enough to be read and yet summarize my views on a Theory of Software.

    I agree that a Theory of Software would be valuable. I see the development as having 2 parts:

    1) observe what works and what doesn’t in practice and extract workable principles based on it — the Agile approach strives to do this

    2) begin with first principles and form hypotheses, develop practices based on the hypotheses, and return to step 1.

    I will attempt a summarization of a majhor set of reasonig from first principles.

    Assert: Software development (of the sort that needs a Theory of Software) is done by teams of people using practices and processes in an organizational / management / cultural context.

    Assert: These people are knowledge workers, and much can be learned from a study of the work done with other knowledge workers. The book “Changing Softeare Development” Learning to Become Agile” by Allan Kelly makes this case very well.

    Assert: Software encapsulates knowledge: software allows us to do what the developer does without knowing what he knows.

    Corollary: Since software is developed by people, the process sthould take into account their strengths and weaknesses. First and foremost, software development is about strategies for managing complexity and supporting attention to detail, both of which are weak in people. At any level of abstraction and at any point in the space, the size of the “working set” has to be small enough to be encompassed by a single mind. We use various mechanisms of decomposition, chunking, and pattern recognition to reduce the complexity to a level that we can manage.

    Corollary: Software development can never proceed in a totally top-down manner. People think associatively, possible even holographically. Our systems whould allow us to capture knowledge whenever it occurs and make us of it later. David Parnas in “A Rational Design Process: How and Why to Fake it” contends that no system of any size was ever built strictly top-down and none likely ever will be, but that we can endeavor to construct the trail of documentation that would have resulted had we been able to do so.

    Corollary: Since software is built by teams, all of the team-building and communication processes are of great importance. As the productivity of individual varies dramatically, so does the productivity of teams. When a team jells, individuals do what is needed when it is needed with little or no direction, at any point the focus is on the person who is “on” and has something to contribute at that time. Individual strengths are capitalized upon, and individual weaknesses are strengthened.

    Corollary: Since development is done in a management context, a workable Theory of Software must address that context as well as the skills of the developers. A team whose management requires that 100% of the requirements be done and frozen before development can begin starts out with a tremendous handicap. The Future Combat Systems Program run by Boeing for the Army was planned with a single requirements phase to cover nearly 3 decades worth of development, and it doomed the project from the start.

    Corollary: Since this is knowledge work, our tools and practices should facilitate that work. There is a reason that wikis work well in discussing a proposed system. There are reasons why Use Cases work well. Humans have been using stories and narrative to communicate for as long as there has been spoken language, and we need to facilitate and utilize that communication. Note that all such collaborative efforts have found the need for regular refactoring when an are gets too jumbled, or enough work has been done to be able to discern usable patterns in the material. Conventional documents have more in common with PhD theses than with communication tools. Requirements as merely lists of shall statements lose all context, grouping, rationale, and the coherence that makes them understandable.

    And now we can begin to look at technical issues.

    Assert: At every level of abstraction in the development of a system, the general activity is the same: Given a source of system objectives (needs, desired outcomes, functions, and features) evolve concurrently a system concept (concept of operations), an architecture (what is doing the work), and an expanded description at the new, lower level of abstraction. The process is essentially the same at the high end of system concept development from needs expressed by potential users and domain experts to software coding based on design documents. We don’t need different practices for each level of abstraction.

    Corollary: Where we use natural language, we must endeavor to use it with the same precision as is demanded during coding. Names need t be unambiguous, statements need to be clear and complete, we need data dictionaries to support controlled vocabularies and possibly even full ontologies in the problem domain.

    Assert: The approaches we apply to code to improve clarity, modularize and group descriptions, and reduce duplication work at all levels of abstraction. The application of aspects to Use Cases is one good example, but there are others. Use of interfaces and patterns in requirements can reduce volume, improve clarity, and improve maintainability. For example, for disk-based data, we used to use CRUD (Create, Read, Update, Delete). I routinely see these called out as separate actions on individual data items today. These things cannot be tested independently. They need to be grouped into the equivalent of an interface and specified with a single requirement.

    There is much more to be said, clearly, but I believe that detailed effort on the above with application of experience in software development can provide the basis for a Theory of Software Development that is robust and workable because it si based in reality.

    I first got into computers to get away from people, and discovered during a 45-year career that is is all about people. It is about more than psychology, which Gerald M. Weinberg has studied and written on so effectively I believe passionately that appropriate incorporation of knowledge management can bring a next jump in productivity.


    Thanks,

    Garold L. Johnson

  2. Ron Phillips | December 19, 2009 at 6:38 pm Reply

    avatar

    Hi Ivar,
    A noble goal indeed, and much needed. As James indicates, the current state of practice has much to do with the governance of software efforts. Unenlightened management still (gasp!) views the engineering aspects of software (planning, specifications, testing of specs, shared vision) as expensive overhead.
    Does software construction need to be saddled with government-imposed standards (for example, like building construction or construction of jet aircraft)? I hope not but until those paying the bill realize that the TCO is inextricably tied to what they view as overhead, the state of practice can’t advance much.
    Thanks for your efforts and vision to spread much-needed education and credibility for stronger engineering practices.

    Ron

  3. James Ladd | December 19, 2009 at 5:58 am Reply

    avatar

    Hi Ivar,

    We will continue “presenting the society with the image that software is fragile, full of defects and poorly done” because Software Engineering has little or nothing at all to do with these outcome.

    The Customer has different priorities to the Engineer and these typically drive in a direction that leaves the result as fragile, full of defects and poorly done. The Customer wants the product done, the feature implemented and shipped to make a sale. Whatever that takes, and well crafted isnt on the radar especially if it will slow things down.

    Well crafted doesn’t have to slow things down, but there isnt enough skilled developers to get things done quickly and well. Again, the Customer wants the cheap option here too.

    What the Customer wants is cheap, emergency, temporary housing, but then get upset when you need to spend time and money making it work once they decide they don’t want to stay in it for 10 days, but for 10 years.

    A discipline put in place by Software Engineering is not going to work, the Customer (Product buyer or Service buyer) needs to put things in place, but they wont, because they have other things that take priority.

    I’d like to discuss this more, if you have a contact I could use?

    Rgs, James.