Making the Next Wave in the Agile World – Finding the Kernel of Software Engineering

The 6th annual Agile China was held in Beijing from September 1 – 3, 2011. Agile China is a summit of agile world enthusiasts gathering in China. For the past five years, it’s attracted world-class agile experts, such as Martin Fowler, Kent Beck, Dave Thomas, James Grenning, and Mary Poppendieck et al, and tens of  thousands of attendees across China and overseas. It represents the highest level of development and adoption of agile development in China.

This year’s conference proved to be another successful event. The three-day event was packed by some 700 attendees. The conference was opened by Mr. Wang Jun, General Secretary of China System and Software Process Improvement Association, the host of the conference, and keynote by Linda Rising, the authour of “Fearless Change”, followed by parallel sessions on agile training camp, agile new trend, and agile testing and quality control. People gathered to share their experience and to explore new ideas and  trends. The conference atmosphere was full of energy and excitement. Particularly, I truly enjoyed the free style of discussions and presentations – in a real agile spirit. Read More

Semat – what happens? by Ivar Jacobson

I would like to draw your attention to three recent blog entries:

1) "You are a developer - what is in Semat for you".

2) "Agile in everything".  One of the underlying principles of Semat is that working with methods needs to be agile (not just the methods themselves but working with them).  This implies features not previously found in how to define, use and adapt methods.

3) "A Major Milestone: On the way to a new standard".  An RFP of a standard based on the key ideas of Semat has been issued by OMG.  Letters of Intent are due on November 22, 2011; submissions are due on February 22, 2012.

This is very good progress, but honestly I don't feel the acceptance of the RFP is a sufficient step to declare success.  In the blog "A Major Milestone: On the way to a new standard", we finish by saying:

"Getting the RFP approved by OMG was one of the major milestones of Semat. Quoting Churchill: “Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Now we need to create something that will go beyond anything previously done by any standards body working with methods: getting the standard adopted by the broad developer community.

This is a challenge that cannot be overestimated.  This requires new fresh ideas that are not typical for standard bodies and methods work.  Fortunately, the Semat teams have several such new ideas. ‘Separation of concerns’ and ‘agile in everything’ will guide us, but more is needed.”

We have fresh new ideas for how to describe methods and practices in a very light way, ideas that significantly will improve readability.  The kernel will allow us to not just learn practices easily, but most importantly also allow us to use them during real work.  Earlier approaches have been completely silent on use, but modern approaches such as Kanban and Lean rely on similar ideas.

The number of people working on Semat has more than doubled over the last couple of months.  New chapters of Semat are set up in China and Latin America.  Still we would like to welcome more talented people to work with us.


Use Cases – What is New? by Ivar Jacobson

As we refine and improve use cases we are careful to make sure that we don't impact any of these things that are key to their popularity and success. Use cases as we deal with them today have gone through a major face-lift.  Without really changing the key ideas, the impact of the changes is dramatic.  The result is a fundamentally more efficient way of developing software than the original use cases.

What is new about use cases?
The impact comes essentially from two areas: user stories and aspect-orientation.  The result is that we adapt them for backlog-driven development and managing cross-cutting concerns.

User stories:
In the past we had two concepts – use cases and scenarios.  Scenarios were a kind of user story.  In 2003 we introduced the concept use-case module (published in a paper [1] and in the aspect book [2]).  A use-case module was a slice through the system.  It included a use case (or a part of a use case), its analysis counterpart, its design, its code and its test.  Influenced by Scrum and user stories we sharpened these concepts and improved the terminology.  Now we talk about use cases, stories and use-case slices. Thus we have now:
1)    Use cases are, as they have always been, sets of structured stories (user stories if you want) in the form of flows of events, special requirements and test scenarios.
2)    Each story is managed and implemented as a use-case slice, which takes the story from its identification through its realization, implementation and test allowing the story to be executed.
3)    Thus a use-case slice is all the work that goes together with a particular story. Each story and thus its slice is designed to be a proper backlog element, and realized within an iteration or a sprint.
4)    The use-case strategy (starting from a use-case model) makes it significantly easier compared to the traditional user story strategy to identify the right user stories to work with and to understand how the ones selected make up the whole system.  As the use cases are now developed slice-by-slice, the size of the use cases is not any longer a problem!
Thus, use cases are what they always have been.  Stories are abstract scenarios a la user stories.  Use-case slices are use-case modules made smaller, suitable as backlog entries.   The terms scenario and use-case module will thus be replaced by story to remove the ambiguity between the abstract story-like scenarios and the concrete test scenarios and use-case slice.

Note: this can be compared with the user story approach where:
1)    The stories are captured as a set of unstructured user stories.
2)    Each user story is managed and implemented as one-or-more user story slices, which take the story from its identification through its realization, implementation and test allowing the user story to be executed.
3)    If a user story is too much too implement in one go the story is sliced up into a number of smaller user stories and the original user story disposed of. This illustrates how it is the user story slices that are implemented and not the user stories.
4)    Additional story types, such as Epic and Theme, are added to act as placeholders for user stories that we know will have to be sliced before they can be implemented.

Aspect-orientation has inspired us to deal with not just application-specific use cases (functional requirements), but also with infrastructure use cases (non-functional use cases).  The latter are dealt with as cross-cutting concerns, allowing us to add behavior to an existing system without actually changing the code of the existing system.  Examples of such non-functional behavior are persistency, logging of transactions, security.  This has helped us to deal with requirements (and their realizations) for systems of systems (enterprise systems, product lines, service-oriented architectures), and for partial systems such as frameworks and patterns.  See our book on aspects [ref. 2]

Thus, the key ideas have not changed, but they have been augmented with features that support backlog-driven development and working with non-functional requirements.

Use cases with stories and story slices address many of the issues now raised with the sole user story strategy.  Use cases with cross-cutting concerns address many of the problems analysts have raised with non-functional requirements.  To people, who already had adopted use cases, the new changes are not seen as large, but its impact on the way we develop software is dramatic.

-- Ivar

[1] Jacobson Ivar, Use cases and aspects - working seamlessly together, Journal of object technology, July-Aug 2003

[2] Jacobson Ivar, Ng Pan Wei, Aspect-oriented software development with use cases, Addison-Wesley, 2005.

Use-cases – why successful and popular? by Ivar Jacobson

I am pleased, honored and gratified that use cases are still a popular way of working with requirements.  Googling “use case” yields 6 times more hits than Googling “user story”, but software development should not be driven by popularity.  Instead we should use the most practical way of working.  And, of course we have learnt something from other techniques.  For instance, as I will discuss in my next blog, user stories and aspect-orientation have inspired us to make use cases even better while maintaining their core values.

The popularity of use cases has led to some misunderstandings and some distortions of the original technique.  This is natural, and while it is encouraging to see authors take the original concept and adapt it to solve new problems, some of the misconceptions and distortions have clouded the original vision.

Common misunderstandings

Before further discussing the improved use cases, let’s first discuss common misunderstandings about what we have had since their inception (1986-1992).  Many people believe that:

1) Use cases are for requirements only, which is not true. In fact, from the very beginning, they have also been used to analyze behavior, design software, and drive test identification, just to name a few uses.

2) Use cases are heavyweight; that you need to write fat specifications of each use case, which is also not true.  In fact, use cases can be exceptionally lightweight (a brief description only), to lightweight (just an outline of the flows), to comprehensive (full descriptions of all behavior), and every variation in between.  For most systems an outline can be very valuable and yet still be very lightweight.  Today, we express this in a better way: when describing use cases, focus on the essentials, what can serve as placeholders for conversations.

3) Use cases are a technique for decomposing the behavior of a system, which is also not true.  Some authors have introduced levels of decomposition, and others try to show use cases “calling” other use cases as if they were subroutines.  Neither of these is right.  A use case shows how a system delivers something of value to a stakeholder. Use cases that need to be “composed” in order to provide value are not real use cases.

4) Use cases are complicated.  In fact, using use cases, if done right, makes a system easier to understand.

o It is impossible to understand what a system does from looking at many hundreds of user stories; the equivalent use-case model might express the system’s behavior in a few handfuls of use cases. 

o A user is represented by a stick figure and a use case is represented by an oval.  Their interconnection by a simple line.

o The relationship between a use case and its scenarios are likewise very easy to represent. 

o To solve this problem with user stories, people have started to invent concepts such as themes and epics, making a case that the user story by itself is an incomplete concept. 

o The use-case approach can accommodate a wide range of levels of detail without introducing new and potentially confusing concepts.

5. Use cases are only seen as being good for green field development, which of course is not true.  They are great to explain large legacy systems as with such systems there is often little or no documentation left.  Use case modeling is a technique that is cheap and easy to get started with to capture the usages of the system.

What people like about use cases

The reason use cases have become so widely accepted is that since their introduction they are useful in so many ways in software development. 

1) A use-case model (a picture) already mentioned, which thus allows you to describe even a complex system in an easy to understand way, and which tells in simple terms what the system is going to do for its users. 

2) Use cases give value to a particular user, not to an unidentifiable user community.

3) Use cases are test cases, so when you have specified your use cases, you have also after complementing with test data, specified your test scenarios,

4) Use cases are the starting point to design effective user experiences, for instance for a web site.

5) Use cases ‘drive’ the development through design and code.  Each use case is a number of scenarios; each scenario is implemented and tested separately.

Moving forward
As we refine and improve use cases we are careful to make sure that we don't impact any of these things that are key to their popularity and success.  In my next blog I will describe how we adapted use cases to backlog driven development and managing cross-cutting concerns.

-- Ivar

Semat – moving forward by Ivar Jacobson

Semat moving forward

During the last many months I have been very silent, but not inactive. I have been very active working with a dozen other people on moving Semat forward.  You will soon hear a lot more from us, but I would already now like to give you a quick update on the progress.

As you may recall, the Grand Vision of Semat was to re-found software engineering based on a widely agreed upon kernel representing the essence of software engineering.  The kernel would include elements covering societal and technical needs that support the concerns of industry, academia and practitioners.

The troika (Bertrand, Richard and I) were pleased, honored and gratified to find that in a short period of time, a dozen corporate and academic organizations, and some 3 dozen well-known individuals from the field of software engineering and computer science, became signatories to support the vision.  In addition, more than 1400 other supporters agreed to the call.

In November 2010, the troika agreed that we would move the work on the kernel to OMG (Object Management Group) to get the proper governance we needed.  Since then we have been working in three different but overlapping groups on three tasks:

Moving the development of the kernel to OMG.

In order to move the work to OMG, OMG first needed to submit a request for proposal (RFP).  A couple of people from Semat have worked together with a couple of OMG members to specify an RFP for what now is called 'A domain-specific language and a kernel of essentials for software engineering.’  Early December 2010, an early version of this RFP was presented to the Analysis and Design Task Force of OMG in Santa Clara. It was very well received.  Several other OMG members have now joined us to work on the RFP, which will be published within a few weeks.  March 21-24 the RFP will be discussed at an OMG meeting in Arlington/Washington DC.  We hope and expect it to be approved, and thus the work on proposals can start.  Anyone can submit a proposal, and so we will too.

Our proposal to a kernel

Semat itself will of course not give a proposal to the RFP, but key players are now working together to continue the work we started within Semat.  There is one team lead, Paul MacMahon, who along with 12-15 participants will now continue the work in a couple of tracks.  The idea of doing architectural spikes is continued.  The plan is still to be able to deliver something that can be used by the industry by April 1.  Personally, I think the work has slowed down because of the work with OMG and the continued work on Semat, which I will describe next.  However, we will deliver something of interest and also of value in a couple of months.

The kernel is just a first step in the Grand Vision of Semat.  However, much more work needs to be done.

Even if the development of the kernel now has been moved under the OMG’s umbrella, Semat still has a lot of work to do. We need for example to:

  • be a demanding “customer” of OMG, making sure that as a community, we get what we want,
  • support the community in its effort to get reusable practices,
  • move the work to the academic community to inspire the development of new curricula and useful research.

Thus, a vision for the next couple of years is needed.  A team of 8 people have been working for more than a month to develop a proposal for a Three Year Vision of Semat. This proposed vision should be published within a couple of weeks.  We will focus on what impact we expect to have on three key user groups: the practitioner, the industry and the academia.  The impact should be measurable and not just hand-waving.  How we will work to get the results specified in the vision will be discussed separately.  First we want to agree on where we want to go.

As I am sure you understand, working to ensure that the vision of Semat becomes reality is a challenging task to say the least.  However, it is one well worth the effort.  Please join us.

Potential Barriers to the Adoption of Iterative Practices

Understanding the potential barriers to change will help you make the right choices of timing, project, and approach. Some of the more important questions to ask include in the following:

  • How supportive is senior management of the change? The measurements and milestones
    they establish can easily derail an iterative approach, as is the case when they ask even seemingly
    innocent questions such as “When will the design be completed?” or “When will requirements be
  • What is the scope of your authority to make changes? How much of the development lifecycle
    are you responsible for? For example, the requirements might have already been specified in a
    format and to a level of detail that would make it more difficult to adopt an iterative approach.
  • What are the team’s feelings about the changes? How enthusiastic is the team about iterating?
    To achieve the transition to iterative development, you will need the support of the team,
    especially the other members of the leadership team.
  • What else does the team have to do? How many other projects and initiatives is the team involved
    in? If the team is not focused on and dedicated to the project, the transition to iterative
    practices will probably be slower and take more time and energy to complete.
  • What capability do you need to improve? It is important to understand the capability of the
    team and how well the current capability supports the proposed iterative approach. For example,
    is there any testing capability in the team? Testing will be needed from the first iteration, which is
    often a problem in companies organized around the phases of a waterfall approach. Read More

Progressive Thinking About Planning

The progressive, adaptive, iterative approach is founded on the following principles:

  • Proceed in small steps, continuing steadily by increments.
  • The plans must adapt to the project’s changing circumstances.
  • The characteristics that distinguish the product or service must be progressively elaborated.
  • They should be broadly defined early in the project and made more explicit and detailed as a better understanding of the project emerges.
  • Large projects must be broken up into smaller projects.
  • Requirements errors can be 100 times more costly to fix after deployment than at requirements time.
  • Initial estimates have a large margin of error—most waterfall projects cost two to three times what was initially predicted.
  • The first one never works; always develop things at least twice.
  • Initial designs will be flawed.
  • Requirements must evolve alongside demonstrable versions of the product.
  • Teams need to own and be accountable for their achievements and plans.
  • You should believe the project results, not the project plan.
  • Detailed planning should be limited to the next milestone.
  • Projects should adopt an active and aggressive approach to addressing project risk.

Progressive, iterative processes were developed in response to the problems inherent in the waterfall approach. Instead of developing the whole system in one go, an increment is selected and developed, then another increment, and so on. The selection of the contents of the first increment is based on risk, addressing the highest-priority risks first. To address the selected risk(s), select a subset of the requirements to be implemented and tested. Develop the minimal set of functionality that enables objective verification (through a set of executable tests) that the risks have been successfully addressed. Then select the next highest risks to be addressed in the next iteration, and so on.
One of the great ironies of modern software development is that in the paper that led to the popularity of the waterfall approach to software development, Winston Royce argued that projects should always go through at least two development cycles because the first version never works. He was actually arguing that projects should take a more incremental approach, not that they should adopt a prescriptive waterfall development approach. Progressive, iterative approaches take this principle one step further by splitting the lifecycle up into many smaller increments, each explicitly addressing the project’s major risks.

Measuring Outcomes not Causes – Making Measures Inspirational not Punitive

As discussed in an earlier blog post, you need to measure the desired results.  If Agility and Agile Transformation is going to make you better, then look at the things it is going to make better and set some targets.  If it is going to make you faster, then look at the things it is going to make faster and again set some targets and track against them.  If it is going to make you cheaper and happier, then focus on those things.  This approach begins to make your measurement framework much more inspirational.

Historically, many organisations struggle with their Organisational Measurement getting any traction.  I have seen many organisations where every year they introduce some key performance indicators and by the end of they year, they have abandoned them.  I have seen many organisations with large dashboards with graphs no one understands and no one looks at but a lot of money is spent on them.

You need to change this whole approach.  Measurement needs to change at the organisational level to become more agile – valuing people over processes – collaboration over contracts etc. All the great things from the Agile Manifesto should be visible within your measurement program as well. 

A balanced scorecard organized around the four goals of better, faster, cheaper and happier is a great place to start.  Recently, we worked with KPN and implemented a organizational measurement program using the better, faster cheaper, happier framework. It was an excellent framework for their organisation. It allowed them to look at what is important in terms of getting better – Were they becoming more predictable and were they delivering a higher level of quality?, faster – cheaper - , and happier -    The framework allowed them to look at the problem areas and phrase questions that business and IT could understand. It was so successful that everyone in the whole organization, including the business, embraced the measures with both enthusiasm and commitment. See the case study / webinar for more details.

A better, faster, cheaper, happier balanced scorecard provides a way to communicate the effectiveness of IT in a way that everybody in the organisation can embrace and understand. It also allows you to measure success in a way that is independent of the processes used by the projects and teams being measured – essential if you want to demonstrate that an agile approach is better than more traditional approaches or build a platform to enable you to continually improve and adapt your agile way-of-working.

Light Weight Software Development Process #2

The Software Development Process Challenge

*The first post in this series can be read here

Before I describe the deck of cards, I’d like to set the stage for using the cards. We can view software development from three time-scales (see Figure 1). Successful software development process is in essence, being able to coordinate the three time-scales effectively.

  • The broadest time scale covers from the beginning to the end of a software development cycle which is marked by several key business decision making milestones. This is of great interest to stakeholders and decision makers on whether development can proceed or whether the software is suitable for release.
  • The next time scale breaks the lifecycle into time-periods – months, or weeks, or what is known as an iteration. It is a fixed time-box where a number of target objectives are to be met by a development team. If there are multiple teams, then each team would be assigned their specific set of objectives. This is where team leaders operate.
  • The lowest time scale is what a team member does. The work done here can be in terms of hours or days depending on the complexity of the work.

Light Weight Software Development Process #2

Figure 1:The Three Perspectives to Software Development.

A major problem I see in many development organizations is the serious disconnect between the three levels. The objectives set at the higher level do not easily translate to work at lower levels. Lower levels are unable to see their contribution to higher level objectives. There is a  miscommunication between the levels, and poor coordination within the same level, which leads to blockages which should not even happen. We need to solve this.

More accurate requirements: Who framed Roger Rabbit?

Last June at Innovate 2010 in Florida Kurt Bittner envisioned the new role and responsibilities of the next generation business analyst. If you were not able to attend, his presentation is available online so you can check it out: Transforming the role of the Business Analyst. The need for a different role and responsibilities is to provide solutions for ongoing problems a lot of companies are faced with. These are common problems like:

  • Users expecting  functionality they did not initially ask for
  • Users demanding functionality they will never use
  • Contradictory of conflicting requirements

In order to be more successful, a number of changes are to be made and lessons are to be learned. One of them is that business analysts need to be more focused on desired outcomes rather than features. And another is that business analysts need to probe into root causes rather than being satisfied with just identifying the wants. Being focused on outcomes and unraveling root causes can be hard work and sometimes it is easy to mix them up or to get stuck. A smarter way it is to be more aware of the language that is used for questioning and context frames . Read More

Page 5 of 1612345678910...Last »