Ivar Jacobson's Blog

Traditionally many large software organizations have one group to write requirements, other groups design and code, and still others to test, etc. Thus, every group has some form of specialist competency.  This is a kind of “siloed” organization. Project work is moved from group to group, with hand-offs between groups that result in  delay and inefficiency due to loss of time and important information at each hand-off.   This is not agile. Read the rest of this entry »

I am deadly tired of all the buzz that we are suffering from.  Our view on how software should be developed seems to change dramatically every second or third year, more frequently than the whims of fashion.  Big companies around the world carelessly discard expensive process and tool investments, almost before they have even tried them.  Instead of learning from experience they heedlessly start with something they believe is fundamentally new.  In reality very little has changed.  As in the fashion world, there is much ado about next to nothing. In something as trivial as clothing this may be acceptable, but with the size of our investment in software this is wasteful, expensive and absurd.

The latest trend is “being agile” (as exemplified by Scrum).  The “agile” movement has reminded us that people matter first and foremost when developing software.  This is not really new - this theme resurfaces every decade or so as naive managers try to mechanize and commoditize what is basically an exercise in creative problem solving.  It is important that we not lose track of how to work as a team, how to collaborate, how to document what we do, and how to plan our work on daily, weekly, and monthly timescales, etc. But in bringing these things back to focus, much is lost or obscured by new terms for old things, creating the illusion of something completely new.   

The result of this is a lot of wasted effort as old truths are rediscovered but cloaked in apparent new clothing. Younger and less experienced coworkers promote new trends, following new gurus, supported by the hype of a media always hungry for “news”.  Managers who have lost touch with actual development find themselves in a hopeless situation: resist the newest fashion and they brand themselves as out of touch. Pilot projects are run to force proof of merit for the new approach, but motivated developers can make anything work on a small scale.  As a result the new approach overtakes the old, and all that was working with the old approach is thrown out along with the parts that were not.  Only too late do they discover that the new approach itself has parts that don’t work along with the parts that do. 

At the root of this problem is a deep misunderstanding of the nature of software development.  Researchers have tried to attack this problem with new theories like formalism to prove correctness of programs, or through formal languages which never have been adopted outside the academic world.  Industry efforts have spent years standardizing swollen meta-models that defy easy understanding. 

Universities and technical institutes teach us a particular way of working.  Every project adopts a special method which we must first learn and master before we can begin to work.  Every time we change jobs we have to learn a new approach before we can get on with the real task at hand.  This is not effective; we cannot learn from experience as we are forever starting over.   

We need to stop forever chasing after fads and easy answers that forever disappoint.   But how?  This is a problem I have thought about for at the least ten years and of course I have a concrete idea on how we can get there.  However, I will have to wait to tell you that until my next blog.  Let me however say this much:  We need to start small and simple and keep it small and simple.  We did so with SDL and UML.  We can do it again.

That would be smart!

What they don't teach you at institutes of technology – how to be smart

I am slowing down a bit this year.  For many years I have made more than ten around-the-world trips and lots of local traveling, but this year I have made only eight. J  During this time, I have given a dozen keynotes and more than fifty other kinds of talks.  I am often invited to universities or institutes around the world, and I have been offered professor jobs, surprisingly even in Sweden.  Thus, I feel I have quite a good understanding of what they teach you about software.

Interestingly, most professors in software have never built any useful software.  This is natural since they have grown up in the academic world where software engineering is not as fine as computer science.  Of course, this is not true for professors in your part of the world, but elsewhere. J  Thus, most professors can’t possible teach in software engineering with any passion.

As a consequence new ideas about software engineering have come from practitioners around the world.  The best ideas come from practitioners with an appreciation of science, people that agree with Kurt Lewin: "There is nothing as practical as a good theory".  Examples of good theories were Newton’s laws, and Einstein’s relativity theory.  Most research represents useless theories.  Of course, this is not true in your part of the world. J

Personally, I adopted this motto thirty years ago and it has been leading my work all these years: components, use cases, Objectory that became RUP, SDL and UML, and now practices.  Many other people such as Kent Beck and Ken Schwaber have made similar contributions.  This is the true essence of software engineering and it stands on a sound theoretical basis, meaning it can be explained so others can understand it and practice it.  But, very few universities teach it and even fewer do it well.  As I said, this is of course not true for universities in your part of the world. J

However, these exceptional institutes that teach you software engineering practices don’t teach you how to be smart when working with these practices.  And this is fine.  I am happy if they teach good practices, but to provide real value, each practice needs to be applied in a smart way.   The software industry has the capability to help itself in this regard, but I really don’t see that happening.

Smart is about being agile, but it is also about doing the right things, the right way.   You can become smarter through training one practice at a time.  However, without experience your alternatives are too many and only a few of them are smart.  Experience is of course something you can get sooner or later, but it takes time if you must learn by your own mistakes.  This is where the utilization of what we call “smart cases” becomes essential.  We have a large number of smart cases that relate to working with people, teams, projects, architecture, modeling, documentation, testing, process, outsourcing, and many more. 

Each smart case has a key motto for how to be smart, for example:

·       People:  Software is developed by people, not by process and tools.

·       Teams:  A software team is like a sport team with all needed competencies to win.

·       Project:  Think big, build in many steps.

·       Architecture:  Start to build a skinny system, add muscles in later steps.

·       Testing:  Whatever you do you are not done until you have verified that you did what you wanted to do.

·       Documentation: Focus on the essentials - the placeholders for conversations – people figure out the rest themselves.

·       Process: Don’t throw out your baby with the bathwater: start from your existing way of working, find your pain points and change one practice at the time.

And on it goes.

All of this to say, we should learn software engineering practices at school.  We should shorten the time to get experience of practices by collecting reusable smart cases.  We have started this.  We hope others will contribute.  I think that would be smart by the industry. J

Scaling Scrum?

Scrum is excellent for lightweight management of small projects developed by co-located teams. However, it is not designed for large complex systems, or systems that build a SOA (banks, car companies) or PLA (product line architecture for instance Ericsson) or for large organizations with or without outsourcing. Moreover, Scrum is just one practice. You need many more to develop software.

But can’t we just scale Scrum? Scaling is challenging. In the 1970’s people tried to scale to ‘programming in the large’. Later people tried to scale objects to ‘objects in the large’ (mega-programming). Both never made it. Hundreds of books and even more papers were written on the subjects, but it didn’t help. Instead ideas that were scalable at their roots became successful such as components and objects. At Ericsson we never had to struggle with ‘programming in the large’ since we went straight to components and later to objects.

Why is it so difficult to scale? Explaining this is not difficult but takes a lot of space. Instead let me make an analogy. Suppose you have been able to produce a new chemical substance by using a test tube and a Bunsen burner and the new substance helps with an important problem. Now you want to produce tons of the new substance. Nobody would dream of industrializing the laboratory method by simply building a larger laboratory with hundred meter long test tubes and fifty meter high Bunsen burners. Instead a chemical process would be built in which you would not recognize the original method.

Now, why do we need to scale Scrum itself? Why not let Scrum stay the pearl it is. Treat it as one of your practices in your portfolio of practices also including for example practices for requirements to test, architecture, components. Scale by composing Scrum with other practices.

To the fathers of Scrum I would say: Please, don’t extend Scrum with all kinds of other practices such as use cases, user stories test-driven design, etc. and brand the whole soup as Scrum. Doing so will make Scrum heavier and it will threaten the survival of Scrum because of deficiencies in these other practices.

We help companies around the world to adopt Scrum. However, we don’t see Scrum as the center of all you need. Instead, the center is the practices you already have in your organization, the ones your people already have adopted. Replace them carefully with better practices – one by one. Scrum could be one of these better practices but not the only one.

Do you agree with this approach? Now, it doesn’t happen by itself. It is clear that a practice-based approach is what is needed.  We have been promoting and working with practices since 2003.  Now, even IBM and Microsoft have started to join this movement. And this is really good for the industry. It is very smart!

Traditional project management approaches focus on planning in detail, assigning the resulting tasks to people and then tracking "progress" as measured by completed tasks. The problem with measuring progress this way is that completing a task, while important, is hard to correlate with progress against the overall goal - just because you've completed 20% of the tasks does not mean that you're 20% done - and for tasks that take a long time to complete the self-reported estimates of "percent complete" is often merely "wishful thinking".

My preference is to measure progress in a concrete and measurable way - in the form of tested scenarios, following an iterative project management approach.  In other words, planning works iteration by iteration, with each iteration developing and testing one or more scenarios.  At the end of each iteration, you have a set of developed and tested scenarios, making progress easier to measure: knowing that you've developed and tested 20 out of 100 scenarios is a lot more meaningful than knowing that you've completed 20% of the tasks - especially if those tasks are focused on creating documentation rather than running and tested code.  Scenarios correlate nicely with business value - each scenario should be useful to at least some subset of the stakeholders.  In my view, only when you've successfully tested a scenario can you claim to have "earned value".

During a recent trip to China and Australia I observed that everyone wants to be agile.   It may be that Northern Europe and the USA are a bit ahead but the trend is clear all over the world.  In a round table meeting with CIO’s, I usually ask what people are particularly interested in right now.  Five years ago a common answer was we are trying to adopt the Unified Process.  Now, the same question returns the answer we are trying to move to agile.  Thus you would assume that people know what agile is.     

The last month I gave four public presentations with around 100-200 people each.  I met with about twelve companies.  At every occasion, I asked what really is new with agile.  Here are typical unfiltered answers: “rapid iterations”, “working software”, “coping with change”, “communication”, “flexible”, “adaptable”, “eliminate waste”, “accepting changes”, “small iterations”, “feature-driven”, “continued integrations”, “test driven development”, “no documentation”, “people before process”, “adapt to change”, “the name”, “the team sets their own priority”, “early stakeholder involvement”.

An absolute majority, around 60%, said that agile is about iterations (or sprints to use the Scrum terminology).  It is a bit disappointing that people don’t know that iterative development was introduced more than 25 years ago by Barry Boehm.  He called it spiral development.

It is even more disappointing to hear that people think that RUP is not iterative, but based on the waterfall model.  In fact, if you wanted to use RUP for waterfall development you would have to make a real effort to restructure RUP.  We clearly said that everyone should move to iterations for the same reasons that people now like about agile: rapid, working software, change, flexible, risks, etc.

Given that around 60% think that agile is about iterations, and RUP was designed to support iterations, is RUP agile?  My answer is that RUP can be applied in an agile way but RUP itself is not agile.  Thus there need to be something more. 

20% of the answers were about technical ideas such as feature-driven, test-driven, user stories, etc.  However, none of these ideas would have created a revolution on their own.

10% of the answers were about light process – light to understand, light to use and light on documentation.  Now we start to come to the core of agile.  I truly believe that in the past we have been too ambitious in describing process, in adopting too much process and in documentation.  The reality is that even if people write a lot, very few people will ever read it.  Thus the trend towards light will sustain.  However, it is easy to be light.  The trick is to be as light as possible but not lighter.  I believe you will find our work on EssUP and EssWork new and fresh.

The last 10% were about how to work together daily, weekly, monthly, etc.  It is about communication, people and teams, about how to organize teams, how to take decisions, how to protect the team from the outside.  This is what we call social engineering.  Agile has put the finger on the fact that we need highly motivated and competent people to be successful with software development.  No process has ever developed software.  It has always been done by people.  We have of course always known this, but we have not pushed it as much.  The focus on people is really what makes agile unique, and this is why agile originally broke through.

Now, it doesn’t really matter what people think agile is.  Agile has become more of a philosophy.  It appears that everything good is now agile.  Thus it is not really easy to tell what agile is.  However, one thing we know.  Everyone will subscribe to being agile (as they should) so one day agile will go without saying. 

Let me though make a cautious reservation.  There is an obvious danger that as it continues, agile will be discredited because the concept is sometimes used as an excuse for doing shoddy work, for having no requirements, for developing whatever the developers feel like doing.  This is not in the spirit of "true agility" but if it continues it will give agility a bad name.

Whatever happens we will one day get a new fashion.  I can’t tell you what it will be but be sure of one thing: it will be smart, very smart.

The style in which we develop software is as fashion influenced as the clothing style, maybe even more.  We don’t dress ourselves dramatic different more than every five to ten years, but when it comes to software we change our development style dramatically every five years. Just a few years ago the Unified Process was a la mode and every software organization was adopting UP one way or the other.  Just 2-3 years ago the hottest style was Extreme Programming XP, but now I hear very little about XP.  Now, everyone is running after the next “silver bullet” Scrum.

But what about Scrum, isn’t it fantastic?  When I for the first time met Ken Schwaber, a father of Scrum, I said: “What I like about Scrum is that it describes how really good project managers motivate people and run an iterative project.”  I continued: “…and its beauty comes from being agnostic to how you actually do things…you can pick your own way to do requirements, design, code and test.” I summarized: “Can I label Scrum ‘a common sense project management approach’?”  “Yes”, said Ken, “that label works”.  After this introduction we had a lot to talk about.

I can assure you that the founders of Scrum didn’t view their baby as a silver bullet, but with all the pressure from people who love fashion and want to create buzz, they may very well have to accept that they have created a silver bullet.

I think Scrum is a great project management practice.  It also includes some good social engineering work patterns which make it agile. 

However, that is all it is. 

• You still need all the other practices that a good master of software needs to excel in, such as requirements, tests, architecture, components, modeling, configurations, and tools, etc.  This is all quite basic but important stuff. 
• You also need to know how to build complex architectures such as SOA, PLA, EDA, and EA, architectures that are not just paper-ware but executable. 
• Finally, Scrum is not enough if you want to know how to scale to large distributed organizations which may have outsourced projects.

What would make sense to do?

• If you already are a Scrum fan you can say that you need Scrum++ where the Scrum stands for the tip of the iceberg and the ++ stands for the bottom of the iceberg which constitute all the other good practices that you will need.
• If you are not a Scrum fan, you could consider replacing your current project management practice with a Scrum practice.  You can still work as you did before so you don’t need to throw out the baby with the bathwater.  However, you would use Scrum for project management for smaller projects.  For larger organizations you will need more.
• In both cases you need to use modern social engineering work patterns as you need to be agile.

The ability to mix and match practices from different camps is what I and my company have been working with over the last four years, and now we can do it.  That will allow us to move forward and stay away from the silver bullets, the hype, the buzz and the fashion and treat software with the respect it deserves.

This would be smart.  Smart by the industry. 

There are a number of studies that cite poor performance of software projects - The Standish Group being the authors of one of the more often cited, their Chaos Report (and old version from 1995 is posted here, and although the data is old the conclusions are not dramatically changed).  The gist of these studies is that the majority of projects (as high as 70%) fail when measured against original schedule, budget, and expected features. I would be the last to argue the general conclusion: that it is very hard to manage a project to success. 

Most projects lack clear direction and purpose, and many are rife with disagreements about what success looks like.  There is, however, something in the assumptions behind these studies that rings hollow: that the initial schedule, budget and expectations for projects is a reliable milepost against which to measure. Most projects are vaguely conceived at best - they often lack a clear understanding of why they should exist and what problems they need to solve.  At their initiation they are usually poorly scoped and vaguely purposed, and the funding associated with them is often assigned based on an  allocation of an arbitrarily assigned budget.  Their schedules, at least those produced at the start of the project, are largely speculative endeavors, a mixture of gut and guesswork, that bears little basis in reality.  Measuring project performance against the initial schedule, scope and budget is of little value except to illustrate the point that there is a large disconnect between the expectations of business sponsors and the ability of teams to deliver against those expectations.  There are, to be sure, rampant problems with performance, but there are also widespread woes of expectations that are just as important to address.

Where should we start?  The first place is probably with project funding and measurement. The real thing of importance to measure is whether the project produced (or exceeded) the business value  expected of it.  If a delay in the project caused a market window of opportunity to be missed, that is significant, but it is the decline in value delivered that needs to be measured, not a schedule variance that cannot be correlated with economic activity.  Forcing a focus on business value produced would also put the right attention on the role of the business in following-through on their assertions of the value that will accrue from having the solution.  Requesting projects based on business needs has an opportunity cost - choosing one project over another should affect the value delivered to shareholders - and accountability for assertions by the line of business is just as important as accountability for project delivery.

If we shift our attention to value delivered rather than meeting schedule and budget, we may free the development team to find better ways to deliver the value, which may or may not include the initial set of features envisioned by the business sponsor.  Initial feature lists are usually vaguely conceived and don't provide a very good target for delivery.  Work is usually required to ascertain the real needs from this initial list of "features", some of which contribute to satisfying real needs but many of which are simply good initial starting points for discussion about real needs.  It may very well take longer than expected to solve the real problems (it usually does, as we all tend to be more optimistic than we should about how long things will take).

The problem is that most teams are set up to fail from the start.  By measuring them against budgets and schedules based on arbitrary assumptions and often a poor understanding of the real business value that needs to be produced, we find them constantly struggling against a plan that cannot possibly succeed.   Measuring against initial schedule, budget and expected features is not merely meaningless, it's actually part of the problem.  We need to shift our focus to better articulating problems to be solved and needs to be satisfied, and measuring business value produced.  Once we start to do that, we can focus on the plans and milestones needed to ensure the delivery of business value.

Management styles have a huge effect on software development teams.  A significant shift required in  the movement toward a more agile approach is a change in the way that teams are managed and measured.Management styles can be described in  a number of ways - Theory X and Theory Y are popular approaches dating back to the 1960's but still applicable today.   Basically, Theory X assumes that people are basically untrustworthy slackers who need to be constantly monitored and told what to do and are only working because they need the money.  Theory Y assumes that people want to do a good job and are motivated by more than money, and that people produce their best results when they are working in a supportive environment that frees them to be creative and productive.It should come as no surprise that a necessary condition for a movement toward agile teaming is a "Theory Y" management culture - a team trying to adopt an agile approach in an organization with a "Theory X" management culture is doomed to failure and frustration; it will constantly be fighting the management system and the surrounding culture. In these organizations, the management culture (and its supporting measurement system) must change along with the team approach in order for both to be successful. Before you can change it, however, you need to understand it.

I often get called into companies who are thinking of going "Agile".  They have invested many thousands of dollars on a very complex SDLC taking the best ideas from the industry, but that process is not being followed and their teams are effectively not working together, and now think that Agile will solve those problems.  They come to IJC with a simple question "how do I adopt agile ?".  But when you dig, you find that this question is not as easy as it would appear.  When you question their motivation and their constraints, you find a whole list of issues and problems that Agile by itself can not solve.  Issues can range from off-shoring, to quality and performance.  Issues that no one approach can solve.

Agile software development seems to be a way of describing everything that is good in software process today.  It combines techniques and team practices with ways of changing organizations.  Thus, when people talk about agile they are talking about so many different things it is really hard to get a handle on them all - it is like saying I like European food and everyone knows what that means.  Do I mean Spanish, Swedish, French...?  Of course I can not mean English.  There is a lot of great stuff that has the label Agile, but the area that I will discuss today are the team techniques that Agile has brought to the table.  In particular SCRUM. 

First let's define a structure that SCRUM fits into.  SCRUM provides a fantastic set of very simple project management processes that help teams better function, but it does not provide great guidance in the areas of actually building software or making sure that project fits into something much larger.  Thus, I always position SCRUM in the context of three other processes.  Firstly, on top is an organizational process.  These are typically described in SDLC milestones, gates or phases.  They represent the key stages a project MUST go through from a funding and control point of view.  The second view of process is the team.  How does the team function to deliver software in support of this lifecycle?  This I label as team.  The third is the techniques that an individual must employ to actually build software.  Techniques such as OOAD or Test Driven Development - Also this is where techniques such as Use Case Driven development fit. SCRUM fits nicely into the middle layer - It is a set of team techniques that really help the team become a team.  But without the top and bottom views of process being in place SCRUM by itself would not work. 

Firstly SCRUM provides a very simple set of roles.  Product Owner (the go to person who owns the problem you are solving), the SCRUM Master (the person who runs the SCRUM meetings and protects the team) and the Development Team (the people that do the work).  I also often add a fourth role that of the Technical Owner, the person who owns the system from a technical point of view and is the go to person about all things about the architecture.  Secondly it provides three meetings.  A kick off meeting at the start of a sprint.  Oh, I had better introduce the idea of a sprint - A sprint is like an iteration.  A small chunk of time when stuff happens - has a clear set of goals and delivers stuff.  In terms of what it delivers they are classically stories, but can also be features or other units of stuff.  What is important about a sprint is that real work is done, that means working software is produced, tested and deployed (maybe to a pseudo production environment).  The second type of meeting, and the one that gets all the press is the daily Scrum - this meeting is short, say 15 minutes, and focused on three questions.  How did you do, what are you doing and what is stopping you?  These three questions enable the project to move along at a rapid, focused rate.  The third meeting is all about retrospective and proving you did what you were promising in your sprint.   So, SCRUM provides a great way of organizing the team, but without the lifecycle the team can not decide on what sprints they will do and the management can not appreciate the value of this sprint in the context of something bigger.  Without some techniques for building software individuals do not know what to do. 

So SCRUM must always be part of something bigger to really make an impact.  I would therefore argue that agile on its own is like a single food item; really to enjoy it, it needs to be put into the context of a meal.