Scaling Scrum?

Scrum is excellent for lightweight management of small projects developed by co-located teams. However, it is not designed for large complex systems, or systems that build a SOA (banks, car companies) or PLA (product line architecture for instance Ericsson) or for large organizations with or without outsourcing. Moreover, Scrum is just one practice. You need many more to develop software.

But can’t we just scale Scrum? Scaling is challenging. In the 1970’s people tried to scale to ‘programming in the large’. Later people tried to scale objects to ‘objects in the large’ (mega-programming). Both never made it. Hundreds of books and even more papers were written on the subjects, but it didn’t help. Instead ideas that were scalable at their roots became successful such as components and objects. At Ericsson we never had to struggle with ‘programming in the large’ since we went straight to components and later to objects.

Why is it so difficult to scale? Explaining this is not difficult but takes a lot of space. Instead let me make an analogy. Suppose you have been able to produce a new chemical substance by using a test tube and a Bunsen burner and the new substance helps with an important problem. Now you want to produce tons of the new substance. Nobody would dream of industrializing the laboratory method by simply building a larger laboratory with hundred meter long test tubes and fifty meter high Bunsen burners. Instead a chemical process would be built in which you would not recognize the original method.

Now, why do we need to scale Scrum itself? Why not let Scrum stay the pearl it is. Treat it as one of your practices in your portfolio of practices also including for example practices for requirements to test, architecture, components. Scale by composing Scrum with other practices.

To the fathers of Scrum I would say: Please, don’t extend Scrum with all kinds of other practices such as use cases, user stories test-driven design, etc. and brand the whole soup as Scrum. Doing so will make Scrum heavier and it will threaten the survival of Scrum because of deficiencies in these other practices.

We help companies around the world to adopt Scrum. However, we don’t see Scrum as the center of all you need. Instead, the center is the practices you already have in your organization, the ones your people already have adopted. Replace them carefully with better practices – one by one. Scrum could be one of these better practices but not the only one.

Do you agree with this approach? Now, it doesn’t happen by itself. It is clear that a practice-based approach is what is needed.  We have been promoting and working with practices since 2003.  Now, even IBM and Microsoft have started to join this movement. And this is really good for the industry. It is very smart!

 I often get the question:  “RUP was your baby, but how do you look upon RUP today?”  In an interview a couple of years ago I responded jokingly: “Yes, RUP is my baby, but you know babies grow up and some of them need correction.”  RUP was created in Sweden under the name Objectory.  This was in 1987. 

Objectory was really new since its base could be used to describe all aspects of software development.  Our focus was to identify all of the rules that we use when we develop good software: what are good use cases, good components, good test cases, etc.  The literature in software engineering was totally empty in this space at the time. In order to do this we used object modeling to describe Objectory.  Now we could let the base grow forever – technically speaking.    Any company could make their own new process using Objectory.

Objectory became very successful.  It survived the merge with Rational in 1995.  The next version was named Rational Objectory Process, but after the success of UML, the name was changed to the Rational Unified Process or RUP as we all know it.  In the work of merging the two companies, the process content grew significantly.  However, hidden behind the size, our innovation of how to create a process base and the rules for goodness survived.  The crown jewels of Objectory had survived!

Thus, RUP is one of my babies.  The challenges with Objectory and consequently with RUP were serious. 

1. Adoption
   Although our way of working in creating a process base was new and great, adoption became quite expensive.  To truly be successful with the adoption and consistent usage was to involve customization, mentoring and consulting which most customers couldn’t afford.  Thus the success rate was too low.
2. Growth
   RUP was developed inside one company and all new ideas from the outside world had to be filtered by our own people and added to RUP.  It goes without saying that ideas come from anywhere in the world and by having a process be company owned makes it difficult to assimilate these great ideas.  The use of community much like what has been done with open source models would allow for continuous input and improvement.  This also allows for scrutiny. 
3. Execution 
   All processes are paper-ware and often managed by people who don’t use the process.  This quickly becomes a problem in most organizations because the process becomes a religious document and not a realistic one.  This then has the effect that what teams do and what the process tells them to do is often not in synch.  And since the team is in charge (doing the work) the value of process becomes small.

These challenges paved the way for the agile movement.  The different agile methodologists have one thing in common: they hate RUP.  But that is NOT very smart. There are lots of good ideas in RUP.  Many companies have made the effort in adopting RUP and done so quite successfully.  This has provided them with a very competent staff and something to build on for the future.

However, “the baby needs correction”, and I and my company have done so.  The adoption problem is attacked by focus on the essentials and being light instead of trying to be complete.  We use cards and game boards to describe the process.  Moreover, you don’t need to adopt a complete process all at once, just adopt a practice at the time.  The growth problem is not a problem anymore, since we downplay big process and we make instead practices first class citizens.  Practices can come from any source including the RUP and much of the practice work we have done is in support of RUP adoption. Practices can be composed to create your own way of working.  Finally, the execution problem which is the hardest to achieve, is for the time being our little secret, but you are welcome to learn more about it by studying EssWork and EssUP. 

Trust me, this is very smart.

The style in which we develop software is as fashion influenced as the clothing style, maybe even more.  We don’t dress ourselves dramatic different more than every five to ten years, but when it comes to software we change our development style dramatically every five years. Just a few years ago the Unified Process was a la mode and every software organization was adopting UP one way or the other.  Just 2-3 years ago the hottest style was Extreme Programming XP, but now I hear very little about XP.  Now, everyone is running after the next “silver bullet” Scrum.

But what about Scrum, isn’t it fantastic?  When I for the first time met Ken Schwaber, a father of Scrum, I said: “What I like about Scrum is that it describes how really good project managers motivate people and run an iterative project.”  I continued: “…and its beauty comes from being agnostic to how you actually do things…you can pick your own way to do requirements, design, code and test.” I summarized: “Can I label Scrum ‘a common sense project management approach’?”  “Yes”, said Ken, “that label works”.  After this introduction we had a lot to talk about.

I can assure you that the founders of Scrum didn’t view their baby as a silver bullet, but with all the pressure from people who love fashion and want to create buzz, they may very well have to accept that they have created a silver bullet.

I think Scrum is a great project management practice.  It also includes some good social engineering work patterns which make it agile. 

However, that is all it is. 

• You still need all the other practices that a good master of software needs to excel in, such as requirements, tests, architecture, components, modeling, configurations, and tools, etc.  This is all quite basic but important stuff. 
• You also need to know how to build complex architectures such as SOA, PLA, EDA, and EA, architectures that are not just paper-ware but executable. 
• Finally, Scrum is not enough if you want to know how to scale to large distributed organizations which may have outsourced projects.

What would make sense to do?

• If you already are a Scrum fan you can say that you need Scrum++ where the Scrum stands for the tip of the iceberg and the ++ stands for the bottom of the iceberg which constitute all the other good practices that you will need.
• If you are not a Scrum fan, you could consider replacing your current project management practice with a Scrum practice.  You can still work as you did before so you don’t need to throw out the baby with the bathwater.  However, you would use Scrum for project management for smaller projects.  For larger organizations you will need more.
• In both cases you need to use modern social engineering work patterns as you need to be agile.

The ability to mix and match practices from different camps is what I and my company have been working with over the last four years, and now we can do it.  That will allow us to move forward and stay away from the silver bullets, the hype, the buzz and the fashion and treat software with the respect it deserves.

This would be smart.  Smart by the industry. 

I often get called into companies who are thinking of going "Agile".  They have invested many thousands of dollars on a very complex SDLC taking the best ideas from the industry, but that process is not being followed and their teams are effectively not working together, and now think that Agile will solve those problems.  They come to IJC with a simple question "how do I adopt agile ?".  But when you dig, you find that this question is not as easy as it would appear.  When you question their motivation and their constraints, you find a whole list of issues and problems that Agile by itself can not solve.  Issues can range from off-shoring, to quality and performance.  Issues that no one approach can solve.

Agile software development seems to be a way of describing everything that is good in software process today.  It combines techniques and team practices with ways of changing organizations.  Thus, when people talk about agile they are talking about so many different things it is really hard to get a handle on them all - it is like saying I like European food and everyone knows what that means.  Do I mean Spanish, Swedish, French...?  Of course I can not mean English.  There is a lot of great stuff that has the label Agile, but the area that I will discuss today are the team techniques that Agile has brought to the table.  In particular SCRUM. 

First let's define a structure that SCRUM fits into.  SCRUM provides a fantastic set of very simple project management processes that help teams better function, but it does not provide great guidance in the areas of actually building software or making sure that project fits into something much larger.  Thus, I always position SCRUM in the context of three other processes.  Firstly, on top is an organizational process.  These are typically described in SDLC milestones, gates or phases.  They represent the key stages a project MUST go through from a funding and control point of view.  The second view of process is the team.  How does the team function to deliver software in support of this lifecycle?  This I label as team.  The third is the techniques that an individual must employ to actually build software.  Techniques such as OOAD or Test Driven Development - Also this is where techniques such as Use Case Driven development fit. SCRUM fits nicely into the middle layer - It is a set of team techniques that really help the team become a team.  But without the top and bottom views of process being in place SCRUM by itself would not work. 

Firstly SCRUM provides a very simple set of roles.  Product Owner (the go to person who owns the problem you are solving), the SCRUM Master (the person who runs the SCRUM meetings and protects the team) and the Development Team (the people that do the work).  I also often add a fourth role that of the Technical Owner, the person who owns the system from a technical point of view and is the go to person about all things about the architecture.  Secondly it provides three meetings.  A kick off meeting at the start of a sprint.  Oh, I had better introduce the idea of a sprint - A sprint is like an iteration.  A small chunk of time when stuff happens - has a clear set of goals and delivers stuff.  In terms of what it delivers they are classically stories, but can also be features or other units of stuff.  What is important about a sprint is that real work is done, that means working software is produced, tested and deployed (maybe to a pseudo production environment).  The second type of meeting, and the one that gets all the press is the daily Scrum - this meeting is short, say 15 minutes, and focused on three questions.  How did you do, what are you doing and what is stopping you?  These three questions enable the project to move along at a rapid, focused rate.  The third meeting is all about retrospective and proving you did what you were promising in your sprint.   So, SCRUM provides a great way of organizing the team, but without the lifecycle the team can not decide on what sprints they will do and the management can not appreciate the value of this sprint in the context of something bigger.  Without some techniques for building software individuals do not know what to do. 

So SCRUM must always be part of something bigger to really make an impact.  I would therefore argue that agile on its own is like a single food item; really to enjoy it, it needs to be put into the context of a meal.