Traditional project management approaches focus on planning in detail, assigning the resulting tasks to people and then tracking "progress" as measured by completed tasks. The problem with measuring progress this way is that completing a task, while important, is hard to correlate with progress against the overall goal - just because you've completed 20% of the tasks does not mean that you're 20% done - and for tasks that take a long time to complete the self-reported estimates of "percent complete" is often merely "wishful thinking".

My preference is to measure progress in a concrete and measurable way - in the form of tested scenarios, following an iterative project management approach.  In other words, planning works iteration by iteration, with each iteration developing and testing one or more scenarios.  At the end of each iteration, you have a set of developed and tested scenarios, making progress easier to measure: knowing that you've developed and tested 20 out of 100 scenarios is a lot more meaningful than knowing that you've completed 20% of the tasks - especially if those tasks are focused on creating documentation rather than running and tested code.  Scenarios correlate nicely with business value - each scenario should be useful to at least some subset of the stakeholders.  In my view, only when you've successfully tested a scenario can you claim to have "earned value".

 I often get the question:  “RUP was your baby, but how do you look upon RUP today?”  In an interview a couple of years ago I responded jokingly: “Yes, RUP is my baby, but you know babies grow up and some of them need correction.”  RUP was created in Sweden under the name Objectory.  This was in 1987. 

Objectory was really new since its base could be used to describe all aspects of software development.  Our focus was to identify all of the rules that we use when we develop good software: what are good use cases, good components, good test cases, etc.  The literature in software engineering was totally empty in this space at the time. In order to do this we used object modeling to describe Objectory.  Now we could let the base grow forever – technically speaking.    Any company could make their own new process using Objectory.

Objectory became very successful.  It survived the merge with Rational in 1995.  The next version was named Rational Objectory Process, but after the success of UML, the name was changed to the Rational Unified Process or RUP as we all know it.  In the work of merging the two companies, the process content grew significantly.  However, hidden behind the size, our innovation of how to create a process base and the rules for goodness survived.  The crown jewels of Objectory had survived!

Thus, RUP is one of my babies.  The challenges with Objectory and consequently with RUP were serious. 

1. Adoption
   Although our way of working in creating a process base was new and great, adoption became quite expensive.  To truly be successful with the adoption and consistent usage was to involve customization, mentoring and consulting which most customers couldn’t afford.  Thus the success rate was too low.
2. Growth
   RUP was developed inside one company and all new ideas from the outside world had to be filtered by our own people and added to RUP.  It goes without saying that ideas come from anywhere in the world and by having a process be company owned makes it difficult to assimilate these great ideas.  The use of community much like what has been done with open source models would allow for continuous input and improvement.  This also allows for scrutiny. 
3. Execution 
   All processes are paper-ware and often managed by people who don’t use the process.  This quickly becomes a problem in most organizations because the process becomes a religious document and not a realistic one.  This then has the effect that what teams do and what the process tells them to do is often not in synch.  And since the team is in charge (doing the work) the value of process becomes small.

These challenges paved the way for the agile movement.  The different agile methodologists have one thing in common: they hate RUP.  But that is NOT very smart. There are lots of good ideas in RUP.  Many companies have made the effort in adopting RUP and done so quite successfully.  This has provided them with a very competent staff and something to build on for the future.

However, “the baby needs correction”, and I and my company have done so.  The adoption problem is attacked by focus on the essentials and being light instead of trying to be complete.  We use cards and game boards to describe the process.  Moreover, you don’t need to adopt a complete process all at once, just adopt a practice at the time.  The growth problem is not a problem anymore, since we downplay big process and we make instead practices first class citizens.  Practices can come from any source including the RUP and much of the practice work we have done is in support of RUP adoption. Practices can be composed to create your own way of working.  Finally, the execution problem which is the hardest to achieve, is for the time being our little secret, but you are welcome to learn more about it by studying EssWork and EssUP. 

Trust me, this is very smart.

During a recent trip to China and Australia I observed that everyone wants to be agile.   It may be that Northern Europe and the USA are a bit ahead but the trend is clear all over the world.  In a round table meeting with CIO’s, I usually ask what people are particularly interested in right now.  Five years ago a common answer was we are trying to adopt the Unified Process.  Now, the same question returns the answer we are trying to move to agile.  Thus you would assume that people know what agile is.     

The last month I gave four public presentations with around 100-200 people each.  I met with about twelve companies.  At every occasion, I asked what really is new with agile.  Here are typical unfiltered answers: “rapid iterations”, “working software”, “coping with change”, “communication”, “flexible”, “adaptable”, “eliminate waste”, “accepting changes”, “small iterations”, “feature-driven”, “continued integrations”, “test driven development”, “no documentation”, “people before process”, “adapt to change”, “the name”, “the team sets their own priority”, “early stakeholder involvement”.

An absolute majority, around 60%, said that agile is about iterations (or sprints to use the Scrum terminology).  It is a bit disappointing that people don’t know that iterative development was introduced more than 25 years ago by Barry Boehm.  He called it spiral development.

It is even more disappointing to hear that people think that RUP is not iterative, but based on the waterfall model.  In fact, if you wanted to use RUP for waterfall development you would have to make a real effort to restructure RUP.  We clearly said that everyone should move to iterations for the same reasons that people now like about agile: rapid, working software, change, flexible, risks, etc.

Given that around 60% think that agile is about iterations, and RUP was designed to support iterations, is RUP agile?  My answer is that RUP can be applied in an agile way but RUP itself is not agile.  Thus there need to be something more. 

20% of the answers were about technical ideas such as feature-driven, test-driven, user stories, etc.  However, none of these ideas would have created a revolution on their own.

10% of the answers were about light process – light to understand, light to use and light on documentation.  Now we start to come to the core of agile.  I truly believe that in the past we have been too ambitious in describing process, in adopting too much process and in documentation.  The reality is that even if people write a lot, very few people will ever read it.  Thus the trend towards light will sustain.  However, it is easy to be light.  The trick is to be as light as possible but not lighter.  I believe you will find our work on EssUP and EssWork new and fresh.

The last 10% were about how to work together daily, weekly, monthly, etc.  It is about communication, people and teams, about how to organize teams, how to take decisions, how to protect the team from the outside.  This is what we call social engineering.  Agile has put the finger on the fact that we need highly motivated and competent people to be successful with software development.  No process has ever developed software.  It has always been done by people.  We have of course always known this, but we have not pushed it as much.  The focus on people is really what makes agile unique, and this is why agile originally broke through.

Now, it doesn’t really matter what people think agile is.  Agile has become more of a philosophy.  It appears that everything good is now agile.  Thus it is not really easy to tell what agile is.  However, one thing we know.  Everyone will subscribe to being agile (as they should) so one day agile will go without saying. 

Let me though make a cautious reservation.  There is an obvious danger that as it continues, agile will be discredited because the concept is sometimes used as an excuse for doing shoddy work, for having no requirements, for developing whatever the developers feel like doing.  This is not in the spirit of "true agility" but if it continues it will give agility a bad name.

Whatever happens we will one day get a new fashion.  I can’t tell you what it will be but be sure of one thing: it will be smart, very smart.

The style in which we develop software is as fashion influenced as the clothing style, maybe even more.  We don’t dress ourselves dramatic different more than every five to ten years, but when it comes to software we change our development style dramatically every five years. Just a few years ago the Unified Process was a la mode and every software organization was adopting UP one way or the other.  Just 2-3 years ago the hottest style was Extreme Programming XP, but now I hear very little about XP.  Now, everyone is running after the next “silver bullet” Scrum.

But what about Scrum, isn’t it fantastic?  When I for the first time met Ken Schwaber, a father of Scrum, I said: “What I like about Scrum is that it describes how really good project managers motivate people and run an iterative project.”  I continued: “…and its beauty comes from being agnostic to how you actually do things…you can pick your own way to do requirements, design, code and test.” I summarized: “Can I label Scrum ‘a common sense project management approach’?”  “Yes”, said Ken, “that label works”.  After this introduction we had a lot to talk about.

I can assure you that the founders of Scrum didn’t view their baby as a silver bullet, but with all the pressure from people who love fashion and want to create buzz, they may very well have to accept that they have created a silver bullet.

I think Scrum is a great project management practice.  It also includes some good social engineering work patterns which make it agile. 

However, that is all it is. 

• You still need all the other practices that a good master of software needs to excel in, such as requirements, tests, architecture, components, modeling, configurations, and tools, etc.  This is all quite basic but important stuff. 
• You also need to know how to build complex architectures such as SOA, PLA, EDA, and EA, architectures that are not just paper-ware but executable. 
• Finally, Scrum is not enough if you want to know how to scale to large distributed organizations which may have outsourced projects.

What would make sense to do?

• If you already are a Scrum fan you can say that you need Scrum++ where the Scrum stands for the tip of the iceberg and the ++ stands for the bottom of the iceberg which constitute all the other good practices that you will need.
• If you are not a Scrum fan, you could consider replacing your current project management practice with a Scrum practice.  You can still work as you did before so you don’t need to throw out the baby with the bathwater.  However, you would use Scrum for project management for smaller projects.  For larger organizations you will need more.
• In both cases you need to use modern social engineering work patterns as you need to be agile.

The ability to mix and match practices from different camps is what I and my company have been working with over the last four years, and now we can do it.  That will allow us to move forward and stay away from the silver bullets, the hype, the buzz and the fashion and treat software with the respect it deserves.

This would be smart.  Smart by the industry. 

I am often asked the question: “will MDD ever be successful and will the tools ever really be put in place to support it?” and I was recently asked this again, so here are my thoughts.  

The following is solely my opinion and can be argued or agreed with, but it comes from 15+ years of experience building application and data modeles, modeling tools including ERwin, Rational Rose and other tools, writing 2 books on UML and working directly with clients who are modeling. 

Model Driven Development in concept is great, but to date the tools and the people who would have to use them have not been able to keep up with the concepts.  There are some very good tools on the market like Telelogic Tau, Rational RoseRT and a few others which do a fairly complete job of generating the needed code, but this is usually focused in the "systems" space and has not translated well to IT as it is based on state's and generating the logic from those states, etc.   

On the IT side, we do have similar concepts, but they start from business process models using tools like WebSphere Business Modeler and similar tools from BEA for example which connect to Business Process runtime engines and generate the connection points of existing applications to automate communication and business process execution.   

This all said, the uptake MDD has not been that of other areas for what I believe are 3 reasons: 

1.      Developers are developers because they want to write code and don't see models as being complete enough nor for their role to build models beyond simple architecture structures.

2.      Most projects today are starting with something already in production and therefore doing modeling of the architectures, use cases and processes are quite good to understand what needs to be done, how it connects together and prioritize work, but it makes it difficult to generate a system that is only a piece to connect up or an update.

3.      I believe #3 can stand on its own, but also lends itself to the first 2 comments and that is the creation of a "Black Box".  Using MDD tools creates a black box effect where runtime generation, transformations and other constructs are managed by a proprietary engine which is difficult to alter and manage. 

a.      For Developers, they often think they can do it better and faster and don’t want to rely on something they cannot touch. 

b.      Because of the black box approach, it is often requires a rewrite of the engines that have already been put into place for the existing systems causing added cost that nobody is willing to fund. 

We have tried in the past similar types of technologies which few have been successful as well.  Runtime Data Access is a great example where tools and companies popped up in the late 90's which created object-to-data mapping and automated the creation of the runtime access at what they claimed to be much faster and cheaper than doing it yourself.  Yes, this was good at least in theory, but few bought into it.  Why?  Black box approach, hard to test, developers think they can write faster algorithms, management doesn't trust something they cannot see, etc.  This is very similar to MDD and its lack of success in my opinion.   

That all said, I do have some collegues who are using MDD on some very interesting projects building components for airplanes for example which they feel are quite successful, but these also seem to be few and far between.

A while back I was giving a presentation on requirements errors and someone made the observation that they thought the term "requirement" was, in itself, misleading. The  gist of his argument is that the very term implies something that is "required", that it has to be done.  In reality, most "requirements", at least as initially stated, are somewhat vague statements of things that could be done, but they often need quite a bit of work to determine what is really needed.  It would be better to call them, at least initially, "wishes".

Unfortunately we are probably stuck with the term "requirement", but it can help to realize that as initially identified "requirements" tend to be vague, filled with incorrect assumptions about what the real problems and solutions look like.  Some organizations call these sort of things "stakeholder requests", which gives them a way to capture them without anyone inadvertently assuming that these are the actual requirements.  They then go through an investigation and refinement process before discovering the actual "requirements".  Along the way, various other kinds of things may be discovered: needs, goals or objectives, desired outcomes, as well as terms (such as would be found in a glossary), among other things.

The important thing is not how you categorize (this can get out of hand as well), but that you recognize that even when someone tells you something is a requirement you should not just accept their statement at face value - you need to make sure you understand the real need, and especially what they need to achieve. Only by doing so will you know whether the thing they are asking for is really "required".

A software system with an Event-Driven-Architecture (EDA) is built around the idea that events are the most significant elements in the system and that they are produced somewhere in the system and consumed somewhere else in the system.

There are a number of studies that cite poor performance of software projects - The Standish Group being the authors of one of the more often cited, their Chaos Report (and old version from 1995 is posted here, and although the data is old the conclusions are not dramatically changed).  The gist of these studies is that the majority of projects (as high as 70%) fail when measured against original schedule, budget, and expected features. I would be the last to argue the general conclusion: that it is very hard to manage a project to success. 

Most projects lack clear direction and purpose, and many are rife with disagreements about what success looks like.  There is, however, something in the assumptions behind these studies that rings hollow: that the initial schedule, budget and expectations for projects is a reliable milepost against which to measure. Most projects are vaguely conceived at best - they often lack a clear understanding of why they should exist and what problems they need to solve.  At their initiation they are usually poorly scoped and vaguely purposed, and the funding associated with them is often assigned based on an  allocation of an arbitrarily assigned budget.  Their schedules, at least those produced at the start of the project, are largely speculative endeavors, a mixture of gut and guesswork, that bears little basis in reality.  Measuring project performance against the initial schedule, scope and budget is of little value except to illustrate the point that there is a large disconnect between the expectations of business sponsors and the ability of teams to deliver against those expectations.  There are, to be sure, rampant problems with performance, but there are also widespread woes of expectations that are just as important to address.

Where should we start?  The first place is probably with project funding and measurement. The real thing of importance to measure is whether the project produced (or exceeded) the business value  expected of it.  If a delay in the project caused a market window of opportunity to be missed, that is significant, but it is the decline in value delivered that needs to be measured, not a schedule variance that cannot be correlated with economic activity.  Forcing a focus on business value produced would also put the right attention on the role of the business in following-through on their assertions of the value that will accrue from having the solution.  Requesting projects based on business needs has an opportunity cost - choosing one project over another should affect the value delivered to shareholders - and accountability for assertions by the line of business is just as important as accountability for project delivery.

If we shift our attention to value delivered rather than meeting schedule and budget, we may free the development team to find better ways to deliver the value, which may or may not include the initial set of features envisioned by the business sponsor.  Initial feature lists are usually vaguely conceived and don't provide a very good target for delivery.  Work is usually required to ascertain the real needs from this initial list of "features", some of which contribute to satisfying real needs but many of which are simply good initial starting points for discussion about real needs.  It may very well take longer than expected to solve the real problems (it usually does, as we all tend to be more optimistic than we should about how long things will take).

The problem is that most teams are set up to fail from the start.  By measuring them against budgets and schedules based on arbitrary assumptions and often a poor understanding of the real business value that needs to be produced, we find them constantly struggling against a plan that cannot possibly succeed.   Measuring against initial schedule, budget and expected features is not merely meaningless, it's actually part of the problem.  We need to shift our focus to better articulating problems to be solved and needs to be satisfied, and measuring business value produced.  Once we start to do that, we can focus on the plans and milestones needed to ensure the delivery of business value.

Management styles have a huge effect on software development teams.  A significant shift required in  the movement toward a more agile approach is a change in the way that teams are managed and measured.Management styles can be described in  a number of ways - Theory X and Theory Y are popular approaches dating back to the 1960's but still applicable today.   Basically, Theory X assumes that people are basically untrustworthy slackers who need to be constantly monitored and told what to do and are only working because they need the money.  Theory Y assumes that people want to do a good job and are motivated by more than money, and that people produce their best results when they are working in a supportive environment that frees them to be creative and productive.It should come as no surprise that a necessary condition for a movement toward agile teaming is a "Theory Y" management culture - a team trying to adopt an agile approach in an organization with a "Theory X" management culture is doomed to failure and frustration; it will constantly be fighting the management system and the surrounding culture. In these organizations, the management culture (and its supporting measurement system) must change along with the team approach in order for both to be successful. Before you can change it, however, you need to understand it.

I often get called into companies who are thinking of going "Agile".  They have invested many thousands of dollars on a very complex SDLC taking the best ideas from the industry, but that process is not being followed and their teams are effectively not working together, and now think that Agile will solve those problems.  They come to IJC with a simple question "how do I adopt agile ?".  But when you dig, you find that this question is not as easy as it would appear.  When you question their motivation and their constraints, you find a whole list of issues and problems that Agile by itself can not solve.  Issues can range from off-shoring, to quality and performance.  Issues that no one approach can solve.

Agile software development seems to be a way of describing everything that is good in software process today.  It combines techniques and team practices with ways of changing organizations.  Thus, when people talk about agile they are talking about so many different things it is really hard to get a handle on them all - it is like saying I like European food and everyone knows what that means.  Do I mean Spanish, Swedish, French...?  Of course I can not mean English.  There is a lot of great stuff that has the label Agile, but the area that I will discuss today are the team techniques that Agile has brought to the table.  In particular SCRUM. 

First let's define a structure that SCRUM fits into.  SCRUM provides a fantastic set of very simple project management processes that help teams better function, but it does not provide great guidance in the areas of actually building software or making sure that project fits into something much larger.  Thus, I always position SCRUM in the context of three other processes.  Firstly, on top is an organizational process.  These are typically described in SDLC milestones, gates or phases.  They represent the key stages a project MUST go through from a funding and control point of view.  The second view of process is the team.  How does the team function to deliver software in support of this lifecycle?  This I label as team.  The third is the techniques that an individual must employ to actually build software.  Techniques such as OOAD or Test Driven Development - Also this is where techniques such as Use Case Driven development fit. SCRUM fits nicely into the middle layer - It is a set of team techniques that really help the team become a team.  But without the top and bottom views of process being in place SCRUM by itself would not work. 

Firstly SCRUM provides a very simple set of roles.  Product Owner (the go to person who owns the problem you are solving), the SCRUM Master (the person who runs the SCRUM meetings and protects the team) and the Development Team (the people that do the work).  I also often add a fourth role that of the Technical Owner, the person who owns the system from a technical point of view and is the go to person about all things about the architecture.  Secondly it provides three meetings.  A kick off meeting at the start of a sprint.  Oh, I had better introduce the idea of a sprint - A sprint is like an iteration.  A small chunk of time when stuff happens - has a clear set of goals and delivers stuff.  In terms of what it delivers they are classically stories, but can also be features or other units of stuff.  What is important about a sprint is that real work is done, that means working software is produced, tested and deployed (maybe to a pseudo production environment).  The second type of meeting, and the one that gets all the press is the daily Scrum - this meeting is short, say 15 minutes, and focused on three questions.  How did you do, what are you doing and what is stopping you?  These three questions enable the project to move along at a rapid, focused rate.  The third meeting is all about retrospective and proving you did what you were promising in your sprint.   So, SCRUM provides a great way of organizing the team, but without the lifecycle the team can not decide on what sprints they will do and the management can not appreciate the value of this sprint in the context of something bigger.  Without some techniques for building software individuals do not know what to do. 

So SCRUM must always be part of something bigger to really make an impact.  I would therefore argue that agile on its own is like a single food item; really to enjoy it, it needs to be put into the context of a meal.