The problem with templates

July 8th, 2008

Over the years I've come to regard 'templates' as one of the great evils of software process efforts. I was actually leading the Rational Unified Process team about 9 years ago when we made a big push to create templates for all the artifacts.  At the time I thought it was a good idea since a lot of people had been asking for them. I now regard the decision as having done a lot of damage.

 That statement is likely to shock some - surely I can't be arguing against having standard formats for organizing project information!  Of course I am not arguing against this - but I am against the way that templates are used in many organizations.  

In these organizations, people have little idea of the purpose of many artifacts or work products - they don't understand what they are doing or why, they just know (or believe) that they need to produce these work products.  The templates have been provided as a cheap way of rolling out the process - rather than actually building skills the templates are provided and people are instructed to simply fill out the templates to follow the process.

This approach is wholly ineffective. Organizations fail their people when they require artifacts without explaining why they are needed, or helping them to understand if they are needed, or helping them to improve their skills doing the real work that the artifacts merely document.

The usual result of providing standard templates is that people often blindly fill out the template sections, not knowing what to put in them, feeling compelled to fill out every section because it's there.  Mindlessly filling out templates is a waste of time - the results are valueless, albeit standard, and do not contribute to positive project results.  

I am not arguing for eliminating templates, however - there is some value in having a standard way of formatting and organizing information, but there needs to be some real information in the results produced.  The real problem is lack of knowledge and skills needed to do the work - this is where effort should be spent, not in standardizing templates.  Once people have expertise in doing the work, a consensus can be reached in the organization about how to document the work.  "Templates" will emerge out of that.

Outsourcing made wrong – a real case

June 10th, 2008

A few days ago I wrote about how outsourcing goes wrong.  Now I will explain more in detail a real case.

A couple of weeks ago I was in the US and visited a state government.  They had outsourced the development of their new Enterprise Software to a US based international software house here called Smarty for the nice sum of several hundred million dollars.

  1.  Smarty would develop the enterprise software in three years, then install it, and maintain it for 18 months.  Smarty is responsible for process, architecture, language, tools, and whatever they do will be taken over by the State.
  2. Fixed price from the beginning, but changes after signed off specification would be extra.
  3. Smarty would deliver a specification during the project and agree with the State on this before moving on.
  4. Deliveries were phase based: requirement specification, higher level design, lower level design, code and unit test.
  5.  All difficult tests (integration test, performance test, stress test, system test) would be done by the State.

This is a dream contract for Smarty.  It just cannot go wrong.  The State is locked in early to a fixed price. 

The State has agreed to review and subsequently sign off the requirements once they are thrown over the wall from Smarty.  This sounds OK, but it is horribly wrong.  It is beyond human ability to sign off the requirements for a single product even, yet alone totally impossible for an enterprise system.  No one knows the detailed requirements or can fully understand just papers.  The business people, the users and all other stakeholders have some more or less clear ideas of what they want to get done, but we all know that their detailed requirements always change as soon as they get something to use.  Moreover, one of the most important requirements for the client is to get a system that can be maintained and extended for as long as the business will stay…and that must be thought of as forever.  However, Smarty is protected.

To deliver a specification to be signed off, to deliver drawings, and code is the simple part of the project, since you just deliver paper-ware.  You really don’t need to be delayed in delivering paper-ware, you just write a little less <grin>.  And, being a bit ironic, the good thing from Smarty’s point of view is that paper-ware cannot be validated or tested, so nothing can go wrong.  Thus everything will go very well until integration test and acceptance test.  However, in our case this was moved to the State.  Here hell will break loose.

First, of course, it will be very hard to get the software to work as specified and this is where projects are delayed.  However, most important, it will be extremely hard if not impossible to validate that the software does what the business and its users want it to do. With this approach, the State won’t know what s/he will get until after the whole huge project is at its expected end.  However, Smarty is protected: “we did what we said we would do”.  Moreover, the requirements on maintainability and extensibility will not be tested, because they can’t be tested.  The State won’t know until the software is taken over and it needs to be improved, but then Smarty is long since gone.

The case I have just described is not unique.  Many companies which have moved into outsourcing feel cheated.  They have trusted that the vendor knows how to make its customers successful, and the vendor believes they can achieve that. Once the deal is underway, there is not much more to do.  The client has restructured the company to have hundreds or more people working for her/him in the way the vendor has specified.  The client has made a bad deal and doesn’t want to make this public.  And, since everyone is doing the same thing, the client is in good company and can say:  We just need to make it better next time.

However, this would never have had to happen if the deal had been structured smart.

Outsourcing made wrong

June 10th, 2008

Outsourcing is hot!  Every major corporation around the globe is outsourcing all or part of their software development.  Examples come from all industries: banking, insurance, utilities and government.  They outsource to the large software houses in for instance India or in the US and UK which of course have offshore development in India. And unfortunately the result is lots of unsuccessful projects.  However, the failures are often never talked about because the vendor has got a fantastic contract and the customer won’t speak up about poor contracts that they reviewed and approved.  This is wonderful business for the software houses.  This reminds me of the saying: “My grandfather was a horse trader, my father was a used car salesman, I am myself in the software business.”

Why do these types of projects fail? There are many reasons, but they can all be summarized by saying that they were not smart.  The buyer doesn’t know what he is buying and the vendor doesn’t know how to make it work.  If you were as cautious when buying an enterprise system as you were when buying your own house, you would be much better off.  Would you buy a house from someone by just saying “give me what I need”, and then pay them a fixed price?  No, of course not, but this or similar is what many outsourcing deals are about.  After the agreement the buyer just waits to see what he will get say three years down the road. On that path he will have to sign off on requirements that look good and complete, but which due to the ever-changing nature of software requirements are only correct to say 60%.  Following that path you will get drawings that you don’t know if they ever can be built.  At the end of that path you will have to take over the software without knowing for sure that it works for the people who are supposed to live with it.  This is what people do.  The vendors are protected because you agreed originally to sign off after each phase and you did so because you believed it would work.  It doesn’t work for software.  For enterprise systems it is a proven path to failure. 

The vendor wants happy customers, but s/he just don’t know how to do these kinds of huge projects successfully.  Or maybe the deal on the table is irresistible, and since, most often the vendor is more competent than the buyer and they won’t put themselves in a position to lose anything.

What I say here is controversial. However, someone has to say it.  Now, it should also be noted that there are companies that are very successful in outsourcing.  Their approach to outsourcing is very different based on an understanding on the nature of software, on the importance of starting small but scaling up as quickly as possible (but not quicker <grin>).  I will in another column talk about succeeding with outsourcing.  Some of the things I will talk about are:

  1.  Quickly build an architectural roadmap.
  2.  Separate work that can be given a fixed price from work that cannot reasonable be calculated upfront.
  3.  Split the big contract into many smaller contracts.
  4.  Make room for competition by specifying standards and tools.
  5. Deliver early, get feedback early.

Simply you will have to be very smart.

"Earning" earned value

May 14th, 2008

Traditional project management approaches focus on planning in detail, assigning the resulting tasks to people and then tracking "progress" as measured by completed tasks. The problem with measuring progress this way is that completing a task, while important, is hard to correlate with progress against the overall goal - just because you've completed 20% of the tasks does not mean that you're 20% done - and for tasks that take a long time to complete the self-reported estimates of "percent complete" is often merely "wishful thinking".

My preference is to measure progress in a concrete and measurable way - in the form of tested scenarios, following an iterative project management approach.  In other words, planning works iteration by iteration, with each iteration developing and testing one or more scenarios.  At the end of each iteration, you have a set of developed and tested scenarios, making progress easier to measure: knowing that you've developed and tested 20 out of 100 scenarios is a lot more meaningful than knowing that you've completed 20% of the tasks - especially if those tasks are focused on creating documentation rather than running and tested code.  Scenarios correlate nicely with business value - each scenario should be useful to at least some subset of the stakeholders.  In my view, only when you've successfully tested a scenario can you claim to have "earned value".

Yes, RUP is my baby

May 10th, 2008

 I often get the question:  “RUP was your baby, but how do you look upon RUP today?”  In an interview a couple of years ago I responded jokingly: “Yes, RUP is my baby, but you know babies grow up and some of them need correction.”  RUP was created in Sweden under the name Objectory.  This was in 1987. 


Objectory was really new since its base could be used to describe all aspects of software development.  Our focus was to identify all of the rules that we use when we develop good software: what are good use cases, good components, good test cases, etc.  The literature in software engineering was totally empty in this space at the time. In order to do this we used object modeling to describe Objectory.  Now we could let the base grow forever – technically speaking.    Any company could make their own new process using Objectory.


Objectory became very successful.  It survived the merge with Rational in 1995.  The next version was named Rational Objectory Process, but after the success of UML, the name was changed to the Rational Unified Process or RUP as we all know it.  In the work of merging the two companies, the process content grew significantly.  However, hidden behind the size, our innovation of how to create a process base and the rules for goodness survived.  The crown jewels of Objectory had survived!


Thus, RUP is one of my babies.  The challenges with Objectory and consequently with RUP were serious. 


  1. Adoption
    Although our way of working in creating a process base was new and great, adoption became quite expensive.  To truly be successful with the adoption and consistent usage was to involve customization, mentoring and consulting which most customers couldn’t afford.  Thus the success rate was too low.
  2. Growth
    RUP was developed inside one company and all new ideas from the outside world had to be filtered by our own people and added to RUP.  It goes without saying that ideas come from anywhere in the world and by having a process be company owned makes it difficult to assimilate these great ideas.  The use of community much like what has been done with open source models would allow for continuous input and improvement.  This also allows for scrutiny. 
  3. Execution 
    All processes are paper-ware and often managed by people who don’t use the process.  This quickly becomes a problem in most organizations because the process becomes a religious document and not a realistic one.  This then has the effect that what teams do and what the process tells them to do is often not in synch.  And since the team is in charge (doing the work) the value of process becomes small.


These challenges paved the way for the agile movement.  The different agile methodologists have one thing in common: they hate RUP.  But that is NOT very smart. There are lots of good ideas in RUP.  Many companies have made the effort in adopting RUP and done so quite successfully.  This has provided them with a very competent staff and something to build on for the future.


However, “the baby needs correction”, and I and my company have done so.  The adoption problem is attacked by focus on the essentials and being light instead of trying to be complete.  We use cards and game boards to describe the process.  Moreover, you don’t need to adopt a complete process all at once, just adopt a practice at the time.  The growth problem is not a problem anymore, since we downplay big process and we make instead practices first class citizens.  Practices can come from any source including the RUP and much of the practice work we have done is in support of RUP adoption. Practices can be composed to create your own way of working.  Finally, the execution problem which is the hardest to achieve, is for the time being our little secret, but you are welcome to learn more about it by studying EssWork and EssUP. 


Trust me, this is very smart.

Everyone wants to be Agile

April 30th, 2008

During a recent trip to China and Australia I observed that everyone wants to be agile.   It may be that Northern Europe and the USA are a bit ahead but the trend is clear all over the world.  In a round table meeting with CIO’s, I usually ask what people are particularly interested in right now.  Five years ago a common answer was we are trying to adopt the Unified Process.  Now, the same question returns the answer we are trying to move to agile.  Thus you would assume that people know what agile is.     


The last month I gave four public presentations with around 100-200 people each.  I met with about twelve companies.  At every occasion, I asked what really is new with agile.  Here are typical unfiltered answers: “rapid iterations”, “working software”, “coping with change”, “communication”, “flexible”, “adaptable”, “eliminate waste”, “accepting changes”, “small iterations”, “feature-driven”, “continued integrations”, “test driven development”, “no documentation”, “people before process”, “adapt to change”, “the name”, “the team sets their own priority”, “early stakeholder involvement”.


An absolute majority, around 60%, said that agile is about iterations (or sprints to use the Scrum terminology).  It is a bit disappointing that people don’t know that iterative development was introduced more than 25 years ago by Barry Boehm.  He called it spiral development.

It is even more disappointing to hear that people think that RUP is not iterative, but based on the waterfall model.  In fact, if you wanted to use RUP for waterfall development you would have to make a real effort to restructure RUP.  We clearly said that everyone should move to iterations for the same reasons that people now like about agile: rapid, working software, change, flexible, risks, etc.


Given that around 60% think that agile is about iterations, and RUP was designed to support iterations, is RUP agile?  My answer is that RUP can be applied in an agile way but RUP itself is not agile.  Thus there need to be something more. 


20% of the answers were about technical ideas such as feature-driven, test-driven, user stories, etc.  However, none of these ideas would have created a revolution on their own.


10% of the answers were about light process – light to understand, light to use and light on documentation.  Now we start to come to the core of agile.  I truly believe that in the past we have been too ambitious in describing process, in adopting too much process and in documentation.  The reality is that even if people write a lot, very few people will ever read it.  Thus the trend towards light will sustain.  However, it is easy to be light.  The trick is to be as light as possible but not lighter.  I believe you will find our work on EssUP and EssWork new and fresh.


The last 10% were about how to work together daily, weekly, monthly, etc.  It is about communication, people and teams, about how to organize teams, how to take decisions, how to protect the team from the outside.  This is what we call social engineering.  Agile has put the finger on the fact that we need highly motivated and competent people to be successful with software development.  No process has ever developed software.  It has always been done by people.  We have of course always known this, but we have not pushed it as much.  The focus on people is really what makes agile unique, and this is why agile originally broke through.


Now, it doesn’t really matter what people think agile is.  Agile has become more of a philosophy.  It appears that everything good is now agile.  Thus it is not really easy to tell what agile is.  However, one thing we know.  Everyone will subscribe to being agile (as they should) so one day agile will go without saying. 


Let me though make a cautious reservation.  There is an obvious danger that as it continues, agile will be discredited because the concept is sometimes used as an excuse for doing shoddy work, for having no requirements, for developing whatever the developers feel like doing.  This is not in the spirit of "true agility" but if it continues it will give agility a bad name.


Whatever happens we will one day get a new fashion.  I can’t tell you what it will be but be sure of one thing: it will be smart, very smart.

Software development a la mode

April 5th, 2008

The style in which we develop software is as fashion influenced as the clothing style, maybe even more.  We don’t dress ourselves dramatic different more than every five to ten years, but when it comes to software we change our development style dramatically every five years. Just a few years ago the Unified Process was a la mode and every software organization was adopting UP one way or the other.  Just 2-3 years ago the hottest style was Extreme Programming XP, but now I hear very little about XP.  Now, everyone is running after the next “silver bullet” Scrum.


But what about Scrum, isn’t it fantastic?  When I for the first time met Ken Schwaber, a father of Scrum, I said: “What I like about Scrum is that it describes how really good project managers motivate people and run an iterative project.”  I continued: “…and its beauty comes from being agnostic to how you actually do things…you can pick your own way to do requirements, design, code and test.” I summarized: “Can I label Scrum ‘a common sense project management approach’?”  “Yes”, said Ken, “that label works”.  After this introduction we had a lot to talk about.


I can assure you that the founders of Scrum didn’t view their baby as a silver bullet, but with all the pressure from people who love fashion and want to create buzz, they may very well have to accept that they have created a silver bullet.


I think Scrum is a great project management practice.  It also includes some good social engineering work patterns which make it agile. 


However, that is all it is. 

  • You still need all the other practices that a good master of software needs to excel in, such as requirements, tests, architecture, components, modeling, configurations, and tools, etc.  This is all quite basic but important stuff. 
  • You also need to know how to build complex architectures such as SOA, PLA, EDA, and EA, architectures that are not just paper-ware but executable. 
  • Finally, Scrum is not enough if you want to know how to scale to large distributed organizations which may have outsourced projects.


What would make sense to do?

  • If you already are a Scrum fan you can say that you need Scrum++ where the Scrum stands for the tip of the iceberg and the ++ stands for the bottom of the iceberg which constitute all the other good practices that you will need.
  • If you are not a Scrum fan, you could consider replacing your current project management practice with a Scrum practice.  You can still work as you did before so you don’t need to throw out the baby with the bathwater.  However, you would use Scrum for project management for smaller projects.  For larger organizations you will need more.
  • In both cases you need to use modern social engineering work patterns as you need to be agile.


The ability to mix and match practices from different camps is what I and my company have been working with over the last four years, and now we can do it.  That will allow us to move forward and stay away from the silver bullets, the hype, the buzz and the fashion and treat software with the respect it deserves.


This would be smart.  Smart by the industry. 

Will MDD Ever Come to Fruition?

March 30th, 2008

I am often asked the question: “will MDD ever be successful and will the tools ever really be put in place to support it?” and I was recently asked this again, so here are my thoughts.  

The following is solely my opinion and can be argued or agreed with, but it comes from 15+ years of experience building application and data modeles, modeling tools including ERwin, Rational Rose and other tools, writing 2 books on UML and working directly with clients who are modeling. 

Model Driven Development in concept is great, but to date the tools and the people who would have to use them have not been able to keep up with the concepts.  There are some very good tools on the market like Telelogic Tau, Rational RoseRT and a few others which do a fairly complete job of generating the needed code, but this is usually focused in the "systems" space and has not translated well to IT as it is based on state's and generating the logic from those states, etc.   

On the IT side, we do have similar concepts, but they start from business process models using tools like WebSphere Business Modeler and similar tools from BEA for example which connect to Business Process runtime engines and generate the connection points of existing applications to automate communication and business process execution.   

This all said, the uptake MDD has not been that of other areas for what I believe are 3 reasons: 

1.      Developers are developers because they want to write code and don't see models as being complete enough nor for their role to build models beyond simple architecture structures.

2.      Most projects today are starting with something already in production and therefore doing modeling of the architectures, use cases and processes are quite good to understand what needs to be done, how it connects together and prioritize work, but it makes it difficult to generate a system that is only a piece to connect up or an update.

3.      I believe #3 can stand on its own, but also lends itself to the first 2 comments and that is the creation of a "Black Box".  Using MDD tools creates a black box effect where runtime generation, transformations and other constructs are managed by a proprietary engine which is difficult to alter and manage. 

a.      For Developers, they often think they can do it better and faster and don’t want to rely on something they cannot touch. 

b.      Because of the black box approach, it is often requires a rewrite of the engines that have already been put into place for the existing systems causing added cost that nobody is willing to fund. 

We have tried in the past similar types of technologies which few have been successful as well.  Runtime Data Access is a great example where tools and companies popped up in the late 90's which created object-to-data mapping and automated the creation of the runtime access at what they claimed to be much faster and cheaper than doing it yourself.  Yes, this was good at least in theory, but few bought into it.  Why?  Black box approach, hard to test, developers think they can write faster algorithms, management doesn't trust something they cannot see, etc.  This is very similar to MDD and its lack of success in my opinion.   

That all said, I do have some collegues who are using MDD on some very interesting projects building components for airplanes for example which they feel are quite successful, but these also seem to be few and far between.

Giving requirements a bad name

March 29th, 2008

A while back I was giving a presentation on requirements errors and someone made the observation that they thought the term "requirement" was, in itself, misleading. The  gist of his argument is that the very term implies something that is "required", that it has to be done.  In reality, most "requirements", at least as initially stated, are somewhat vague statements of things that could be done, but they often need quite a bit of work to determine what is really needed.  It would be better to call them, at least initially, "wishes".


Unfortunately we are probably stuck with the term "requirement", but it can help to realize that as initially identified "requirements" tend to be vague, filled with incorrect assumptions about what the real problems and solutions look like.  Some organizations call these sort of things "stakeholder requests", which gives them a way to capture them without anyone inadvertently assuming that these are the actual requirements.  They then go through an investigation and refinement process before discovering the actual "requirements".  Along the way, various other kinds of things may be discovered: needs, goals or objectives, desired outcomes, as well as terms (such as would be found in a glossary), among other things.


The important thing is not how you categorize (this can get out of hand as well), but that you recognize that even when someone tells you something is a requirement you should not just accept their statement at face value - you need to make sure you understand the real need, and especially what they need to achieve. Only by doing so will you know whether the thing they are asking for is really "required".

Do we need Event-Driven Architecture?

March 7th, 2008

A software system with an Event-Driven-Architecture (EDA) is built around the idea that events are the most significant elements in the system and that they are produced somewhere in the system and consumed somewhere else in the system.

The business value is that you can easily extend such a system with new things that are ready to produce or consume events that already are in place in the system.  Of course you can add new events as you go. 

Yes, this is absolutely great.  If you build something new there is no reason why you shouldn’t use this kind of architecture.  However, focusing on the events is not the only thing you should do.   

Instead, you should just build an architecture in which you have components or services and some kinds of “channels” between some of these components.  Over a channel an event can flow from one component – the producer – to another one – the consumer.  These components are loosely coupled and can exist in a distributed world.  Some of these events are such that you broadcast them to anybody that has subscribed to them. 

Thus don’t constrain your architecture to just be event-driven.  There is really no money to save by doing just that.  Let it be components with channels.  The channels I am talking about were already adopted in the telecom standard SDL back in 1982.  In EDA it is basically a mechanism for brokering events.   In the OMG standard CORBA from the early 1990s it was called the “Event Service”.  What a coincidence!  Actually one way of thinking about EDA conceptually is really that it is all that CORBA was meant to be, but in the Web/Internet world. 

The most interesting components are services.  You get service-oriented architecture at the same time, and more. 

However, those of you who think this is fundamentally new have really not done your homework.  It is probably true that the three letter combination EDA is new as it was for SOA.  We have also got some new great platforms that make it easier to implement these ideas.

Over the years I have seen trends in the component world that put more focus on the components than the channels (and thus the events) between them.  Other times it has been the other way around.  However, there is absolutely no reason to choose.  You should allow for both. But what we don’t need are more buzzwords. They don’t help us at all. 

To summarize, you should go for a component architecture without any compromises.  This is what made the Ericsson AXE system such an incredible success story more than 30 years ago.  And thanks to its architecture it is still probably the best selling product of its kind in the world.  However, Ericsson had to build its own infrastructure managing components with channels since such solutions didn’t exist at the time.

Of course, this is still new to people who have not previously developed a component architecture.  Thus those people have to come up to speed and that means training and mentoring.  And, to start with you need some good technical practices.  It is as easy as that!