In my last blog I described our immaturity when it comes to software development. Companies follow the latest fashion and jump from method to method without standing on what they already have adopted and that is working.  This is expensive and it results in bad software.  When I am upset I say that it is stupid and ridiculous.  How do we change this?

We need a basic theory describing what software development actually is. In my opinion, this theory is right in front of us - we just need to grab it. We can start with all these methods, processes, and practices and find the "truth" of software development. We have already done this in my company and the result has been used by hundreds of companies around the world. Read More

I am deadly tired of all the buzz that we are suffering from.  Our view on how software should be developed seems to change dramatically every second or third year, more frequently than the whims of fashion.  Big companies around the world carelessly discard expensive process and tool investments, almost before they have even tried them.  Instead of learning from experience they heedlessly start with something they believe is fundamentally new.  In reality very little has changed.  As in the fashion world, there is much ado about next to nothing. In something as trivial as clothing this may be acceptable, but with the size of our investment in software this is wasteful, expensive and absurd.

The latest trend is “being agile” (as exemplified by Scrum).  The “agile” movement has reminded us that people matter first and foremost when developing software.  This is not really new - this theme resurfaces every decade or so as naive managers try to mechanize and commoditize what is basically an exercise in creative problem solving.  It is important that we not lose track of how to work as a team, how to collaborate, how to document what we do, and how to plan our work on daily, weekly, and monthly timescales, etc. But in bringing these things back to focus, much is lost or obscured by new terms for old things, creating the illusion of something completely new.   

The result of this is a lot of wasted effort as old truths are rediscovered but cloaked in apparent new clothing. Younger and less experienced coworkers promote new trends, following new gurus, supported by the hype of a media always hungry for “news”.  Managers who have lost touch with actual development find themselves in a hopeless situation: resist the newest fashion and they brand themselves as out of touch. Pilot projects are run to force proof of merit for the new approach, but motivated developers can make anything work on a small scale.  As a result the new approach overtakes the old, and all that was working with the old approach is thrown out along with the parts that were not.  Only too late do they discover that the new approach itself has parts that don’t work along with the parts that do. 

At the root of this problem is a deep misunderstanding of the nature of software development.  Researchers have tried to attack this problem with new theories like formalism to prove correctness of programs, or through formal languages which never have been adopted outside the academic world.  Industry efforts have spent years standardizing swollen meta-models that defy easy understanding. 

Universities and technical institutes teach us a particular way of working.  Every project adopts a special method which we must first learn and master before we can begin to work.  Every time we change jobs we have to learn a new approach before we can get on with the real task at hand.  This is not effective; we cannot learn from experience as we are forever starting over.   

We need to stop forever chasing after fads and easy answers that forever disappoint.   But how?  This is a problem I have thought about for at the least ten years and of course I have a concrete idea on how we can get there.  However, I will have to wait to tell you that until my next blog.  Let me however say this much:  We need to start small and simple and keep it small and simple.  We did so with SDL and UML.  We can do it again.

That would be smart!

What they don't teach you at institutes of technology – how to be smart

I am slowing down a bit this year.  For many years I have made more than ten around-the-world trips and lots of local traveling, but this year I have made only eight. J  During this time, I have given a dozen keynotes and more than fifty other kinds of talks.  I am often invited to universities or institutes around the world, and I have been offered professor jobs, surprisingly even in Sweden.  Thus, I feel I have quite a good understanding of what they teach you about software.

Interestingly, most professors in software have never built any useful software.  This is natural since they have grown up in the academic world where software engineering is not as fine as computer science.  Of course, this is not true for professors in your part of the world, but elsewhere. J  Thus, most professors can’t possible teach in software engineering with any passion.

As a consequence new ideas about software engineering have come from practitioners around the world.  The best ideas come from practitioners with an appreciation of science, people that agree with Kurt Lewin: "There is nothing as practical as a good theory".  Examples of good theories were Newton’s laws, and Einstein’s relativity theory.  Most research represents useless theories.  Of course, this is not true in your part of the world. J

Personally, I adopted this motto thirty years ago and it has been leading my work all these years: components, use cases, Objectory that became RUP, SDL and UML, and now practices.  Many other people such as Kent Beck and Ken Schwaber have made similar contributions.  This is the true essence of software engineering and it stands on a sound theoretical basis, meaning it can be explained so others can understand it and practice it.  But, very few universities teach it and even fewer do it well.  As I said, this is of course not true for universities in your part of the world. J

However, these exceptional institutes that teach you software engineering practices don’t teach you how to be smart when working with these practices.  And this is fine.  I am happy if they teach good practices, but to provide real value, each practice needs to be applied in a smart way.   The software industry has the capability to help itself in this regard, but I really don’t see that happening.

Smart is about being agile, but it is also about doing the right things, the right way.   You can become smarter through training one practice at a time.  However, without experience your alternatives are too many and only a few of them are smart.  Experience is of course something you can get sooner or later, but it takes time if you must learn by your own mistakes.  This is where the utilization of what we call “smart cases” becomes essential.  We have a large number of smart cases that relate to working with people, teams, projects, architecture, modeling, documentation, testing, process, outsourcing, and many more. 

Each smart case has a key motto for how to be smart, for example:

·       People:  Software is developed by people, not by process and tools.

·       Teams:  A software team is like a sport team with all needed competencies to win.

·       Project:  Think big, build in many steps.

·       Architecture:  Start to build a skinny system, add muscles in later steps.

·       Testing:  Whatever you do you are not done until you have verified that you did what you wanted to do.

·       Documentation: Focus on the essentials - the placeholders for conversations – people figure out the rest themselves.

·       Process: Don’t throw out your baby with the bathwater: start from your existing way of working, find your pain points and change one practice at the time.

And on it goes.

All of this to say, we should learn software engineering practices at school.  We should shorten the time to get experience of practices by collecting reusable smart cases.  We have started this.  We hope others will contribute.  I think that would be smart by the industry. J

Scaling Scrum?

Scrum is excellent for lightweight management of small projects developed by co-located teams. However, it is not designed for large complex systems, or systems that build a SOA (banks, car companies) or PLA (product line architecture for instance Ericsson) or for large organizations with or without outsourcing. Moreover, Scrum is just one practice. You need many more to develop software.

But can’t we just scale Scrum? Scaling is challenging. In the 1970’s people tried to scale to ‘programming in the large’. Later people tried to scale objects to ‘objects in the large’ (mega-programming). Both never made it. Hundreds of books and even more papers were written on the subjects, but it didn’t help. Instead ideas that were scalable at their roots became successful such as components and objects. At Ericsson we never had to struggle with ‘programming in the large’ since we went straight to components and later to objects.

Why is it so difficult to scale? Explaining this is not difficult but takes a lot of space. Instead let me make an analogy. Suppose you have been able to produce a new chemical substance by using a test tube and a Bunsen burner and the new substance helps with an important problem. Now you want to produce tons of the new substance. Nobody would dream of industrializing the laboratory method by simply building a larger laboratory with hundred meter long test tubes and fifty meter high Bunsen burners. Instead a chemical process would be built in which you would not recognize the original method.

Now, why do we need to scale Scrum itself? Why not let Scrum stay the pearl it is. Treat it as one of your practices in your portfolio of practices also including for example practices for requirements to test, architecture, components. Scale by composing Scrum with other practices.

To the fathers of Scrum I would say: Please, don’t extend Scrum with all kinds of other practices such as use cases, user stories test-driven design, etc. and brand the whole soup as Scrum. Doing so will make Scrum heavier and it will threaten the survival of Scrum because of deficiencies in these other practices.

We help companies around the world to adopt Scrum. However, we don’t see Scrum as the center of all you need. Instead, the center is the practices you already have in your organization, the ones your people already have adopted. Replace them carefully with better practices – one by one. Scrum could be one of these better practices but not the only one.

Do you agree with this approach? Now, it doesn’t happen by itself. It is clear that a practice-based approach is what is needed.  We have been promoting and working with practices since 2003.  Now, even IBM and Microsoft have started to join this movement. And this is really good for the industry. It is very smart!

Let’s be realistic.  In a column like this I can only give you a hint of what you as a buyer need to do.  However, it is amazing how many outsourcing agreements would benefit from this basic advice.  But note that basic means that there may be exceptions.

1.       Split the work into smaller, separately defined pieces that potentially can be given to different vendors.

2.       Separate work that is exploratory in nature from work that is mere production.  Production can be calculated at a fixed price, whereas exploratory work can’t reasonably be calculated in advance.  For instance don’t ask for a fixed price until you know the key requirements (say 60-80% of all requirements), and until you know how you want it to be built (you must have an architecture that is implementable). 

3.       It is beyond human ability to specify all requirements for software upfront.  Don’t accept a contractual model where all requirements have to be specified and agreed upon early.  Even worse is to agree on requirements upfront and then pay an unspecified amount for each change.  This is nothing less than horse dealer contracts. 

4.       Outsourcing software development without an architecture is like asking someone to build a house without an architecture.  You would never do that. However, for software, drawings are not enough.  You need to make sure that the architecture can be realized.  All important risks (performance, etc.) must have been eliminated.  That requires you need some executable code – usually 5-10% of the final code must be in place before someone can calculate a fixed price.

5.       As for a house, you need to inspect at certain points to make sure that the work is on track.  Inspection in our case must include executable code to be meaningful.

Now, this is not easy to do.  Still, this is just the basics.  For large outsourcing projects such as building an enterprise wide SOA system, the required competencies are much higher. 

6.       You need to be able to split the work among several subcontractors.  Thus, you need to set a building standard.  Otherwise different vendors will deliver non-compatible software.  It is not enough to specify what platform to use, but you need to specify how to make drawings (UML for example), what design practices to use (use cases for example), when inspection needs to happen and more.

The general advice is to split the work in smaller manageable pieces with fast deliveries.  Think big, but buy small!  Western companies outsourced big M$ projects to India.  Japanese companies are not giving up control that easily.  Their outsourcing contracts to China are in the order of $100k.  Many western companies are now taking back (insourcing) their software to maintain and develop it at home.  The new strategy is to insource the business critical applications, but outsource less critical software.  Reflecting on this, it seems we are going back to where we were before the outsourcing frenzy started.

Again, if you don’t have this competence or you cannot get it, don’t even bother to outsource your software development.  And, competency is not enough.  You need to be smart!

Outsourcing is not new, we have done outsourcing for decades.  The only thing that is new is that we are now buying software from companies located in different parts of the world.  These companies have different business culture and their employees speak other languages. 

Let’s not make any mistake: outsourcing done well is very hard.  It is much harder to be successful with outsourcing than to develop the software ourselves.   So why do we then do it?  Because we don’t have the needed competencies or resources ourselves or because we believe we will get the job done for much less money. Unfortunately, many companies discover far too late that they have made themselves fools and don’t get what they had hoped to get.  Since they have given up their own resources they have no easy way forward. And of course they cannot admit it. There is a new trend arising.  Outsourcing companies take back its software development.  We have got insourcing.

However, some companies are successful with outsourcing.  Thus outsourcing will continue to be important, but it won’t be the silver bullet it has been for the past couple of years.  There is no magic. 

Actually outsourcing software is quite similar to building your own home with subcontractors.  The most obvious thing to do is to create a competitive environment.  You want to have alternatives and not be stuck with one vendor – a vendor that may want to give you a low price for the first order and then lock you in for the future.  Competition as we all know is not just about money, but also about quality (in a wide sense) and schedule.  In summary what you want is good software, quickly and at low cost.

In my next column/blog I will give concrete advices on how to succeed with outsourcing.  Until then a few words about project managers.

You will need a strong and competent project manager to succeed with outsourcing.  This person needs a very solid and modern background from the entire lifecycle of software development including maintenance.  If you don’t have such a project manager, don’t even bother to outsource complete software products.  Many vendors will happily tell you that you don’t need to worry, since they have it.  That is a dream scenario for those vendors.   If you don’t have that strong project manager you need to get someone from outside.  Hire someone from a company that won’t be involved in the bidding and that won’t get a kick-back from any of the candidate vendors.  That someone should also understand the basics in negotiations.  You can be sure that the bigger software vendors have experts in negotiations, and, they will know when they are dealing with pigeons.

 The only way to succeed in outsourcing is that you will have to be smart.  Very smart!

In the work that we do people often want a recipe for developing software, a series of steps that predictably produce a result. Recipes are good, whether in cooking or in other areas, but they are not enough, and not everything that is interesting can be reduced to a simple recipe.

Over the years I've had the chance to observe how people learn. Reflecting on how I learn new things as well, I've come to the conclusion that many people, myself included, don't learn very well from following a recipe. In fact I'm rather hopeless at following step-by-step instructions.  As a kid I liked to tear things apart, figure out how they worked, and then put them back together. And most of the time they actually worked when I did eventually get them back together.

Taking something apart and putting it back together is a special case of learning by example where the thing you are taking apart is the example.  Once you've done this enough you can start improvising and designing new ways to solve the problem.

A lot of software development works the same way - whether it is a piece of code or a requirements specification, a lot can be learned from tearing apart a good example, understanding why it is good and how it works, and then, over time, starting to improvise those lessons learned on new problems.  In fact, given the choice between templates and a good example I'll choose the good example any time.  Even if you don't understand all the principles right away, most of us are clever enough to copy the parts that work that we don't understand and be creative in areas where we need to.  Over time we learn and the need to mimic goes away.  This is the way that all of us learned our native tongues, and the approach still serves us well today.

Over the years I've come to regard 'templates' as one of the great evils of software process efforts. I was actually leading the Rational Unified Process team about 9 years ago when we made a big push to create templates for all the artifacts.  At the time I thought it was a good idea since a lot of people had been asking for them. I now regard the decision as having done a lot of damage.

 That statement is likely to shock some - surely I can't be arguing against having standard formats for organizing project information!  Of course I am not arguing against this - but I am against the way that templates are used in many organizations.  

In these organizations, people have little idea of the purpose of many artifacts or work products - they don't understand what they are doing or why, they just know (or believe) that they need to produce these work products.  The templates have been provided as a cheap way of rolling out the process - rather than actually building skills the templates are provided and people are instructed to simply fill out the templates to follow the process.

This approach is wholly ineffective. Organizations fail their people when they require artifacts without explaining why they are needed, or helping them to understand if they are needed, or helping them to improve their skills doing the real work that the artifacts merely document.

The usual result of providing standard templates is that people often blindly fill out the template sections, not knowing what to put in them, feeling compelled to fill out every section because it's there.  Mindlessly filling out templates is a waste of time - the results are valueless, albeit standard, and do not contribute to positive project results.  

I am not arguing for eliminating templates, however - there is some value in having a standard way of formatting and organizing information, but there needs to be some real information in the results produced.  The real problem is lack of knowledge and skills needed to do the work - this is where effort should be spent, not in standardizing templates.  Once people have expertise in doing the work, a consensus can be reached in the organization about how to document the work.  "Templates" will emerge out of that.

A few days ago I wrote about how outsourcing goes wrong.  Now I will explain more in detail a real case.

A couple of weeks ago I was in the US and visited a state government.  They had outsourced the development of their new Enterprise Software to a US based international software house here called Smarty for the nice sum of several hundred million dollars.

1.  Smarty would develop the enterprise software in three years, then install it, and maintain it for 18 months.  Smarty is responsible for process, architecture, language, tools, and whatever they do will be taken over by the State.
2. Fixed price from the beginning, but changes after signed off specification would be extra.
3. Smarty would deliver a specification during the project and agree with the State on this before moving on.
4. Deliveries were phase based: requirement specification, higher level design, lower level design, code and unit test.
5.  All difficult tests (integration test, performance test, stress test, system test) would be done by the State.

This is a dream contract for Smarty.  It just cannot go wrong.  The State is locked in early to a fixed price. 

The State has agreed to review and subsequently sign off the requirements once they are thrown over the wall from Smarty.  This sounds OK, but it is horribly wrong.  It is beyond human ability to sign off the requirements for a single product even, yet alone totally impossible for an enterprise system.  No one knows the detailed requirements or can fully understand just papers.  The business people, the users and all other stakeholders have some more or less clear ideas of what they want to get done, but we all know that their detailed requirements always change as soon as they get something to use.  Moreover, one of the most important requirements for the client is to get a system that can be maintained and extended for as long as the business will stay…and that must be thought of as forever.  However, Smarty is protected.

To deliver a specification to be signed off, to deliver drawings, and code is the simple part of the project, since you just deliver paper-ware.  You really don’t need to be delayed in delivering paper-ware, you just write a little less <grin>.  And, being a bit ironic, the good thing from Smarty’s point of view is that paper-ware cannot be validated or tested, so nothing can go wrong.  Thus everything will go very well until integration test and acceptance test.  However, in our case this was moved to the State.  Here hell will break loose.

First, of course, it will be very hard to get the software to work as specified and this is where projects are delayed.  However, most important, it will be extremely hard if not impossible to validate that the software does what the business and its users want it to do. With this approach, the State won’t know what s/he will get until after the whole huge project is at its expected end.  However, Smarty is protected: “we did what we said we would do”.  Moreover, the requirements on maintainability and extensibility will not be tested, because they can’t be tested.  The State won’t know until the software is taken over and it needs to be improved, but then Smarty is long since gone.

The case I have just described is not unique.  Many companies which have moved into outsourcing feel cheated.  They have trusted that the vendor knows how to make its customers successful, and the vendor believes they can achieve that. Once the deal is underway, there is not much more to do.  The client has restructured the company to have hundreds or more people working for her/him in the way the vendor has specified.  The client has made a bad deal and doesn’t want to make this public.  And, since everyone is doing the same thing, the client is in good company and can say:  We just need to make it better next time.

However, this would never have had to happen if the deal had been structured smart.

Outsourcing is hot!  Every major corporation around the globe is outsourcing all or part of their software development.  Examples come from all industries: banking, insurance, utilities and government.  They outsource to the large software houses in for instance India or in the US and UK which of course have offshore development in India. And unfortunately the result is lots of unsuccessful projects.  However, the failures are often never talked about because the vendor has got a fantastic contract and the customer won’t speak up about poor contracts that they reviewed and approved.  This is wonderful business for the software houses.  This reminds me of the saying: “My grandfather was a horse trader, my father was a used car salesman, I am myself in the software business.”

Why do these types of projects fail? There are many reasons, but they can all be summarized by saying that they were not smart.  The buyer doesn’t know what he is buying and the vendor doesn’t know how to make it work.  If you were as cautious when buying an enterprise system as you were when buying your own house, you would be much better off.  Would you buy a house from someone by just saying “give me what I need”, and then pay them a fixed price?  No, of course not, but this or similar is what many outsourcing deals are about.  After the agreement the buyer just waits to see what he will get say three years down the road. On that path he will have to sign off on requirements that look good and complete, but which due to the ever-changing nature of software requirements are only correct to say 60%.  Following that path you will get drawings that you don’t know if they ever can be built.  At the end of that path you will have to take over the software without knowing for sure that it works for the people who are supposed to live with it.  This is what people do.  The vendors are protected because you agreed originally to sign off after each phase and you did so because you believed it would work.  It doesn’t work for software.  For enterprise systems it is a proven path to failure. 

The vendor wants happy customers, but s/he just don’t know how to do these kinds of huge projects successfully.  Or maybe the deal on the table is irresistible, and since, most often the vendor is more competent than the buyer and they won’t put themselves in a position to lose anything.

What I say here is controversial. However, someone has to say it.  Now, it should also be noted that there are companies that are very successful in outsourcing.  Their approach to outsourcing is very different based on an understanding on the nature of software, on the importance of starting small but scaling up as quickly as possible (but not quicker <grin>).  I will in another column talk about succeeding with outsourcing.  Some of the things I will talk about are:

1.  Quickly build an architectural roadmap.
2.  Separate work that can be given a fixed price from work that cannot reasonable be calculated upfront.
3.  Split the big contract into many smaller contracts.
4.  Make room for competition by specifying standards and tools.
5. Deliver early, get feedback early.

Simply you will have to be very smart.