Kurt Bittner avatar

The Achilles Heel of Agile

The Achilles Heel of AgileAccording to at least one source, Achilles, the hero of the Greeks in the Trojan War, was said to have been made invulnerable when his mother, Thetis, dipped him in the Styx, the great river of the Underworld, immersing all but the heel by which she held him.  From this we are given the expression the "Achilles Heel", which implies that something is not only a source of weakness, but that the cause of the weakness is also the source of great strength.

Agile has an Achilles Heel: the Product Owner.  The involvement of an empowered, knowledgeable, committed and engaged Product Owner is a source of tremendous strength and benefit of the agile approach; in fact, the Product Owner is the single indispensable person on the project, without whom nothing can be done. Like Achilles, the Product Owner seems to need to have powers beyond that of mortals: they must negotiate consensus with all the stakeholders in the business, they need to be almost omniscient in their knowledge of the business domain and the goals of the project, and they must be unfailing in their vision of the solution to be developed.  Oh yes, they also need to be available any time the development team needs them for feedback.  When presented in these terms, the Product Owner seems just as mythical a being as Achilles himself.

The reality, on real projects, is that it is almost impossible for one person to do all these things well.  If the Product Owner is a skilled negotiator of consensus among stakeholders they are not going to be in the team room all the time.  If they have a strong vision for the solution that vision is likely needed elsewhere as well. Visionaries also sometimes have a great "global view" but are sketchy on important details.  If they don't have a compelling vision for the solution they may vacillate or even flip-flop on important decisions.  If they lack important negotiating skills they may tell the team one thing only to have their decisions overruled later by stakeholders who were not involved.

The reality is that there is often no one person who can fill all the roles the Product Owner must fill. In these cases you will have to find different people to play different aspects of the Product Owner role.  The best Product Owner is probably one who can negotiate consensus, make effective and timely decisions based on that consensus, and be available on a predictable basis to work with the team (though not all the time). When she lacks understanding of the details of how something should work, she will enlist the support of the right people and get them engaged as needed. The grand vision is important but as long as it is provided by one of the stakeholders the Product Owner need not be the "big picture visionary".

The most important thing the Product Owner can provide the team is consistency, in participation, and in direction. If she can do that, she need not be some mythical hero. As long as the right people can be engaged at the right time, and as long as the guidance provided by the Product Owner accurately reflects the desires of the stakeholders and does not fluctuate over time, the agile team can make great progress toward delivering a solution that meets the needs of the business.

Warming Up to Agile

Warming Up to AgileSuccessful agile development introduces a lot of new things, or at least things that are often new to the people starting to adopt an agile approach.  Working with a backlog, engaging a product owner, and utilizing new techniques for planning and estimation often obscure the fact that in order to work in an agile way you have to be highly proficient at what we might call foundational software engineering. Some agile approaches talk about "sprinting", and perhaps this is a good metaphor.  Before you can sprint, you need to be in reasonable shape and you need to warm up a bit in order to avoid injury. Agile software development, delivering working, tested code in short time increments, requires some pretty solid skills in some specific areas. For the moment I'll focus on a couple of key areas we often find wanting when working with teams just starting their agile journey.

In order to develop in short cycles, let's say 2 weeks, you are going to need to build software quickly - certainly daily, and ideally continuously.  Concretely this means that builds need to be automated and executable anytime someone needs to do a build.  If you have to ask someone to do the build, and if it requires manual effort, you simply won't be able to build effortlessly. You need to be able to run a build without thinking about it, because you will have a lot of other more important things to think about.  If you're thinking about "going agile" and don't yet have an automated build capability, focus on that first.

Once you've got the build automated, you'll need to automate tests as well.  When we say that an agile approach focuses on producing working software, the evidence that it works is provided by tests. If you can't run at least unit tests every time the software is built you will never keep up with the demands on testing.  Realistically speaking, just automating unit tests is not enough; you need to be able to do integration, system and regression testing for every build, at least in theory, but unit testing is a good place to start.  In addition, all tests other than User Acceptance and Usability tests should be automated.  Building up from automated unit tests, you should be able to test everything in the system except the User Interface through programmatically-invoked tests.  If you don't do this, you will never keep ahead of the testing workload and your test coverage will remain weak.  If you don't have automated testing, runnable after every build, you will be able to "play" at being agile but you will never really achieve the kind of quality that you will need. At some point you will suffer from lack of testing and something important will blow up in production and that will be the end of your agile experiment.  As with builds, if you are not automating testing now you really need to get your testing act together before you get very far down the agile path.

While you're working on automating your builds, there is another skill in which you'll want to be proficient: refactoring.  There are a number of good books and web resources about this topic, but knowing is different than doing.  In short, you'll want to be comfortable with making significant changes in the structure of the code from one build to the next, done in concert with changes to the unit tests, so that the build and automated tests don't break from one build to the next.  You won't have the time to step back and do "big design" efforts, so you're going to have to get comfortable with making small changes that, over time, add up to big changes, all while continuing to do other development. This takes time and practice, and once you start sprinting you will either have to have it mastered or get good at it very quickly.

This leads to another foundational skill: source code management (SCM).  First, you need to use a source code control system to keep track of changes so that you can roll back to prior versions in the event you change your mind about an approach.  You need to be able to keep work that you are doing out of the build until everyone is ready for it.  You need to version everything, including unit tests.  Sometimes, when undertaking major refactoring, you may need to branch the code, and then merge it later once the changes have been proven. This is all really basic SCM, but it is essential to being able to achieve reasonable velocity and throughput for the team.  In fact, I've put things a bit out of order, because it's really virtually impossible to automate the build or refactor without solid mastery of SCM.

These are by no means the only things that you'll need to master in order to work in an agile way, but they are the foundations on which the rest of agile development is built. Teams that have not gained proficiency in these foundational skills need to focus on doing so before embarking further down the agile path.  There will be plenty of other new things to learn along the way.

Less is more

The motto "less is more" is often used in art and architecture circles to refer to a philosophy of minimalism ( http://en.wikipedia.org/wiki/Minimalism). Minimalism is related to a concept often referred to as Occam's Razor (http://en.wikipedia.org/wiki/Occam%27s_razor), or the idea that the simplest possible explanation should be chosen when considering alternatives.

These concepts came back to me as I considered the situation on a team with which we had been working.  The team was making reasonable progress, but it was clear that a couple of team members were struggling a bit.  The dilemma was this: does the team join together to help bring the team members along or do they, to use the vernacular from a well-known television program, "vote them off the island".

Conventional wisdom says that more people can get more work done than fewer people, but as Fred Brooks observed long ago in "The Mythical  Man Month", more people frequently do not get as much done, and adding people to a project can actually slow down velocity.  To apply Occam's Razor, we should have exactly as many people as we need on a project to get the work done, not a person more.  To apply the principle of minimalism, we actually ought to be a little understaffed.

People tend to perform better when there is a little stress.  As hunter-gatherers long ago, we probably hunted better with a little hunger driving us on; a full stomach leads to a kind of sleepy complacency.  We need goals that are just a little beyond our reach, as observed Robert Browning when he said that "a man's reach should exceed his grasp".

All this is a nice theory, but what about our team?  In true self-organizing fashion, the team decided that it would make more progress with fewer people.  The result?  The velocity of the team actually went up, proving that, at least for this team, less was indeed more.

The Product Owner from Hell

On at least two occasions we've run into a Product Owner that was, to be honest, an agile team's worst nightmare.The Product Owner is supposed to represent the interests of the business, providing information on needs, requirements, priorities, and the like.  This one, however, was a former IT guy, steeped in the traditional approach.  Instead of merely establishing priorities and representing the needs of the business, he also wanted the team to have a detailed task plan that he felt he needed to approve.  He didn't understand or accept the idea of a self-organizing team.  In short, the guy was a control freak who felt that being the product owner put him in charge of everything.

While we want the Product Owner to be engaged and feel a sense of commitment, this guy needed to be committed, or at least get some therapy!  The twin ideas that there is no detailed iteration plan, and by this I mean a plan with formal tasks and dependencies, and that the team is self-directing, flew in the face of everything this former IT manager held dear. 

The agile team needs to act as a team, based on bonds of professionalism and mutual accountability.  There is no room for micromanagement, and being appointed the Product Owner does not come with the right or responsibility to tell the team how to do their jobs.  The Product Owner has clear responsibility for the what and the why, but not the how or when.

When you find yourself faced with such a Product Owner, the only suitable reaction is to stage a coup; you need a different Product Owner.  If that won't work, you need to accept that your project is not going to be agile.  You might even want to think about finding a new project.

Funding New Projects

In a prior blog entry I noted the traditional approach to funding new development efforts:

  1. the business gets an idea and writes a few sentences of description,
  2. IT tries to figure out what the business really meant and comes up with an estimate based on their best guess, and
  3. this amount gets locked in as a "committed" estimate to which IT is held accountable. 

The only thing that should be committed is the people who think this is a sane approach to funding projects.

A better model works like this:

  • The business identifies a set of desired business outcomes that they would like to achieve, along with the net present value of achieving those desired outcomes.
  • Based on this value, the business decides how much it is willing to spend to achieve those desired outcomes.  Usually this is based on the required rate of return of investments in the business.  This amount establishes an upper limit on the project that could deliver those outcomes.
  • The IT planning process then focuses on determining whether the desired outcomes can be achieved within the constraints established by the business planning process.  If they cannot, further funding for the project is stopped and more worthy projects are funded.  Several alternative approaches may be tried and rejected, and deciding project feasibility may require some prototyping and evaluation to determine viability.

This is how research and development funding works and, let's face it, software development has a lot in common with R&D: both deal with a lot of unknowns at the start of a project, so much so that it is usually not possible to precisely define the exact work that will be needed to achieve a particular end.  Both R&D and software are essentially exploratory: at the beginning of the effort the exact result is not known with a high degree of assurance so we have to evolve toward a solution.  The current funding model's assumptions that the solution is easily definable and therefore should be easy to estimate just doesn't work for most new software development efforts.

How to Fund Software Maintenance Efforts

Part of the problem with funding projects is that, from a funding perspective, we treat software development as a one-time event.  When we create a new application I think we all expect that maintenance will need to be made over time, but this is not reflected in the funding model.  Each round of maintenance usually requires a project charter and funding approval.  In obtaining funding approval the maintenance project now has to compete with all other projects for funding approval.  This sounds sensible until you think of the implications.

If we have a building with a leaky roof, we know that we have to fix it or more damage will occur.  In software it is just the same: if the existing application supports the wrong business process, damage to the business occurs as long as the problem goes unaddressed. Forcing application maintenance into the project approval process can delay, sometimes indefinitely, the funding of needed maintenance, resulting in higher costs later on. Read More

Fixed Scope + Fixed Schedule + Fixed Budget = Certain Failure

It is that time of year again, when organizations are putting the final touches on their project funding plans for tomorrow.  Over the past several months, a strange tale has played out across the globe, a farcical tale that if you tried to tell anyone outside IT about they would scarcely believe you.  It plays out like this:

Someone in the business gets something resembling an idea - not something fully formed, but at least an inkling that suggests that there might be some benefit achieved in developing something.  A few sentences are written about it and here is where the silliness begins.  All these little fragments of ideas are gathered together and then IT is asked to estimate the cost and schedule - based on little more than the few sentences of description.  If they are lucky they might get to ask some questions.

Everyone knows this is ludicrous, they will say, but they say they have no choice.  They put all manner of disclaimers on the estimates but these are forgotten over time.  What is recorded and remembered is a time and dollar cost estimate for each project - based on, remember, a few sentences of description.  This sounds so absurd that I know you would be laughing if you did not see it every year in your own organization.  But wait, it gets worse. Read More

The Journey Toward Self – Organizing, Self-Directing Agile Teams: The Role of the Agile Coach

The agile ideal is for a team to be composed of equals, peers who neither direct each other nor oversee each other, but who work collectively toward a common goal. Team members are self-directing in the sense that they, seeing the goals of the team, choose to work on things for which they are most capable. Since many teams start from a traditionally managed, externally directed world, the journey toward this idea takes time and adjustment for everyone involved. A team typically cannot become instantaneously self-directing.

Typically the journey is assisted and facilitated by a coach who helps the team learn new techniques and overcome barriers. The very existence of the coach suggests that the team is not wholly self-directing, at least not yet; the coach will naturally provide some direction to team members during the learning process. The degree to which the coach will "direct" depends on the progress the team has made toward becoming truly agile.

A typical problem is reflected in a recent question raised by a client. We recommended use of a particular newer framework (SOAJ) in place of the Hibernate/Spring framework familiar to the team, based on our understanding of the goals of the project and the skills of the team; in short, the new framework would reduce coding and increase code quality while improving performance. Some team members felt more comfortable with their familiar framework, however, and the team was divided on the use of the new framework. If the team is self-directing and self-organizing, who makes the decision in the event of a split decision?

It often falls to the agile coach to make these decisions. The coach generally has greater experience and is better equipped to make the choice when the team lacks experience. Coaches must be careful not to become dictators, however benevolent, and teams must take care not to become over-reliant on the coach. Over time, the incidence of coach intervention should decrease as the team gains experience and confidence.

Coaching is more than mere facilitating new practices: there is always an element of leadership in successful coaching engagements. Teams learn best by doing, and the coach must lead by doing as well. Over time, team members step forward and the coach steps back until, one day, the team is leading and directing itself. At first, however, the coach takes more of a directing role.

Potential Barriers to the Adoption of Iterative Practices

Understanding the potential barriers to change will help you make the right choices of timing, project, and approach. Some of the more important questions to ask include in the following:

  • How supportive is senior management of the change? The measurements and milestones
    they establish can easily derail an iterative approach, as is the case when they ask even seemingly
    innocent questions such as “When will the design be completed?” or “When will requirements be
    signed-off?”
  • What is the scope of your authority to make changes? How much of the development lifecycle
    are you responsible for? For example, the requirements might have already been specified in a
    format and to a level of detail that would make it more difficult to adopt an iterative approach.
  • What are the team’s feelings about the changes? How enthusiastic is the team about iterating?
    To achieve the transition to iterative development, you will need the support of the team,
    especially the other members of the leadership team.
  • What else does the team have to do? How many other projects and initiatives is the team involved
    in? If the team is not focused on and dedicated to the project, the transition to iterative
    practices will probably be slower and take more time and energy to complete.
  • What capability do you need to improve? It is important to understand the capability of the
    team and how well the current capability supports the proposed iterative approach. For example,
    is there any testing capability in the team? Testing will be needed from the first iteration, which is
    often a problem in companies organized around the phases of a waterfall approach. Read More

Progressive Thinking About Planning

The progressive, adaptive, iterative approach is founded on the following principles:

  • Proceed in small steps, continuing steadily by increments.
  • The plans must adapt to the project’s changing circumstances.
  • The characteristics that distinguish the product or service must be progressively elaborated.
  • They should be broadly defined early in the project and made more explicit and detailed as a better understanding of the project emerges.
  • Large projects must be broken up into smaller projects.
  • Requirements errors can be 100 times more costly to fix after deployment than at requirements time.
  • Initial estimates have a large margin of error—most waterfall projects cost two to three times what was initially predicted.
  • The first one never works; always develop things at least twice.
  • Initial designs will be flawed.
  • Requirements must evolve alongside demonstrable versions of the product.
  • Teams need to own and be accountable for their achievements and plans.
  • You should believe the project results, not the project plan.
  • Detailed planning should be limited to the next milestone.
  • Projects should adopt an active and aggressive approach to addressing project risk.

Progressive, iterative processes were developed in response to the problems inherent in the waterfall approach. Instead of developing the whole system in one go, an increment is selected and developed, then another increment, and so on. The selection of the contents of the first increment is based on risk, addressing the highest-priority risks first. To address the selected risk(s), select a subset of the requirements to be implemented and tested. Develop the minimal set of functionality that enables objective verification (through a set of executable tests) that the risks have been successfully addressed. Then select the next highest risks to be addressed in the next iteration, and so on.
One of the great ironies of modern software development is that in the paper that led to the popularity of the waterfall approach to software development, Winston Royce argued that projects should always go through at least two development cycles because the first version never works. He was actually arguing that projects should take a more incremental approach, not that they should adopt a prescriptive waterfall development approach. Progressive, iterative approaches take this principle one step further by splitting the lifecycle up into many smaller increments, each explicitly addressing the project’s major risks.

Page 1 of 212