The Kernel Journals 1: The Hegelian Dialectic of Software Engineering

The Kernel Journals 1: The Hegelian Dialectic of Software EngineeringWe in the software development industry face a seemingly intractable problem. We have learnt the lesson that prescriptive process is a bad thing. Process bureaucrats sitting in ivory method towers, telling highly-skilled professionals how to do their job and setting the process police on them if they don’t follow their instructions to the letter, can (unsurprisingly) be really quite damaging. It disempowers the development team  and engrains apathetic attitudes along the lines of “When we inevitably under-deliver, it will not be our fault, but the fault of these ludicrous process hoops that we are forced to jump through, instead of being able to focus on writing great software”.  The agile revolution was software engineering’s way of learning this lesson, and the agile manifesto pledge to value “people over process” and “software over documentation” has got to be right. But (… there was always a “but” coming …), we are already finding that the opposite extreme of little or no explicit process isn’t going to cut it either, because it leaves too many problems unsolved, such as:

  1. Avoiding Confusion – there is a big and noisy world of competing processes and methods out there. How do software development teams find the right practices for their specific needs?
  2. Team efficiency – do we really want our development teams to constantly have to reinvent new solutions to old process problems when they should be focusing on writing great software?
  3. Supporting Diversity – some software engineers need / want more explicit practice guidance than others. How do we provide just enough guidance without being overly prescriptive?
  4. Consistency – if every team works differently, how do we get resource mobility / flexibility?
  5. Governance – how does the sponsoring organization manage the risks associated with its investments in software and ensure visibility of progress towards a successful outcome?

We face the following Hegelian Dialectic (I graduated in philosophy – you have to humor me!):

  1. Thesis: Prescriptive process kills the golden goose (development team creativity and motivation)
  2. Antithesis: Removing the prescriptions can distract, confuse and frustrate the team and its stakeholders
  3. Synthesis: We need some kind of smart practice framework that enables us to provide clear direction on what needs to be done and just enough default standard guidance on how to do it, but enables teams to easily refine or replace practices that are not exactly right for them.

This smart practice framework is known as a process kernel. In this short series of blogs I will describe in a little more detail at what a kernel is and report on my experiences of using process kernels in practice to enable my customers to find answers to some of these seemingly intractable process problems.

3 Comments
  1. Roly Stimson | February 18, 2010 at 4:29 am Reply

    avatar

    Christer

    Many thanks for reading the blog and for your comments. I agree with you that the “too much / too little” process pendulum that I described in my blog is definately far from new. I personally think the pendulum is currently swinging ever wider and the debate very much cenre stage thanks to the radical rethinking that the agile school brought and the push now to further propogate its benefits at scale and in the mainstream. I also passionately agree that we need standards and rules. What we need to find in software development is the same kind of acceptable balance between constraints / responsibilities and the individual freedom needed for creative progress that we strive for in society as a whole. There is probably no one magic formula for this and it will always be to some extent be a shifting balance of tensions and compromises. I also agree that ivory tower process is fundamentally ill-conceived and that shared / standard practice should always be based on many years of shared and refined experience. Finally I agree that people are the ultimate key – they are the one true source of all new software solutions and the value these deliver. Truly great people are however by definition in relatively short supply, as are tuly great teams. They are also grown, not born. The problem for process for me is how to we make great teams out of the mix of people at our desposal and how we get many of these teams operating effectively within broader supplier organsiations to the meet the larger goals of both supplier and customer. In my future blogs I will be talking in more detail about some process concepts that I have found useful in this regard.

  2. Christer Sandahl | February 15, 2010 at 4:38 pm Reply

    avatar

    I don’t find the above opposite ends problem (no process – pedantic process) to be very unique.

    These ends are everywhere. For example, if no rules were applied in traffic, there would be a terrible disorder and damages, and if too much protecting rules are applied, the traffic would die down. Or if we had not rules in the society, the same happen.

    In the development world it is the same in many situations. If we don’t apply a structured architecture analyses, the integration will almost newer be finished, and not with enough quality. Or if the project leaders don’t push his nasty rules, nothing would be finish.

    The important is to get the rules and processes right. For me, this is much about knowledge and understanding. If a process is developed by bureaucrats, unsuitable for operative development work, the process gets poor. And the reverse, if highly talented engineers bring together a scientific approach with real practical stomach feeling, there is hardly any limit for process efficiency.

    The problem target management and company leadership to select the right people to hire. Or for an employee to select the preferred company to work on. But don’t blame the process, it can go wrong or right like anything else can and does.