I’m glad to see these points being made. It has always bugged me that both Waterfall and Agile, as commonly defined by the opposite party, don’t typically happen. In practice, Waterfall iterates and Agile plans ahead. I’m sure both sides have their fanatics, but they hardly represent the norm.

I recently had breakfast with one of the engineering chairs at UC Berkeley, and he was very clear about the fact his super-huge government projects are not completely pre-planned and micromanaged. They iterate and evolve. This was confirmed by personnel in the administration office who oversee the contracts and spending. These are the projects that are often cited as examples of big bad Waterfall, yet they don’t work that way.

Likewise, the Agile folks I’ve spoken to freely admit to some up-front planning. I believe even Kent Beck himself once said that the first few iterations of a typical XP project tend to be research-oriented.

In my opinion, it all comes down to the kind of risks you’re facing. Bridge builders are not likely to encounter a sudden mid-construction shift from a suspension design to a cable-stayed design, but the impact to the project would be massive if it did occur. Low likelihood, high consequence, plan ahead. Your typical software project, on the other hand, is likely to need a sudden integration with an outside service or a switch to a new database vendor, the impact of which should be relatively isolated in a well-designed application. High likelihood, low consequence, less planning required.

It’s important that the above not be taken as gospel. A change in bridge design is only high-consequence because we humans presently lack the ability to just push a few buttons and *BAM* there’s a bridge. That may someday become possible, and if it does you can be sure civil engineers will be all over it. Similarly, not all software projects are necessarily low-consequence — just ask the ESA about the Ariane 5.

Understanding the risk is key. If you are able to plan less and change later with little consequence, go for it. If not, don’t.

You have one good paragraph here so let’s start there:

“The “Waterfall” process (analyze->design->build->test) was a straw-man from day one…. We should start testing before we finish coding, which we should start before we finish design, which we should start before we finish analysis. And all these activities should be mingled together. He recommends that we build the software more than once in order to learn form the experience and with the intention of throwing the earlier stuff away. And so on. ”

Exactly – the version of “Waterfall” that you refer to is a very dead straw horse and yet agilists crowd continue to beat that dead straw horse as if that is the horse that “non Agilists” actually ride when they are doing a project?

Waterfall is of course a straw man — and noone uses the straw version of it. Everyone “adapts” the waterfall process to be more incremental and iterative, whether it’s through pilot projects, testing and coding through the design phase, incremental releases, partitioning, etc.

It gets back to the central part of my thesis which is that Traditional Methods (so called “Waterfall” ) and “New Methods” (so called Agile) are really points on a continuum and not discrete.

Additionally your usage of documentation clearly separates you from the XP camp and shows that there is a wide continuum in the agile space…some of which might even intersect the more traditional and iterative models.

Some of the agile rhetoric along these lines is interesting and goes like this: Are you agile? No? Then you must be Waterfall? And thus that means you test last! Q.E.D.

Or…how about… All projects that aren’t Agile are Waterfall! And all Waterfall projects fail! Therefore there was no successful software developed prior to ~2000! (or 1995, pick a date).

The facts on the ground rather overtly contradict the popularly promulgated fallacies in this area.

Jordan

There is, as with all methods (see below) a disconnect between how Agile is described for pedagogical purposes in the books and what actually happens on the ground.

I very much doubt that Cedric has seen the techniques he criticises used in practice, and Cope is well known for emitting strong opinions based on misinterpretations of something he read once. Architectural melt-down in iteration 3? How on earth would he know?

Meanwhile you wonder why the Waterfallists aren’t rushing to defend that method against your criticisms. Maybe there just aren’t any. Maybe that argument was actually lost a long time ago. A very long time ago have you read Royce’s paper that introduces the Waterfall (“Managing the Development of Large Software Systems”, IEEE WESTCON 1970) ?

On pages 1 and 2 he introduces what we would now call the Waterfall process. And then he says

“I believe in this concept, but the implementation described above is risky and invites failure. The problem is [that the] testing phase which occurs at the end of the development cycle is the first event for which [system behaviour is] experienced as distinguished from analyzed. These phenomena are not precisely analyzable. [...] Yet if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required. [...] The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a l00-percent overrun in schedule and/or costs.”

Whoops! The “Waterfall” process (analyze->design->build->test) was a straw-man from day one. Unfortunately, authors of software engineering textbooks seem never to have got past page 2. In the rest of the paper Royce tells us what we should do instead. We should start testing before we finish coding, which we should start before we finish design, which we should start before we finish analysis. And all these activities should be mingled together. He recommends that we build the software more than once in order to learn form the experience and with the intention of throwing the earlier stuff away. And so on. The seeds are there.

One specific comment he makes is that “During the early phase of software development the documentation is the specification and is the design. Until coding begins these three nouns (documentation, specification, design) denote a single thing.” Despite what your prejudices tell you, this is exactly what Agile teams aim do with TDD: they write specification and design documents before coding—documents that have the signal advantage of being automatically checkable against the implementation. One of the things we aim for in Agile development is to have the acceptance/functional/user level tests written far in advance of development of those features. If you think that this can be done without “strategic” thinking about the solution, I’d like to know how.

As far as “Scrum has a product backlog…” etc… By “destination” I’m not really referring to overall long range destinations like the list of features etc, I’m talking about shorter range destinations like how to implement features, implement interfaces, etc.

Agile tends to do this process in an “ad hoc” fashion — so called emergent design, compared to other methodologies. I’m not saying here that they don’t have a list of features, I’m saying they don’t (generally) have a plan on how to implement those features, they “wing it” using TDD and emergent design, YAGNI and other approaches that lead to much (needless) refactoring rewriting.

In other words, Agilists may know the destinations on the trip ( the “backlog” ) but they don’t do much route planning as far as how to hit those destinations, leading to more U-turns, backtracking, and, according to Jim Coplien among others, an Architectural Meltdown around Iteration Three.

As far as knowing their location — the same can be said about “Waterfall” teams — once they start implementing they can also look at how long it took on certain tasks to refine their estimates…. what is unique about agile in that regard?

As far as “wrong assumptions” — what are my wrong assumptions — I’d love to know? Are you saying Agile doesn’t have a tactical bent?

Interestingly, in this post I have bashed “Waterfall” as hard as “Agile” and yet only the Agile folks seem upset. Why is this?

My thesis is that an approach that encompasses Strategy and Tactics, Agile and Waterfall is the best overall approach. Eg, Have Documents, but be willing to refine them. Have an Architecture, but be willing to refine it.

Additionally the notion that Documents can not be refined, and that a Traditional process cannot respond to changes is ludicrous. The whole notion of “Waterfall” as a fixed an inflexible process is merely a parody, a straw man built up by Agilists in a desperate attempt to make their views seem unique and pertinent when in fact they are not new.

It’s Tactics versus Strategy — except it’s called Agile versus Waterfall.

I think one problem agilists have with understanding this is they think that “winging it” is a Strategy. It isn’t. It’s just a glorification of the tactical.

If one sees that Agilists favor “winging it” over “planning it” — and this is obvious from the literature — then it naturally follows that they favor Tactics over Strategy.

Jordan

Agile teams knowing their actual destination: Scrum, for example, has a product backlog that contains “all functionality desired in the product” (http://www.mountaingoatsoftware.com/product_backlog). Extreme Programming has a Release Planning practice. A typical time span to look at in Release Planning is around 6 months.

Agile teams knowing their current location better than Waterfall teams: Agile teams measure their progress in terms of Running Tested Features. They get regular feedback on whether they are building the right system, ideally based on users actually using implemented features. They know their velocity and are very good at predicting how long implementing remaining features will take, based on historical data about full implementation of previous features. It’s much harder to predict how long testing of all features will take based on how long implementation took. In fact, you don’t know whether you are actually *done* implementing features until you are done testing.

Does that answer your question?

As an aside, I have no problem at all with Agile not being perfect. I do feel an obligation to chime in, though, when I see statements about Agile being made based on what I perceive as wrong assumptions.

Well, my post was not to be an “Agile vs Waterfall” debate. It was to show that there are limitations in both methods, and some combo would be best, something you seem to agree with.

The only reason some people saw it as a “debate” is that it posits that Agile is not perfect on it’s own, leading to some push back from the Agile crowd, but it was not a versus post.

Re Star Wars, the original version of my post didn’t have any such references, obviously, but one concrete example is worth a thousand unapplied abstractions.

One of my difficulties with this post is that most people don’t know the difference between strategy and tactics, and many people don’t care about real military history or have much background in it.

Therefore I added the Star Wars examples in order to broaden the readership that it would make sense to….

Thanks for stopping by

Jordan

Agile vs. Waterfall is a useless debate. Agile is a philosophy that should be laid on top of a waterfall, RUP, or any process. That’s the best solution. Any attempt to productize agile, box it, and sell it… leads to a risk that it is perceived as a methodology and a single solution in agile is better than a single solution in Waterfall.

Cockburn’s Crystal approaches could really support this argument well and would make some great points in this debate… but Cockburn himself said at Agile 2007 that businesses seek structured recipe’s like XP and Scrum to implement. It’s hard to pitch an evolving/adpating unstructured methodology (like Crystal). Thus, organizations create a 200-page methodology that attempts to solve all problems and don’t recognize that nothing can solve all problems and that it’s better to recognize change and constantly think and adapt.

In this debate, I believe you are discussing the productized versions of Agile and Waterfall both of which will eventually fail. I advocate that we look at the problems at hand that block productivity and positive culture for a software team, apply the appropriate philosophy at that time and then apply a matching methodology.

Good thread… it made me think.

Argue based on what? Do you have any supporting arguments/examples/practices to go with your generalizations ?

Jordan

But that’s absolutely *not* what we agilists are doing! In fact I would argue that we know our current location much better than waterfallists, and of course we also know our actual destination.

What we do, though, is to steer our vehicle all the time, thereby being very responsive to changes in the route to the destination, and even to changes in the destination itself.

“If the longest horizon an Agile team looks at is 30 days or less, how can that be considered Strategic?”

Yes, if that were true, your reasoning would make much more sense. It isn’t.