Wednesday, November 17, 2004

Agile Methods vs. the Waterfall Method

I was taught about many different approaches to the software life cycle while I was in college. The waterfall approach to software projects was seen as the wisest way to go, since this approach addressed the concern that a change made at any phase of the life cycle ought to also affect previous phases’ documentation and testing. One of the last courses I took at school included a section that quickly covered agile approaches. In my great enthusiasm over these new ideas, I went to work and shared my enthusiasm with my co-workers. Wouldn’t it be great if we could skip over the weighty documentation phases and start producing a working solution while the business needs are still fresh in our minds? Wouldn’t it be great to work closely with other team members to help each other proof our architectures and brainstorm complex issues as they arise and before struggling alone? My co-workers laughed at my na├»ve enthusiasm. Who want to sit next to someone else and watch while they code all day? In our profession of high intellect and large egos, who wants to be dragged down by being forced to pair up? How can we start to build a solution before every detail of the problem is fully defined?

As some years have passed, or maybe as my employer has changed, the agile methods have really seemed to catch on. Iterative approached are being tested, and in my experience they are successful. I do not claim to be a guru on agile methods, and I do not claim to know the ins and outs of SCRUM versus Extreme Programming. But what is the key difference between the agile methods versus the traditional methods to software engineering?

I think the difference can be summed up in one word: communication.

While communication has always been an important part of software engineering, I fell that the traditional approaches encourage us to lay everything on the table from the start. Go through a phase of requirements gathering. Go through a phase of specification writing. Go through a phase of writing a detailed design document. Finally, build what you planned. Plan your work and work you plan, right?

I say wrong! To me, the key to the success of the agile projects I have worked with lay our ability as a development team to admit that we cannot ask all of the questions up front. Besides the fact that requirements change with time, so does our understanding of the business problems. And I have never seen a requirements document that is not laden with holes and ambiguities. A small detail missed in the requirements phase can be the downfall of the entire project.

Agile methods appreciate this risk and mitigate it through constant communication. There is no one phase where meetings are held with the end users. The end users are involved in meetings with the development team throughout the life of the project. The end users are part of the team and play an active role in communicating with the development team and working with the seeds of the end product from its infancy to its completion.

As a result of my experiences, I hope to never spend a year working on a functional specification again. I hope to never face another deployment phase of a project that follows a year of closed-door development. I hope to say goodbye to the waterfall approach once and for all and hello to agile development.

21 comments:

Anonymous said...

Amen, sister!

RG said...

How awful for you that you had to work with such negative co-workers with their large egos and bad attitudes about the glories of agile methods! My goodness, how could you stand such blowhards?!

;)

Seriously, though...with regard to paired programming, can I make an observation? In general, I've noticed that you are a fairly positive person who likes most people. I won't pretend to speak for everyone, but some more seasoned developers like myself have worked on too many projects with people that we frankly didn't like very much. Perhaps we didn't care for their programming approach. Perhaps they had a personal mannerism or two that we found annoying (like speaking very, very slowly and not saying anything useful). Or perhaps their personal hygiene left something to be desired. At any rate, we know and understand that if paired programming becomes the norm, sooner or later we'll be paired up with these yutzes. And that is a price that is just too high to pay. Ick.

Stop the madness.

Anonymous said...

Hey Val...

Here are a couple of thoughts from someone who manages and has managed software developers for many years.

You know this "Agile Methods" stuff is hardly new. Discussions about group programming efforts, iterative development, and end user involvement from the get-go were prevalent back in the mainframe days before you were born. We just did not have sexy names for it.

I can tell you from experience in the real world, not just in the rarified air of academic computing departments and consulting firms that I can find programmers who work well using “agile methods” as well as developers who do not. I can also cite projects after project that was successful using such methodologies as well as projects that literally crashed and burned.

But the truth is that I can not think of single project that I can say was successful or a failure BECAUSE of the methodology that was used. And by the way, most customers (you know the folks who really pay your salary) don’t give a damn about this methodology crap. In fact, they get bored and annoyed very quickly with it.

Even worse many customers I am familiar with are highly skeptical of “paired programming” and lots of group activities. They are concerned that in many instances such processes are simply excuses to charge them more for less productivity.

In the end, I think all of you ought to be less concerned about the methodology you are using and more concerned about the value you are delivering.

Or put another way, do you think your REAL COMPETITION (the folks coding in New Delhi, Kuala Lumpur, and Bangalore) are spending their time posting on blogs about methodologies? Do you think they engage in hand-wringing exercises complaining about how much people with or without degrees are being paid while they are being twice as productive writing just good-enough code for 30% of what you are being paid?

blameMike said...

Anonymous,

I'd like you to clarify something for me:

What you're getting at is (and correct me if I’m wrong) -- since the off shore developers aren't spending time to improve the software development community by reasoning about different methodologies, patterns, and best practices -- we should follow suit?

The impression I get from your comment is you're a PM type with the horrific "ship it" mentality.

While admittedly, I'm not the biggest fan/advocate of pair programming, I feel these types of discussions only help to strengthen the community.

I guess what I'm getting at is... your comment is negative, and very counter-productive. In the future, keep such comments, to yourself.

Aaron Ballman said...

Re Anon: No, you're probably right -- they aren't putting much thought into methodologies. Which is why they don't worry me one bit. It just means they can churn out shit code faster and sink the companies who employee cheap labor. You get what you pay for...

Anonymous said...

No folks, I am not the point-haired boss of Dilbert fame. I am the CIO of a 2000 employee manufacturing firm who has spent an entire career working in all facets of IT including many years as a software developer.

Please understand if you folks would prefer not to have opposing viewpoints represented then please pardon my intrusion into what will be little more than your mental-masturbation session as you stroke your egos over how smart you folks are for advocating agile methods to one another.

However I hope that my alternative views might be of some help to you folks who appear to be very much like the young people on my staff who turned me on to this blog in the first place.

It seems to me that you folks are sort of “singing to the choir” when you ought to be trying to convince people like me that you are right. After all, it is people at my level who in most cases decide whether or not to buy your services or even decide how you are going to work.

Let me explain to you why I injected the spectre of foreign outsourcing into your discussion. Earlier this year I made the difficult decision to outsource a very important development effort at my company to a firm based in India. This was a very controversial and potentially disruptive decision. However the business case for making this decision was quite sound and it was being advocated by my boss, our company’s CEO, as well as the VP of the division of our company who will be using the software.

Competing with the firm in India was a local IT firm, as well as our own in-house developers. The bottom line was that the firm from India’s project bid was literally 1/3 the cost of the local firm and they guaranteed delivery in about one half the time. Our in-house development staff, who has a back-log of more than three man years could not even come close to meeting the demands of the business unit in question. Both the local firm and my in-house staff are proponents of methodologies such as paired-programming and as part of the bid process they tried to assert that they had a competitive advantage because they were local and would have more intimate contact with the end user and could construct better quality solutions.

Over the past couple months we have implemented the solution crafted by the firm from India. During our review of the code, several of my in-house folks argued that the solution developed in India was not very elegant and did not implement various patterns properly. They in essence tried to suggest that we were buying “crappy code”. And yet, the code was certified by our QA group in record time with minimal defects. Likewise our end users are satisfied with the performance and functionality of the solution.

Now, do I believe that the code from India is perfect? Nope. Do I think our in-house folks might have been able to write better code? Perhaps. Does it matter? Not much.

The software from India was cheap and did what we wanted it to do. Will it scale? Maybe. But then again, a lot of code our in-house folks have written over the years that supposedly would be scalable and reusable turned out to be neither.

Now consider my real dilemma: On the one hand I have folks on my staff advocating agile methods which look to be more costly in terms of manpower and possibly developer productivity. On the other hand, I have a happy VP and CEO who are asking me why we should not do more of our development in India where they seem focused on getting the job done in the quickest and most cost effective manner.

Folks you can choose to believe that everybody in management are a bunch of cretins who do not appreciate the beauty of your code and the wonderful benefits of your methodologies. But I will tell you that we work very hard to make sound business decisions that are good for our company and our bottom line; both in the short term and the long term.

All of us hate the notion of outsourcing to India. We like our in-house programming staff. They are our friends and neighbors. But we will hold our nose and make the best business decision.

So I strongly recommend that when you advocate for agile methods you make certain to make your case in terms measurable speed of delivery, value to customer, and lower costs. Because in the end, flowery declarations about more elegant code and singing “kum-bah-yah” with end users is not going to carry the day.

Valerie Vogt said...

Anonymous-

As the author of this blog, I appreciate conflicting view points, and I quite enjoyed your comments.

I agree with you that scalability and flexibility of a solution are only valuable if it is a requirement of the customer. I wrote a post on Gold-Plating that asked if it is possible that in trying to deliver the most elegant solution, we end up costing our clients money when we are not enhancing their solution in a way that matters to their business.

As far as agile methods, I agree that we need to be able to sell management on the value that agile methods will add to projects. I believe the value added lies in the fact that customers will end up with the product they need rather than the product they intially speced out. There may be projects where the intial specification ends up being everything that is needed. In my experience, which I admit is not as lengthy as your experience but I have been around a while, there are key business needs missed in the specs or business needs change between the time the spec is written and the time the product is delivered.

I am interested to know more about the project you sent out to India. Was the project complex? How large of a project was it? Even if your QA department gave the application a thumbs up, have you needed the product enhanced or have you come across any parts of the application that are not what you initially intended?

--Val

blameMike said...

I'd just like to apologize for being a "Comment Nazi" when replying to the Anonymous CIO.

As Val pointed out, this is her blog. I shouldn't be telling the readers not to comment. The topic of off shoring just really gets under my skin, for the obvious reasons.

My bad.

Valerie Vogt said...

blameMike-

Why does the topic of off-shore work bother you so much? Have you been directly affected by this?

--Val

blameMike said...

Hey Val,

I have not been directly affected by off-shoring (although some of my friends have).

I just hate to see accomplished developers loosing contracts to foreign programming firms, so corporate management can get richer.

RG said...

blameMike -

Ah, yes, those evil corporate managers. How dare they get richer!

Seriously, why does this bother you? I've never understood why it is a problem for organizations and their corporate management to make decisions based on costs. I mean, we can argue that they are wrong about the costs, that they are not truly evaluating accurately the long-term costs of outsourcing work. But to just say they are just getting "richer" seems to apply a negative value judgment to them that isn't necessarily fair.

It never bothers me when other people make more money than me. More power to them. It only bothers me when the government takes away the money that I earned.

blameMike said...

Avonelle,

While my comment was definitely not composed of the most strategic prose possible, I hold steadfast to it.

“Ah, yes, those evil corporate managers. How dare they get richer!

Seriously, why does this bother you? I've never understood why it is a problem for organizations and their corporate management to make decisions based on costs. “

This is probably going to come as a surprise (due to my previous comments), but I currently hold a substantial vested interest in a start-up software development company.

While I am in no way, shape, or form a “corporate manager” (I’m a geek), I am faced with decisions based on cost. So let’s talk off-shoring and the benefits versus the ramifications…

I could easily outsource the development of my company’s products to an off-shore company, and yes we’d save money (from the comment Anonymous CIO made, a fair amount at that), and I may even perhaps get richer from it. However the impact on other American developers, far out weighs my incentive to strike it rich.

“But to just say they are just getting "richer" seems to apply a negative value judgment to them that isn't necessarily fair.”

I agree with the above statement you made. I was absolutely unfair about that.

Michael Craig said...

Waterfall is dead, we all now it but companies are too scared to try anything else unless it is a very small pilot project.

Long live Agile.

What vs How

VinnyR said...

Re: Anonymous CIO of 2000 employee firm

So your QA department signed of on code because it was functional. Thats great. However your poor knowledge of software development shows when you dismiss the proper use of patterns and elegant code. Yes you managed to get the code out quickly, but it will hit you when you need to make any changes to the code. The aim of using patterns is to have clear maintainable code.
What is likely to happen if the code is not well designed is that it may take a lot longer to make what would seem to be a trivial change. All because shortcuts were taken at the start.

VinnyR said...

By the way I have nothing against outsourcing, however it does not absolve responsibility from the client of the outsourcer. The whole process needs to be closely monitored with careful reviews of designs and code.

james said...

I was talking to a good friend the other day about project management styles (scrum vs waterfall), engineering practices (xp, rad, rup, etc), and culture. We were mentioning how agile is becoming more and more 'mainstream' these days because the culture of organizations are more and more accepting of it.

I find it also funny that people with years of "experience" claim that there's no significant proof that methodologies have any effect on efficiency. It's entertaining to also hear that 'paired programming' is thought simply as putting two developers on the same task expecting them to put out "twice as much code". That isn't the intent and someone who has any clue about XP would know this.

Anyhow, Agile when used correctly isn' tnecissarily more productive, but it IS : more visible, more accountable, and more enjoyable to the developers. These things make it great for a culture that accepts such things.

For example, your middle managers who always say "nothing is wrong, everything is going great" have a lot to fear from agile practices, because they cannot hide behind politics and paperwork anymore.

laters!

trishacayetano said...

i prefer agile methods

Anonymous said...

Just curious as to why a CIO of a 2000 worker company takes the time to troll C# blogs. Anyway, lets take this offshoring thing to it's logical conclusion (cost saving wise) and outsource the entire IT department-CIO included, as I suspect there is someone there who can do that job for 1/3 the cost too. Amazing how once one is invited into the farmhouse (Animal Farm) one can take on a different viewpoint. Meanwhile , back here in the barn...

Anonymous said...

I've been involved in building solutions for many years now, during that time one of the key lessons I have learnt is that you need to tailor your approach / method based on the kind of project, client needs, complexity, quality... the list goes on. This could mean you select 'water-fall', agile, or some kind of iterative, prototyping etc...
It isn't as simple as saying 'we do agile development here' and throw the detailed requirements and design templates out the window.... Some projects I have worked on suit agile techniques - e.g. where there are good subject matter experts available, complex abstract concepts are not involved... etc.
Other projects require some of the diciplined aspects that agile steers away from.
You can fail to deliver no matter what method you use...

Anonymous said...

Currently involved in a fairly large waterfall project with a big old-world client. Actually, the project isn't that large, but requires a tight delivery date, so it's been comical watching the process fall all over itself. Literally, meetings to plan meetings, reams of docs no one will read, and condescension to those of us who suggest an Agile approach. No one believes the dates will be hit, but they are working very hard to make it happen. In the end I think it will, but the massive investment in time and effort is epic. If they had applied half the resources to doing actual work, and pared down the process, it would be done by now, at half the cost.

Valerie Vogt said...

I am sorry to hear that you are in an old, stodgy waterfall project. How long are you spending on requirements and design? Will you at least phase development releases?