The Joel on Software Discussion Group (CLOSED)

Earlier this year, there was a discussion here about "No Bug Database", part of my upcoming O'Reilly book.  Based partly on that discussion my co-author and I have re-written the section as "No Bugs".  Read it here:

http://www.jamesshore.com/Agile-Book/no_bugs.html

(Original discussion here: http://discuss.joelonsoftware.com/default.asp?joel.3.383571 )

> We don't need a dedicated testing phase before release.

Sorry, it's just patently stupid as a general statement.

If you work at MS you have to test N different OS releases and Y different hardware combinations. I think that will be in a separate testing phase.

Cisco will run weeks of testing on a various configurations before a release.

And so on.

I thought it was a good article.

James,

Your essay tweaked my interest, as my background is systems quality in all of its aspects, and I've written or co-written 4 books on this subject.

One of my fundamental problems with the essay is that it conflates "agile" and "XP". The two are not the same, and it reduces your credibility when you treat them as synonymous.

Another problem is your introduction of TDD (BTW, not part of XP per se) as a major silver bullet. Whilst TDD can be a mojor help in improving your software designs, it's a big culture change for most developers and comes with its own set of issues that you don't mention at all.

The section of deciding which bugs to fix is far too simplistic. For a much better analysis of this problem, read Eric Sink's essay "My life as a code economist".[1]

Finally, you don't touch on probably the biggest silver bullet of all in reducing your bug count: the quality of the people on the project team.

Finally finally, get rid of all those split infinitives!

[1]
http://www.ericsink.com/articles/Four_Questions.html

The basic premise isn't *no* bugs, but reducing bugs found in QA, by finding and fixing them in the development phase.  Right?

(This presumes the two are monolithic and sequential, so much so that a separate bug fixing phase exists -- in my experience the two are interleaved and iterative.  QA parallels development -- but no matter.)

I'd say two things are missing from this chapter.  First, more hard data beyond the anecdote briefly mentioned in the opening chapter.  Second, economic and schedule analysis to prove out increased efficiency.  Otherwise this might merely be an expensive and time consuming technique to prevent bugs from reaching QA and the bug database (i.e. it could be an optimization to one part of the process at the expense of another, and of overall efficiency).  The article provides no analysis of global cost-benefit at all.

"Typical quality efforts focus on finding more defects, which of course causes development to become more expensive."

Assertions of this sort are unconvincing.  The "of course" needs to be replaced with hard quantitative data or analysis, case studies, or multiples of very specific examples at the *absolute minimum*.

Even working out hypothetical cost and schedule estimates would be more convincing.  What load does this technique place on developers versus what is offloaded from QA?  Which is cheaper, developer or QA hours?  Which role is better staffed?  What are the loads on developers versus QA?  The answers depend on the organization and specific project to a large degree.  It could just be more productive to let QA catch the bugs, document them in a database, and route them back to development.

A better analysis of the technique would permit a prudent project manager to perform the same analysis in the context of a specific organization and project.  Even some case studies would help bolster this chapter.

Assuming that the same number of bugs actually exist, regardless of whether they become documented, the real point of the chapter is who does the work to find them.

Automated unit testing itself is not a novelty.  The TDD notion of writing tests up front introduces the potential for increasing any thrashing during early phase development, before decomposition and APIs have stabilized, as tests need to be re-written.  In the worst case, TDD and early phase detection potentially waste resources finding bugs that would not have entered QA anyway, because the design or source code would have changed prior to early detection.  Aggressive redesign or refactoring can obviate large blocks of work performed according to TDD principles.

Agile and TDD appear antithetical in that the former presumes endemic instability where the latter requires stability to prevent excessive rework.  I would expect adding the cost of breaking existing test suites to render development less Agile.

"Programmers have long known that the longer you wait to fix a bug, the more it costs to fix [Boehm]."

I've seen a similar sentiment elsewhere, but as this doesn't appear to be a direct quote from Boehm, presume this restatement suits the authors purposes.  The usual statement is that a bug becomes increasingly expensive to fix later in the development process (there is no notion of waiting to fix a bug).  (I don't know specifically what Boehm text or quote is being referred to.)

The usual corollary added by experienced software engineers is that while more expensive to fix, it is less expensive to *find* ('customers are just testers who work for free' :) ).  Early detection (and presumably prevention) of bugs is expensive because a large amount of time is spent analyzing software that would not have generated a defect.  Such analysis is often intensively manual and therefore error prone (thus largely defeating its own purpose).  Not until late stages when the work is reduced to machine form can such detection be automated (ignoring here some formal requirements or constraint specification techniques which are not commonplace).

The big win here is automated testing, attached to Agile and/or TDD or not.  It's not something Agile or TDD invented, either.

The biggest thing I have against Agile, that it borrows heavily from existing practise, then criticizes existing practise on the absence of such practise using worst case scenarios as the benchmark.  Agile so far proves to add little or no incremental value to the existing practises it borrows, and instead introduces some remarkable dysfunctionality, especially in requirements analysis, churn and dependency management.  In its defence, Agile's currency has caused some disorganized team to at least adopt some helpful practises not specific to Agile -- but usually practises unique to Agile methodologies are of little positive consequence, and consume a lot of development resources.  (It's Steve Yegge's argument -- the standard existing practises are the chili, and Agile is the dog poo.)

I'm disappointed to by the tenor of the chapter.  It seems that the quality of published writing here is on par with the typical hand waving or emoting of a blog.  I'd expect a higher standard where the authors actually make an effort to provide evidence (beyond any inbred citations to other similar works) and hard analysis -- especially real differential quantitative analysis of actual projects.  That sounds harsh, but if the authors and publishers expect me to pay real coin for paper, the quality better exceed the bits I get from the net for free.  I see little here except some nebulous suggestions consisting of mostly standard practises better illuminated elsewhere, wrapped in a current fad of great cost and little marginal value, and no mention of appropriate context for their application.

The only positive effect of Agile I've observed is that it encourages engineers to provide more accurate estimation data at task granularity.  I think that's just because index cards appeal to the typical engineers sense of aesthetic organization.  Or maybe just the act of writing a number down gives them pause.

I'm willing to be proven wrong -- with emphasis on *proven*.  I've worked in megascale projects with hundreds or thousands of developers, millions of lines of code, and twenty years or better of legacy, that worked only because of BDUF, and small projects with tens of developers and moderate complexity that were failing because of Agile practises.  Nothing I read in that chapter would have improved on the former, or helped in the latter.

I'm always a little amused whenever people demand proof for agile. What proof did we require to do waterfall, or any other methodology, for that matter? What proof did you require before you starting doing what you're doing now? Precious little, I suspect.

And even if we were easily able to provide "proof", how well would it translate to someone else's project and environment?

There's never going to be proof, let's just face it. If you don't believe in agile, don't do it. If you think it might help you, what have you got to lose by trying it?

Agile may borrow heavily from existing practice, I neither know nor care. My experience, however, is that I have never seen those "existing practices" actually used in the real world unless they were introduced via a move to agile methods.

YMMV

I'm always a little amused whenever people...

use this forum to pander their own blogs and books. Isn't this whole topic technically spam?!?!?

It's not spam by any definition I accept. It is on-topic for the forum and a follow-up to a previous discussion.

It's a little optimistic in what will and will not be 'revealed' as a bug.

Also, the biggest problem in production systems is NOT "writing bug-free code" -- though that is very difficult.  "Bug-free code" being code that does what the programmer intended the code to do.  That's code that has no off-by-one errors, no memory leaks, no un-initialized pointers, no unintended disasterous side-effects.

The biggest problem is when that code hits the customer, and then hits the real world.

"It ain't what you don't know, it's what you DO know that ain't so."  Code that works perfectly, technically meets the written requirements, yet does not meet the customer's needs is the biggest problem.  That, and when the actual use patterns of the real world trigger code-state conditions never seen nor planned for in the lab.

Brian Marick has an excellent example of scenario testing, where he demonstrates that agile, TDD, blah blah blah, often won't help in finding the really nasty real-life bugs:

"A customer named Brian hires a car for a three-day business trip. Incidentally, this gives him enough rental points to reach Preferred status. Midway through the rental, he extends it for another week. Several days later, he calls to report that his car has been stolen. He insists that the Preferred benefit of on-site replacement applies, even though he was not Preferred at the start of the rental. A new car is delivered to him. Two days after that, he calls to report that the "stolen" car has been found. It turns out he'd mis-remembered where he'd parked it. He wants one of the cars picked up and the appropriate transaction closed. Oh, and one other thing: the way he discovered the mislaid car was by backing into it with its replacement, so they're both damaged..."

> often won't help in finding the really nasty real-life bugs

That's why we have this thing called system testing. In certain classes of application system may not be necessary and TDD will be sufficient, but these books never make allowance for subtlety. It's all hammer and nail.

No unit test will test interaction and scale. It just tests the unit. Systems are much more interesting and they often, depending on the product, take a long time and require a lot of configuration. If your iteration length is a month and your your system test suite takes a week then developers can't do it.

It's not the ideas in the chapter are wrong. They are just incomplete.

"I'm always a little amused whenever people demand proof for agile."

For a discussion at usual level of discourse found in blogs or forums, the level of discourse in the sample chapter suffices.  For a commercially published book, it does not.

The burden here isn't onerous.  I'm suggesting case studies, specific examples, or even hypothetical quantitative analysis that demonstrates the advantages claimed by the authors of this chapter.

I'm not asking for scientific proof, or an academic level of discourse.  Fred Brooks or Robert Glass are good examples of the level of discourse I'd expect if I'm paying for it.

As it stands, this is a book I'd pick up, thumb through, read several random passages, and then return it to the bookshelf.  And my reading time is at a greater premium than my money.

"My experience, however, is that I have never seen those "existing practices" actually used in the real world unless they were introduced via a move to agile methods."

That's fine, and the fact that my personal experience doesn't agree with your personal experience is why Internet discussions on such topics end up being parallel monologues.  I did say that Agile functions as a mechanism for *some* organizations to adopt some high value pre-existing practises.  That's entirely consistent with your observation, as long as your "I have never seen" qualification stays in place -- my 'some' appears to be the entirety of your experience.  No disagreement.

It's also consistent with the observation that Agile proponents try to lay claim to benefits that don't belong specifically to Agile methodologies.  It's great that Agile's popularity cause such practises to be adopted, in the way that even programmers who didn't grok OOM were finally at least doing structured decomposition using OOPL.  But given that organizational and engineering context vary dramatically, slavish adoption of any methodology without a critical examination of its features and their applicability is misguided.

If the authors of the book expect the public to spend their money and time, they'd better provide some value which sets the book apart from a discussion on the Internet.  Reading through the sample chapter felt like reading a long blog post on the subject.

Part of the problem is the dynamics of the technology cycle.  To get any attention, hyperbolic claims must be made for incremental advances.  Hypothetically and parenthetically it goes like this:

A: "X is great!"

B: "We already have Y which does the same thing as X."

A: "X is better than Y!"

B: "Yeah, but I already know Y, and have a lot of investment there."

A: "Y is a problem, X is the solution!"

It seems almost necessary to demonize the previous generation of technology.  In truth, X is usually only an incremental improvement on Y, but it requires a complete tech refresh to adopt X and abandon Y.  X is typically 1-2% new (and often only incrementally original) ideas, and 98-99% a repackaging of Y with terminology changes (old wine into new bottles).  Hence Ruby v Java, et cetera ad nauseum.

Agile is more deceptive than the norm in going all the way back to BDUF/Waterfall for its Y, and ignoring all the intervening advances in methodology and practise.  The Agile specific parts present concerns in risk management, requirements management, churn and dependency management.

But X being fashionable, it does excel at selling Y to organizations which currently have nothing in its stead.  Agile is better than utter chaos, but just about anything can be substituted for 'Agile' in that statement.

I would spend time and money for a well crafted book on Agile methodology that possessed all the attributes mentioned above.  Judging by the sample chapter, this isn't that book.

<quote>
Brian Marick has an excellent example of scenario testing, where he demonstrates that agile, TDD, blah blah blah, often won't help in finding the really nasty real-life bugs:
</quote>

I'm pretty sure Mr Marick is speaking to those that believe that ONLY doing TDD is sufficient to test a system. There might be those that think that way, but then, there are those that think it's ok to release software without any testing at all.

There's nothing in what Marick says that can't be done in an agile project. Or using any other methodology, for that matter.

Hi, Borealis,

As I'm sure you're aware, rigorous studies of software development methods are hard to come by.  I'm not aware of any studies that cover the precise combination of practices we suggest.

However, test-driven development alone has received some attention.  Laurie Williams found an 18% improvement in functional test pass rate at a cost of 16% more time (read the paper for caveats on the time issue).

http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf

Muller & Padberg present an economic model of TDD here:

http://www.ipd.uka.de/mitarbeiter/muellerm/publications/edser03.pdf

Janzen & Saiedian summarize TDD research in "Test-Driven Development: Concepts, Taxonomy, and Future Direction", IEEE Computer, September 2005.  It's not freely available online, but slides 17 & 18 of http://wwwse.inf.tu-dresden.de/wiki/images/a/af/PRO-AndreasAugustin.pdf show some pertinent tables.

By the way, this isn't a sample of our book--it's a pre-publication review copy.  We will be making further improvements based on reviewers' comments.  You can find additional material at http://www.jamesshore.com/Agile-Book/ .

Bruce,

>> There's nothing in what Marick says that can't be done in an agile project. Or using any other methodology, for that matter. <<

Exactly. What I'm trying to say with this example is that XP (or agile, for that matter) is not fundamentally about bug prevention, or at least not more so than various waterfall methodologies.

The essay in question is trying to link XP and TDD to better bug prevention. And I just don't buy it.

What I do like is that the authors are presenting the essay for analysis and critiques, and I think they will end up with a much better book because of it. It does take some courage to do this.

"As I'm sure you're aware, rigorous studies of software development methods are hard to come by."

Certainly true.

Which is why I suggest good case studies would suffice.  I'm only suggesting the rigor of a better trade publication, not an academic or scientific article.

From the first linked article:

"Lastly, the programmers which followed a waterfall-like process often did not write the required automated test cases after completing their code, which might be indicative of the tendency among practitioners toward inadequate testing."

TDD seems to be a contradictory inclusion in Agile methodologies, as stable requirements are required on which to base the tests.  I suppose that the requirements addressed in any iteration are stable *within* that iteration, which just moves the objection to one of increasing the general work churn created by an adaptive iterative approach.

What is the incremental value of TDD to an organization that already does automated unit testing?  It's unfair for all the benefits to accrue to Agile methodologies by assuming as a starting point an organization that doesn't do much beyond scratch testing by developers.

Again, Agile should only claim benefits added to existing practise (setting aside again its benefit in *introducing* these practises to some organizations).

"By the way, this isn't a sample of our book--it's a pre-publication review copy."

At this time, are the majority of chapters a broad outline of the material you intend to present in the published book?

If "no bugs" is your reward for following the magic methodology du jour, then a tiny QA department who's job consists of keeping the developers honest is management's reward.

QA's job isn't to create quality - it's to find the quality that the developers created in the first place. All these great new methodologies may keep the bug count down, but the testing department's job doesn't go away.

And besides, let's be honest here: either you're writing toy applications, or QA's job will be to find the 1 edge case you missed, even if you think you have 100% coverage and have automated all the integration and other system testing.

Any methodology that starts out "first, assume a programmer who actually has 100% perfect testing but was previously held back by evil management's bad methodology" is doomed to failure.

Hi, Borealis,

"TDD seems to be a contradictory inclusion in Agile methodologies, as stable requirements are required on which to base the tests."

I think you misunderstand TDD here.  TDD doesn't require stable requirements because TDD is an iterative cycle with very small steps.  Each TDD cycle takes under five minutes.  We haven't written about TDD in our book yet, but I summarized TDD in my blog: http://www.jamesshore.com/Blog/Red-Green-Refactor.html

Granted, if your requirements are changing every hour, you have a problem!  But it won't be TDD causing the pain.

"What is the incremental value of TDD to an organization that already does automated unit testing?"

I don't know.  My experience is that post-hoc unit tests are harder to write because the code isn't designed for testability.  Also, I would predict that fewer tests would be written due to the drudgery of writing tests for something that programmers consider "done".  Williams' experience with developers not writing automated tests, even when explicitly directed to do so, didn't surprise me.

"At this time, are the majority of chapters a broad outline of the material you intend to present in the published book?"

No, there are two more parts to the book that will be completely different.  Part I ("Getting Started") discusses when, how, and if you should adopt XP, as well as the distinction between "XP" and "agile".  Part III ("Mastering Agility") discusses how to make trade-offs and customize your process.

We also have yet to write a few more chapters in Part II.  ("No Bugs", by the way, is not a chapter.  It's a section in the "Managing Code" chapter.)

>> And I just don't buy it. <<

What I meant to say here was that I might buy it, but the essay doesn't convince me at all.

>> It's not the ideas in the chapter are wrong. They are just incomplete. <<

Agreed. And also unconvincing, partly for this reason.

The article is simplistic, suffers from tunnel vision, and is generally unhelpful. It feels a little like you are peddling snake oil.

Please spend more time reading Steve McConnell's Code Complete: http://www.cc2e.com/. He has a very good section on quality in software development. It's broad, surveys the whole state of the field instead of one or two themes, and has a great bibliography. Then when you have read, analysed, contemplated, and responded to books, essays and research listed within, perhaps you'll be ready to rewrite your chapter.

"The essay in question is trying to link XP and TDD to better bug prevention. And I just don't buy it... What I meant to say here was that I might buy it, but the essay doesn't convince me at all."

What would it take to convince you?

Another thing I'm hoping you can help me understand is why readers are focusing on TDD.  Several people have said that TDD is insufficient to achieve these results.  I completely agree!  I'm confused because we spend most of our time talking about practices other than TDD.  Two areas we've been dinged on are requirements and system testing, both of which cover. ("Coding isn't your only source of defects..."; "[this] only prevents bugs you expect...")

On the "lack of detail" criticism, are you considering that this section is part of a larger book?  The purpose of this section is to provide an overview--to show how practices described in the rest of the book tie together to work towards the 'no bugs' ideal.  The "Ally" boxes in the right margin are cross-references; each section listed goes into much more detail.  You can find most of them at http://www.jamesshore.com/Agile-Book .  Also, Part I (not yet written) will talk about applicability and adoption issues, including cultural challenges.

Thanks for your help!

> What would it take to convince you?

For me, an admission that even TDD and XP can help but aren't perfect would enhance credibility - I know for a fact that no single methodology can survive every possible screw-up, and I know for a fact that even programming gods have days where they spend hours tracking down a misplaced semicolon that a newbie would get right, therefore I know that the QA dapartment needs a bug tracking database.

Anyone who claims otherwise is peddling snake oil.

Even if they can genuinely offer a methodology that will prevent 9999 bugs in 10000 ever getting off a developer's machine (and I'm being very generous there), those last few bugs can be important.


The phrase "work towards the 'no bugs' ideal" is also very very different to the assertion that no dedicated testing phase is necessary. Working towards an ideal is good, but denial of basic reality is not actually going to help. If trying to get to the ideal manages to get you 90% of the way then you've got a great plan but the dedicated testing phase will be much quicker and easier, but it's still necessary to cover that last 10%.

>> What would it take to convince you? <<

I'll post my reply in your new thread.

Part 3 of this discussion here:

http://discuss.joelonsoftware.com/default.asp?joel.3.430927.5