The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Does TDD really help decrease the bug count ?

I'm looking at TDD with an interested eyes, and I'm wondering if it really helps in designing better and less buggy software ?

What your experience have been with TDD ?
Kjell
Thursday, March 06, 2008
 
 
Well, I'm not sure if it results in a smaller underlying bug count... but what is interesting is that every feature, tweak, etc that you have there should have a test to go with it.  So you're safer in making changes, etc.
KC Send private email
Thursday, March 06, 2008
 
 
My software still has bugs in it at first run. What's different is that the bugs are more often than not issues of improperly integrated components. The components themselves almost always work exactly as expected, but through some configuration error are not setup to work together correctly. In my experience, these bugs are much easier to find and correct than ones that lurk deep in the bowels of the code-base.

I would say that the bug count has decreased overall, but that's not the standard by which I judge TDD. My overall software quality has dramatically increased, in part because of my ability to change what I need to as the need becomes apparent.
Chris McKenzie Send private email
Thursday, March 06, 2008
 
 
In my opinion it does, but also a lot of the gains you get are from using those unit tests for regression testing. You can refactor or add new features to a component and still verify that all current behavior is working as expected. This can save a lot of time.

Currently I've been working on systems where I'm integrating with a number of third party systems (mostly CMS or ERP). Some of these have been built in a way that makes them very unit test unfriendly (internal classes, sealed, dependencies on HttpContext).

I've pushed these behind my own interfaces so that I can test how my code interacts with these services. However the underlying service that works with the untestable code is where I end up finding a lot of run time issues. This alone has shown me the value of the testing.
Nigel Sampson Send private email
Thursday, March 06, 2008
 
 
Actually, TDD isn't really a quality focused practice. Or, at least, that's not it's only focus.

The companion goal of TDD, and this is where I feel it makes it's most significant contribution, is design. TDD is a design practice. By writing the tests first, you're forced to think a little bit more about the design of your components. One thing you'll almost certainly notice with TDD is that a lot of the "chrome" you'd normally add to a class just won't get done.

As far as reducing defect counts, if you have a disciplined, entrenched unit testing tradition in your team you'll probably see a big reduction in defects regardless of whether you test first or last. You may get more by doing full TDD, but it won't be orders of magnitude. This is why I say TDD is more of a design practice than anything else.
Bruce
Thursday, March 06, 2008
 
 
I'm too lazy to go find it, but there was a study compairing test-first and test-last, and the test-last stratergy resulted in better code quality.  I don't remember if there was an effect on the time that was taken.

The only problem with most of the studies on development methodolgy that I have read is that they have used students.  I'm not sure if there is any evidence that students are predictive of general programmers.

Regardless, TDD is a way of creating a detailed design.  In that regard, it improves software.  I think that there are more efficient methods of creating such a design, so I do not think TDD is the most effective strategy.

Thursday, March 06, 2008
 
 
"I think that there are more efficient methods of creating such a design, so I do not think TDD is the most effective strategy. "

Exactly. Agile, TDD, and the like work because they solve specific problems in software development. But these practices are not the only way to solve those same problems and may not be the "best" way to solve those problems. For example, a previous poster talked about how TDD makes you think about your designs up front. But personally, I don't need to use TDD to think more about my design up front. I just do it.

One of the problems (out of many) that team programming helps solve is the issue of programmers sitting around goofing off all day long. But there are easier and cheaper ways of making sure your programmers aren't just sitting around reading JOS. For every single Agile practice you can simply ask yourself what specific problems they are trying to solve. And you quickly see that those same problems can be solved in much easier ways.

So in the end TDD, Agile, and the like really help the most for unskilled, unmotivated programmers. And it is actually much easier just to hack these people from the team than to force them to pair up with someone else or institute some TDD policy. For the rest of the programmers who are decent and work hard, TDD and Agile probably won't show you any real improvement because they are getting the job done anyway.

That's my two cents. Take it or leave it.
dude
Thursday, March 06, 2008
 
 
The reference to "team programming" above actually should have said "pair programming".

Sorry.  :)
dude
Thursday, March 06, 2008
 
 
"So in the end TDD, Agile, and the like really help the most for unskilled, unmotivated programmers."

Agreed... This is pretty much true of any set methodology (XP, Waterfall, etc).

I reckon it works something like this:
- Team A who produce constantly awesome product, and conclude that their approach must be the key.
- This approach turns into a methodology which is shoved at Team B (who are an aimless, unproductive mess).
- Things improve for Team B.  Team A think it was The Methodology that fixed Team B, rather than the fact that applying any methodology INITIALLY causes people to focus and think more about what they are doing.
- Team A fixed Team B, so they conclude that they have just found a silver bullet.
- Team A turn to Team C (who are actually very effective) and tell them things would be so much better if they used The Methodology.  When Team C say "No thanks" Team A accuse them of being a bunch of anachronistic old farts, but curiously never ask the question "So how do YOU do things?".
- Team A write several books, and now make all their money from running seminars on The Methodology.
- Poor old Team B now have The Methodology entrenched, which means they go back to not thinking about things, and are once again an aimless, unproductive mess.  They will have to wait for The Next Methodology.  Sadly Team C has been relegated to Stuffy-Stick-In-The-Mud-ville and so nobody asks them, which is shame because Team B might have been helped for a little while.

The morals of the story are: 
- Effective teams will produce good stuff no matter what.
- Crap teams will improve if they are shaken up every so often (this can even help good teams).
- Methodologies are fine, but for crying out loud pick the best one for the situation (Hint: It is not ALWAYS Agile, or ALWAYS Waterfall).
the Melbournian Send private email
Thursday, March 06, 2008
 
 
My experience has been that unit testing helps decrease the bug count, sometimes dramatically so. However I would be surprised if testing first instead would provide a significant added benefit over testing last (provided the tests are written after each functional unit if you're testing last). I am personally not convinced of the purported design benefits of TDD. An iterative development process, code reviews and unit tests before moving to the next section of code provide so much  opportunity for solidfying a design that I don't find the cost and change in habits incurred by TDD to be worthwhile.
triple-dot
Thursday, March 06, 2008
 
 
<quote>
So in the end TDD, Agile, and the like really help the most for unskilled, unmotivated programmers.
</quote>

I respect your opinion but have to note that this is just about the polar opposite of my experience during 7+ years of working with agile teams.

Agile practices are intended, and focused on above average, motivated developers. Using them with the kinds of developers you referred to would be extremely dangerous.

In fact, your position is so far removed from my experience that I have to ask: have you actually seen TDD used? This is a polite question btw, I'm not questioning your experience or ability, just wondering what you might have encountered to lead you to your conclusion.
Bruce
Thursday, March 06, 2008
 
 
Unit testing has given me the confidence to keep tweaking my  code all the time. Perhaps not entirely bug free, but the product that I work on is much more efficient as a result.

Also, since I started doing TDD, I am much more happier about the code that I end up with. I think that alone is worth the time spent.

My 2 c.
-S
Thursday, March 06, 2008
 
 
Not all bugs are equal.

I'm still growing into TDD, I've only used it for a few small projects so far. My experience is that I tend to create fewer syntax errors such as assignments verses equality, but I create more application logic and design errors because I'm trying to get the unit tests working at each stage instead of focusing on a top-down perspective.

Right now, I personally think TDD is best for maintenance style projects where you're just adding or modifying a module. If you're building the whole thing from scratch, the various test-last methodologies seem to work out better.
TheDavid
Thursday, March 06, 2008
 
 
<quote>
My experience is that I tend to create fewer syntax errors such as assignments verses equality, but I create more application logic and design errors because I'm trying to get the unit tests working at each stage instead of focusing on a top-down perspective.
</quote>

Are you using mock objects? If not, you may find you'll be better able to focus on your top down design by using them during your TDD sessions. They'll allow you to "fake" some of the dependencies while you're working on higher level code.
Bruce
Thursday, March 06, 2008
 
 
"The companion goal of TDD, and this is where I feel it makes it's most significant contribution, is design."

Ditto.  I find that when I code with testability in mind, the end product is a better design almost as a by-product.  When I code without concurrent testing during development, I tend to cut corners without realizing it.
Knight Who Says Ni
Friday, March 07, 2008
 
 
TDD may not.  Unit and automated integration testing,  I would say subjectively 'yes', if your group is smart enough to take lessons learned from one project onto the next.

Example: Project 1, new team, writes code, has some unit tests, goes out to QA, gets a lot of bugs because of i18n problems, long strings, weird characters in fields, etc.  Add unit tests to test persisting these to storage, displaying them in the UI, manipulating them, etc.

Move on to Project 2, a wise team will add these "String of Death" tests to their unit tests from the start, catching a whole class of bugs earlier.

This may apply to TDD.  It may get better over time as people gain experience in what to include tests for.
Dan Fleet Send private email
Friday, March 07, 2008
 
 
In my experience sensible code reviews have reduced bug counts greater than any other development process I've seen employed. And if you do them correctly, they are quick, painless, and very inexpensive.
dood mcdoogle
Friday, March 07, 2008
 
 
Designing for testability is important, otherwise people won't write meaningful unit tests. I've seen a suite of unit tests that simply tested that the program crashed on invalid inputs, because valid inputs were too hard to construct (i.e. required a database). Tested! The program raises an error if it runs on a non-existent database, as required! (No, it wasn't a software-focused shop).
PeterR
Saturday, March 08, 2008
 
 
One thing about test last is that it's easy to skimp or skip the test part and just go with the code since it's now written.
cbmc64 Send private email
Saturday, March 08, 2008
 
 
"One thing about test last is that it's easy to skimp or skip the test part and just go with the code since it's now written."

This imo would be a sign of a lack of discipline that will not magically go away merely because you're testing first. I.e. a lazy developer will skip or skimp regardless of when the tests are written.
triple-dot
Saturday, March 08, 2008
 
 
A coworker was involved in a study of TDD vs. not a couple of years ago. A professional (non-student) team that had done traditional waterfall spec-driven work switched to TDD and agile.

If I remember correctly, TDD cost 30% more in initial dev time. However, the bug fix time dropped to almost zero, so it ended up saving over the lifetime of the project.

Guess it all depends on who you ask. ;-)
Chris Tavares Send private email
Saturday, March 08, 2008
 
 
" - Team A who produce constantly awesome product, and conclude that their approach must be the key.
- This approach turns into a methodology which is shoved at Team B (who are an aimless, unproductive mess).
- Things improve for Team B.  Team A think it was The Methodology that fixed Team B, rather than the fact that applying any methodology INITIALLY causes people to focus and think more about what they are doing.
..."

This is what the life cycle of a silver bullet described:
http://www.stsc.hill.af.mil/crosstalk/2003/07/Sheard.html

TDD is just another name for automatically testing your code. I worked in a team that did this before the whole TDD phrase was invented. It really worked. We were able to make changes in a hugely complex codebase and insure that everything still worked.

But you should not just put unit tests in your regression tester. You should also put your integretion tests there.

The hard part is keeping the discipline to code these tests and execute them. This is especially true if you do not have *full support* from your team and management.
Onno Hovers
Sunday, March 09, 2008
 
 
Some more things about automatic testing.

I use the term automatic testing because I do not like the new overhyped term TDD. The agile crowd did not invent this. They just hyped it.

When doing automatic testing you should make sure that your *whole codebase* is covered. If you don't, you still can't guarantee that your code base is mostly bug-free.
And the "broken windows syndrom" will kick in.

The tests should not just cover the good-weather behavior. You should also cover at least the most common errors. It is in the error handling where bugs usually lurk.

Automatic regression testing is no substitute for real testing. So you should also do that before doing a release. But you won't find as many bugs, and you can release earlier and with more confidence.

There are some things that you can never test properly with automatic regression testing. User interfaces and hardware are examples of this. The trouble is that users and hardware do not always behave as you assume.
Onno Hovers
Sunday, March 09, 2008
 
 
"I use the term automatic testing because I do not like the new overhyped term TDD. The agile crowd did not invent this. They just hyped it."

Automated testing and TDD are fairly different beasts...

Automated testing/unit testing has been around since dinosaurs roamed the earth...  Agile talks a lot about it, but then so does BDUF.  No news there...

TDD involves writing the tests _before_ you write the code.  This is a bit more specialised to agile.

Almost everyone agrees that unit testing (to at least a certain extent) is a good thing.  The jury is still out on TDD however...
the Melbournian Send private email
Sunday, March 09, 2008
 
 
And agile folks (at least the honest ones) never claimed to have invented TDD either.

For what it's worth, since starting doing TDD, I've found an interesting personal change. It keeps my velocity up.

Before doing TDD, depending on how complicated the thing was I was working on, I could go a couple of days between having running code. The longer I worked on something like that, the slower I got. I had to think harder and harder - does this change fit? Will this work based on what I did over there?

With the TDD approach, I don't have to think that hard. My code is always running in some form. The tests tell me what I broke. And I don't have to keep days of coding details in my head anymore.

Even if it doesn't make me more productive overall (and I think it does) the change in attitude leaves me a lot less stressed at the end of the day.
Chris Tavares Send private email
Tuesday, March 11, 2008
 
 
Here is a link to an interesting comparison between Ron Jeffries (TDD Guru), and Peter Norvig (AI Guru), exploring how to write a suduku solver.

http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html

Not sure if TDD reduced bug count, but it certainly didn't lead to a solution.
Pedro Estrada Send private email
Tuesday, March 11, 2008
 
 
Hi All,

I just signed up with the forum and enjoyed reading everyone views and experience here. I have been doing Oracle/Java programming for the past 4 years and have acquired increased interest in TDD and agile practices.

My current employer doesn't follow any methodology and it's hard for me to focus when I want to follow good practices..

Just curious, how to find out what companies are using these practices in North Dallas area?

In my next job, I really want to end up somewhere, where I am able to appy these practices that I read in books. It seems to me that most of the companies bloat the job description and these practices exist in books only. It's sad.

Wednesday, March 12, 2008
 
 
"Automated testing and TDD are fairly different beasts..."

Actually, despite its name TDD is not about testing at  all. The point of TDD is not to write tests, but to specify the desired behavior of software components in a form that can be automatized and verified against the actual code to see if the code satisfies those requirements.

Since unit tests were already the most appropriate tool for that they were first used, and the practice became known as TDD, which leads to many misunderstandings; hence a new term was invented: http://en.wikipedia.org/wiki/Behavior_driven_development
Berislav Lopac Send private email
Sunday, March 16, 2008
 
 
TDD doesn't really decrease bugs - because bugs are caused by mistakes in programming, not by writing tests.

What TDD does is increase your confidence in making changes to the software, because you typically have a battery of tests to run against the code.  So breaking changes are caught earlier, and coders are more confident in making breaking changes because the problems can be identified easier.  However, TDD is a lot of extra code to write and it would be hard to know if overall output is increased - however I suspect that overall quality is increased, and as mentioned by another poster, coding velocity can increase because you are always running stuff.

On the other side issue of methodologies.  I've been in the software development game for about 14 years now and I've seen methodology fads come and go.  I've seen the bright young salesman selling their particular way of doing things, and I've sat through their pitches.

My take: All methodologies have some good ideas.  Very few (perhaps none?) methodologies have a good fit with the people and the project.  Good developers need some structure and tools to work in, but no amount of structure and tools is going to help aimless developers with no clue about how to write code or what code they should be writing.  Their should be structure to a project, but it's best to cherry pick ideas and put them together based on your team/resources/project/timeframe/geographical spread.

The cynic in me says this: a lot of methodologies are published by large software consultancies, and are really aimed at getting large amounts of underskilled and inexperienced developers to write projects.  We all are aware of the companies that hire graduates, turn them into clones with flashy business cards at 22 and then charge them out for $200 / hr.  These same companies love methodologies, perhaps because they can sell the idea that you don't need experienced talent to do projects if you've got a good methodology, or perhaps because it really is the only way to get things done.  But the evil genius who came up with 'pair programming' must have been promoted when he worked out how to get $400/hr from the helpless target without having to increase the number of desks in a room.
Bruce Chapman Send private email
Wednesday, March 19, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz