The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
Fog Creek Copilot

The Old Forum

Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

Am I the only dev who doesn't like unit tests?

I think that in most of cases they are useless. You spend so much time writing tests that won't catch anything new that I wouldn't catch by running and debugging the software myself. Ok, in a few specific cases they make sense, but my company has a code coverage policy to aim at unit test 100% of code. I think this is lame and a big waste of time. However I often see people talking about this kind of policy as a good thing to look for in a company - it's even in the Joel test, isn't it? Ah, don't get me wrong, I like to test my code, my code is usually pretty solid, it's just that I test it by running and debugging the code, entering the different kinds of inputs, trying to break it, seeing how it behaves, etc. Unit tests demand so much time for the benefit of...? I really don't know!
Saturday, January 17, 2009
The most straightforward benefit is that they protect you from introducing new bugs in the future when you update the existing code base.
Saturday, January 17, 2009
But when you're writing code you have a good idea on what impact it can cause to other parts of the software. At least in my current project I do. And if it may cause any impact, of course I'd test that. I think test is important, I just don't like (to be honest I hate) writing unit tests.
Saturday, January 17, 2009
"But when you're writing code you have a good idea on what impact it can cause to other parts of the software"

Not if you're the new guy. Not if you're a contractor. Not if the changes you make are substantial and touch many different areas.

The point is that you can make big sweeping changes without having to memory-map all of the stuff you may have just broken. If you have 100% code coverage and the tests are well written you'll know right away exactly what you broke, long before it mistakenly gets shipped to the customer and you get called in for an emergency bug fix. People forget stuff. People make mistakes. People make wrong assumptions. Your unit tests don't.

Saturday, January 17, 2009
"you have a _good idea_ on what impact it can cause to other parts of the software."

That isn't the same as "I know every single possible side effect this can have, and I can guarantee there are no undersirable side effects".  Big difference.

(And so I don't feel like a complete hypocrit, I don't have many automated test cases -- too much time, have to stub too many things :(  -- and, I've been burned many times by unintended consequences, even though I also test my code by stepping through and watching it execute)
Saturday, January 17, 2009
"I test it by running and debugging the code, entering the different kinds of inputs, trying to break it, seeing how it behaves, etc."

People are clever, but wildly inconsistent. Computers are stupid, but perfectly consistent. In so far as your hand testing is mechanical, then it's a poor use of your time, because people don't do mechanical as well as computers do. We forget, get distracted, get interrupted, and just plain overlook stuff. Get the computer to do the mechanical stuff, while you work on being clever. The trouble you may be having is that people tend to be lazy, and stepping through a program by hand feels like work, but is much easier then tackling new code. Typing is easy, thinking is hard. Thinking generally pays off in the long run though.

In an ideal world you'd do both. You'd step through your code to make sure you correctly understand what it is going, and you' write unit tests to make sure that you'll never get bitten by the same mistake twice.
Charles E. Grant Send private email
Saturday, January 17, 2009
I find (most) unit tests fairly pointless.

Integration tests are much more valuable.
Saturday, January 17, 2009
I actually find unit tests also serve the purpose of being an illustrative use of classes -- it shows how to make them, prod them, what results you expect.

I write the tests afterwards (I'm working in C++, so writing the test and "having it fail" doesn't really work. I just don't get an executable), but I do still write them.

Being able to run the tests again after changes to make sure nothing unexpected has broken is very useful if you're doing big refactorings.
Katie Lucas
Saturday, January 17, 2009
@SomeFool, I was curious about how big is the project you are working on?
Johnny Jiang Send private email
Saturday, January 17, 2009

Unit tests help to spot errors in low-level algorithms. When I was re-writing and upgrading a CDO pricing framework, I was unit-testing on the level of individual modules (like the calculation of conditional loss distribution... for those in the know) and also testing the whole system by giving it different inputs and checking if the prices were reasonable. Had I been doing only the latter, I'd spend much more time chasing stupid bugs deep in the code. Effectively, I'd do unit testing only once and by hand.


"That isn't the same as "I know every single possible side effect this can have, and I can guarantee there are no undersirable side effects". "

Let's not kid ourselves, unit tests are almost never that perfect.

@Charles Grant

"In so far as your hand testing is mechanical, then it's a poor use of your time, because people don't do mechanical as well as computers do."

Unfortunately, writing unit tests is also often quite mechanical :(
quant dev Send private email
Saturday, January 17, 2009
From the above replies, people apparently think that "unit tests" are synonymous with "automated tests" or "regression tests" (and, I don't think they are).

I rarely like unit tests, in fact: I prefer to get coverage (of the code and of the functionality) by writing *automated integration tests* and/or *automated system tests*.

*Unit* tests are perhaps especially useful when there are multiple teams, and:

 - You can't do an integration test of A with B, because B hasn't been written yet

 - And/or, you want to unit test A as a sanity check before we start the [more expensive] integration testing of it with component B

 - And/or, A is supposed to be a reusable component (so it isn't sufficient to test its behaviour within the context of only one system)
Christopher Wells Send private email
Saturday, January 17, 2009
There's two ways of seeing unit tests:

1. Unit tests are horrible, they make you write code which is a waste of time as I can test this other ways much faster
2. Unit tests are teh sex and you should write tests for everything in your code

Neither of those is right. Obviously there are cases when you should be writing tests for everything (critical systems such as control software for planes or power stations). But beyond that unit tests aren't the best suited form of testing for everything. I've written some unit tests for the backend of one of my applications. They're great to have as it makes it easier to see if I break something in some of my algorithms. However, testing the controllers and views would be a major pain and are things I can test fairly easily by hand.

Unit tests have their place, but it depends on the type of project as to how much you should write. Personally I hate writing tests, but I can see their benefit in places.
Martin Pilkington Send private email
Saturday, January 17, 2009
It's not in the Joel Test, no.
Joel Spolsky Send private email
Saturday, January 17, 2009
If you use Python and don't want to write long unit tests, you can get away with doctests (which are basically tests in your comments).
Victor the Python Artist Send private email
Saturday, January 17, 2009

No you are absolutely not alone. In the team I work in, only the guy who introduced the unit test framework raves about it, while everyone else who have voiced their opinions hate it. Luckily, the manager has no opinion about it.
Saturday, January 17, 2009
IMHO, unit tests are useful, BUT many (most?) times people take it way to far.

I have seen managers that expect insane unit test coverage (70% for example).  It's great to test major areas and consumable APIs, but testing all the minutiae is not a good use of skilled and highly educated resoureces.  It would be far more efficient to hire lower cost QA help and along with that get some integration and application level testing too.

Not to mention if you have to write mock classes or mock functions you don't really  have a true test.  You have a sort-of-test.  The more mocks you write the more worthless your tests are.  I have seen teams forced to write unit tests in such volume to cover ever little thing that at least 50% of the code had to mock code.  They ended up identifying non-existing bugs and never exercising modules how they would actually be used so many bugs slipped through.

Finally, many times you have to add methods/functions that were never needed to your code to allow for testing.  That is a major sign that your scope of testing is too low-level.

Saturday, January 17, 2009
I think the real benefit of unit tests are when you take it to the next level and use a test first or test driven approach.  I like to think of writing a test as detailing a technical use case for a component and using that to drive the implementation.

Taking that approach has helped me write better, i.e. simpler and more modular code as I only write the code I need to get the test to pass and I need to implement much clearer boundaries between the modules/components of the system (you also realise what a boon dependency injection is to software development).  I also find that I do less refactoring as my code matches the intent of the requirement more closely on the first pass.

I have also coached a number of teams (e.g. a technology dept. of around 100 developers) on TDD and those that adopted it, did see an improvement in the quality of code delivered.

I think a lot of people come to hate unit tests, as applying unit tests after writing code, well, sucks.  It's hard, frustrating and feels pointless.  If you hate writing unit tests, give the TDD (test first) approach a go for a sprint/iteration (you are doing agile development, right :-) and if you still hate it, fair enough, but the number of people I've seen stick with it is really surprising.
Michael Barker Send private email
Sunday, January 18, 2009
"It's not in the Joel Test, no."

You had to look it up, didn't you? Come can admit it to us....

Sunday, January 18, 2009
"I think the real benefit of unit tests are when you take it to the next level and use a test first or test driven approach."

Don't get me wrong, I use unit tests and I would use them even if it wasn't required, but after years of unit testing I'm less likely to believe in the test driven approach.  They are just as likely to suffer from feature skew as any code.  This merely multiplies the lines of code you have to write and rewrite and rewrite and rewrite.

Let me also note that I am constantly amazed at work how minor changes during development so often "break" builds because of unit test failures.  This is due to the fact that unit tests are, by definition, focused on local scope.  Therefore one is forced to make certain assumptions which often break.  This becomes even more murky when you start adding mock objects.  It may not seem bad when working on a small project, but when you start to scale up to large projects the problem can grow exponentially.

I have personally switched to the unit test school of thought that says if unit tests require special code to work like mock objects/code or require the adding of test support functions to the real code then they are not unit tests.  They are similar to integration tests, but you are integrating with, and via, code that will never be used by the customer and therefore their exercise is not only useless it is dangerous because the results are often meaningless.

So minimal unit tests that require no special shoehorning is fine with me, but a serious and professional development house needs to hire professional QA (that can also code).
Sunday, January 18, 2009
My worry is that many times you have to break certain rules of good coding to get 100% coverage with unit tests.

Many times you have to add accessors to private data so that you can create a test environments.  Sure you can hide these secret accessors form your class' consumer by using interfaces, but it is still there and depending on your language still accessible one way or another.

Test, but don't break good coding practices!
Pretty Boy
Sunday, January 18, 2009
Mock objects and test hooks are still argued over to this day.  The primary argument against them is that the more test code you inject and the more fake objects you add the further you get from what you will actually be shipping.

These types of tests should be classified with other sorts of instrumented build tests not unit testing.  Instrumented code testing requires a fair amount of managing and tweaking.
Mr. Medical Systems
Sunday, January 18, 2009
I used to hate unit-tests but now I really can see their value.
For me the most important thing is confidence. Confidence that the code works as thought.
The code might not be according to specs .. that's something else but at least it does what you expect it to do.
I know :senior developers should not need to write unit-tests as they are one-with-the-compiler  but as a not-so-senior developer I like to have another safety layer.
And yes I do not like debugging very much. I would rather control that my code works as I think than go all deep down to the framework to get a value passed.
For those of you saying that it's a waste of time I think you should remember that using a compiled language such as  C,C++,Java, ,NET is a waste of time due to declaring values and type checking .. but most of C++,Java programmers would not go back to PHP unless their lives depended on it.
Mike L. Send private email
Sunday, January 18, 2009
I think the real answer is "it depends".

For example if you are developing a library of classes that you will release to 3rd party developers then unit tests are probably a good idea to ensure future releases do not break the contract.

As an opposite example, we have a product with built-in scripting language. Instead of writing unit tests for classes that implement the script interpreter we simply have a bunch of self-testing scripts which are run using the interpreter and the results of the execution are verified. This is much better approach since it ensures that we do not break the external contract with end-user - the script language behavior and still allows us to heavily restructure the interpreter internally (for example add a JIT compiler), which would render class-based unit tests useless as most of the tests would break.

So I believe automated tests are important to have but it is not always unit tests that are the preferable form of testing.
Sunday, January 18, 2009
I don't think everyone is saying they are a waste of time.  A lot of people are saying you have to chose your unit testing philosophy carefully.

I agree with those that are pointing out how many unit tests are NOT true unit tests.  I have found that a lot of people do NOT understand the basic boundary of what is and what is not a unit test.
Sunday, January 18, 2009
As always, the truth lies somewhere in the middle.

Unit tests work well for algorithmy sort of things, not so well for (eg. user interface code).

eg. In my program I have unit tests for the cryptographic parts (algorithms with standard test vectors) and a "verify" function for the main data types which walks the trees and gets objects to test their internal consistencies. In test builds the "verify" function is called after every major edit to the data.

As I get better at OO programming, the "verify" function hardly ever finds a problem but it's been a good discipline while I was learning.
Jimmy Jones
Sunday, January 18, 2009
There is a special problem with unit tests that wasn't mentioned so far: How do you test the unit test code? Or is it always assumed as bug-free?
Sunday, January 18, 2009
"The most straightforward benefit is that they protect you from introducing new bugs in the future when you update the existing code base."

I object.

Unit tests DO NOT protect you from introducing new bugs in the future when you update the existing code base.

Unit tests DO help a developer know that their code, as tested by the [prewritten] unit tests, returns the same/expected results when updating the existing code base.
Sunday, January 18, 2009
"How do you test the unit test code? Or is it always assumed as bug-free?"

This is why mock objects, mock code, behind the scenes hooks, and instrumenting can be, at best, a false sense of security or, at worst, dangerously inaccurate.  That is why experienced programmers have a problem with over hyping the pay off of any type of testing.  This is especially true for developers where unit tests are just one of many tasks they have to juggle.

So, minimally intrusive unit tests are fine, BUT the belief that unit tests are god's gift is intellectually dishonest.  And an attempt at 100% code coverage via unit tests is foolhardy.

The fad of pushing more and more testing onto developers was started to save money.  Yet in the end it can cost more than you save.  Yes, developers should do some MINIMAL unit testing, but once you start down the over hyped path of exotics you developers have to juggle more and more work beyond designing, implementing, and bug fixing.

I will argue that hiring QA personnel to focus on writing tests, verifying the correctness of tests, running tests, managing tests, creating all the other categories of tests, etc, etc, etc is far more cost effective than pushing more and more testing onto developers.

Again, I am NOT saying unit testing is evil.  Just that, from my experience, far too many managers and programmers don’t understand the dos and don’ts of developer testing.
Sunday, January 18, 2009
No, you aren't the only dev who doesn't like unit tests. The complex interactions between systems are hard to test. Usually, the only things easy to test are those that you really don't need to test anyway. Finally, programmers who are prone to making a lot of errors wrapping crappy tests.

Sunday, January 18, 2009
Once upon a time, I tried to develop unit tests for some code running on a JBoss server. There was a mishmash of crap dependencies, with various calls to Hibernate (a framework I now loathe), lots of wacky static method calls, the list goes on. I barely got the thing running, and it took forever to initialize (well, like 15 seconds, which is insane for a unit test). And then it broke because someone tweaked the ant build file. And because I'm already bald, well, it's ok, I had no hair to pull out.

The development of this unit test gave me two observations:

1. Mocking DB requests are useless, especially with a persistence framework like Hibernate.
2. Our internal design sucked balls, and that was 99% of the reason developing a unit test was painful.

In the end, if you make testing easy, it will be fun, and you will write less code, and all will be well in the universe. The trick is making it easy.

Here's a few things I've done to start liking testing a bit more:

1. Get the testing pain out of the way. Use a CI server (Hudson) to drive the slow-ass tests. Keep them running, but don't piss your fellow developers off by having these kinds of tests running at any point during a standard build procedure. I once knew this guy who forced a maven build to run an insane amount of nitpicky testing whenever you tried to *build the main component*. In the end, I now spit upon his name whenever I see his handiwork on the screen.

2. Start writing things in Scala. (We're a Java shop, please don't hold it against me.) You could use Groovy, or Ruby here. I just like Scala, because I just understand it at a performance level. But the big win is a massive SLOC reduction. Less code wins.

3. Get as much stuff covered with as few lines of test code as possible. Often, this runs counter to the whole "unit testing" philosophy, but I don't care. Often, this drives me to re-architect code: change how exception handling happens, for example, because you realize that all this error handling happens in the middle of a call stack. It's a way of code spelunking. Over time, I start to refine the conventions I keep, and then break the tests up a bit more.

In my experience, testing often just gets the same short stick that documentation gets. I'm not simply talking about code comments, but just communication of "what got done". More code monkeys go the route of writing something like "crap happened" on a trouble ticket, and then making a ton of changes with no described rationale. Or they go into useless rambles where you end up skipping over the really important 5th sentence in the 4th paragraph of their diatribe.

Testing well is in some way like writing well. Elegant testing is fantastic, but it often takes several revisions for you to get there. The whole "unit tests are always great" nonsense reminds me of coders that would say crap like "for every line of code you should have a comment line". ... huh? what?

Tristan Juricek Send private email
Sunday, January 18, 2009
I agree that more often than not, they are pretty much useless.  I'd take a good QA tester any day.
Jussi Jumppanen Send private email
Sunday, January 18, 2009
I used to be a believer.  There were some pretty powerful experiences I had where my code did seem to improve by writing my tests first.  So I thought I should write tests for everything.

Now I've found a balance.  Unit tests are useful when I am plodding through the development of a complex algorithm and I want to automate some checks.  An algorithm that has many conditions that can get pretty complex with many special cases will get a unit test.  I'll write the test first, just to be able to automate knowing when it's done.  I also write unit tests for exploratory code.  Like for answering "What will this regular expression do?"  The test checks the output, and I can keep experimenting with different regexes until I find one that passes.  After I finish the algorithm, though, I usually throw the tests away.

But now that I think about it, that might not be unit testing.
Eric Normand Send private email
Monday, January 19, 2009
Unit testing does not always mean writing code for testing a unit. Read this topic on wikipedia for more information.

The idea of conducting testing at the unit (basically a public function or method) level is valuable or even indispensable for large code bases. You can even run it just in your mind. The idea is just to make sure each path is correct.

With unit testing in mind, you would also write code in a way that reduces unnecessary paths so that almost all paths will actually be used.

In a large code base, all code should be organized in a hierarchy that code at a high level will actually act as unit tests for lower level code.

This is how I code, even before I knew the term Unit testing.
Monday, January 19, 2009
I wrote a blog post about it called "Unit Testing Sucks":
The Contrarian Software Developer
Monday, January 19, 2009
> You can even run it just in your mind.

So everyone agrees with unit testing: so long as it's not *unit* testing, or, not unit *testing*.
Christopher Wells Send private email
Monday, January 19, 2009
>> So everyone agrees with unit testing: so long as it's not *unit* testing, or, not unit *testing*.

Sure, you can call my approach unit thinking.
Monday, January 19, 2009
If I have 250,000 dollars in my mind, can I then go out and buy a Ferrari? Please let the answer be yes.
Bart Park Send private email
Monday, January 19, 2009
Unit tests and precondition / postcondition assert statements have saved my bacon many times.

I had only done a few ad-hoc unit tests in the PHP and Perl code I'd been working on before.  When I started an all-new C++ project last year I determined to start off right by using cxxtest and test-first design.

It gave me an easy framework to hook my debugger into and it gave me a lot of confidence when I started optimizing the code and rewriting major chunks of it.

At first I was going for 100% code coverage, but I quickly decided that was a waste of time.  I've got something like 75%, but it is all in the most important 75%.

The unit tests have proven super useful in cross-platform porting.  While going from Linux to other Linux, BSD, MacOS, and Windows, the unit tests helped a lot, uncovering problems with super-slow function calls (Windows winsock I am looking at you), memory mapping (BSD msync sucks!) and other things.
Jonathan Briggs Send private email
Monday, January 19, 2009

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz