The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
TechInterview.org
CityDesk
FogBugz
Fog Creek Copilot


The Old Forum


Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

Unit Tests for Dummies

No, not a book title, but a statement of fact. Joel quotes and agrees with Mr. Nitzan that:

"[I]n software development, we like to have people unit-test their code. However, a good, experienced developer is about 100 times less likely to write bugs that will be uncovered during unit tests than a beginner. It is therefore practically useless for the former to write these."

So refreshing and so controversial! Yet I agree completely!

I write the occasional unit test for something that needs to be tested for wide ranges of complex data, but that's it. Am I a good, experienced developer? Why yes. Of course we will now have the debate that points out the fact that deciding not to do unit tests doesn't mean you are any good and I agree with that already.

But let's just agree on one thing: Unit Tests for Bad and Inexperienced Developers.
Dennis Atkins
Tuesday, December 07, 2004
 
 
>But let's just agree on one thing:
> Unit Tests for Bad and Inexperienced Developers.

Unfortunately "Bad and Inexperienced Developers" is usually defined as = everyone excpet me!

Surely a great developer ( such as myself ) would write unit tests for the cases where a user enters 256 non-printable characters into the username field!
Martin Beckett Send private email
Tuesday, December 07, 2004
 
 
Would you allow nonprintable characters to be entered into your text fields?
Dennis Atkins
Tuesday, December 07, 2004
 
 
One major point in favour of unit tests is ensuring that a library does what it says in the manual, accross version changes.

Does the password control allow non printable characters, does it do the same thing after the latest service pack ?
Martin Beckett Send private email
Tuesday, December 07, 2004
 
 
Unit testing isn't just for finding bugs, it supports refactoring too.
John Topley Send private email
Tuesday, December 07, 2004
 
 
"Would you allow nonprintable characters to be entered into your text fields?"

Is the data gathered from an external source?

DON'T TRUST IT. VERIFY.

At best directly behind the entry point into your realm of control.

You know, the RFCs define the maximum length of an URL, it won't be longer, so we don't need to check it. Oops, buffer overflow, browser exploit.

Oops, SQL injection.

Do you think that user input from an OS-controlled input field is uncritical? What makes you sure the input field won't be substituted by a registry entry or some network traffic sometimes in the future?

Oops, admin rights through illegal characters, network exploit.

Even looking at the lowest level functions of your favourite programming language gives a rule of thumb. Parsing rules are made with a white list, not a black list. It is often not (in general, of course) "this and that character is not allowed and causes the program to break catastrophical", instead "parsing is stopped on the first unfitting character".
Don't trust it. Verify.
Tuesday, December 07, 2004
 
 
"Would you allow nonprintable characters to be entered into your text fields?"

Well, of course, *I* wouldn't. But those Bad and Inexperienced Developers in charge of the GUI probably would.
Breandán Dalton
Tuesday, December 07, 2004
 
 
Unit testing to me has less value in general programming to spec (design by contract), than it does in regression testing. What better way of at least getting some idea that changes to a system haven't disrupted previously flawless operation than running the suite of unit tests that you've built up while developing the system? No, it's not foolproof - plenty of bugs exist that still are only really detectable to human testing - but it's definitely better than nothing.

Rock Star developer or not, everyone makes mistakes, spills coffee into the keyboard, etc.
Andrew Cherry Send private email
Tuesday, December 07, 2004
 
 
Everyone says Unit Tests support refactoring. I have found the exact opposite - Unit Tests hinder refactoring. They're just another block of code that has to be rewritten to match the changes to the code structure.
Mr Jack
Tuesday, December 07, 2004
 
 
You are probably trying to unit test too low a level.
The general idea of refactoring is that the method names / external api doesn't change but the internal implementation does.
You unit test against the published API.

In practice of course it's the API that changes as you actually try and use the library and discover that all those carefully thought out interfaces to it are wrong! 

No software ever survives contact with the user
Martin Beckett Send private email
Tuesday, December 07, 2004
 
 
If other developers will ever need to use and modify your code, it's worth writing unit tests at the beginning. It may take time and be boring, but it will require more time and frustration later if someone has to do it then.

It's not a question of how good you are or what kind of mistakes you make in the first version. Unit tests are a safeguard for later, when you or someone else may need to modify your code, possibly in a hurry. They're really an investment to mitigate risk during future maintenance of the code.
BL
Tuesday, December 07, 2004
 
 
I'm sure we'd all love to be rock stars (and imagine ourselves as such), but I've found that a bit of "process" has really helped me.  Unit tests are one example, David Allen's "Getting Things Done" method is another.

For me, the biggest win of unit testing is in writing the tests first, and then the code - i.e., design by contract via test driven development.  Also, the small, easy accomplishments of building each function and ensuring that it passes the test motivate and energize me.

Now I'm sure I must be a rosh katan sort of guy, but my experience has been that a well-measured dose of organization and formality gets me doing my best.

Not much of a rock star, just call me Ringo.
Jesse Smith Send private email
Tuesday, December 07, 2004
 
 
> Am I a good, experienced developer?

A good experienced developer would think about the other people on a project and the people that come after you. When you are dead do you think unit tests will help new people? For others on your project do you think UTs would help them make changes to code you have created?
son of parnas
Tuesday, December 07, 2004
 
 
> design by contract via test driven development

DBC operates at runtime. Your unit tests only test the unit and only are run at test time. That's not good enough and DBC fills the gap.
son of parnas
Tuesday, December 07, 2004
 
 
<quote>
So refreshing and so controversial! Yet I agree completely!
</quote>

I don't.

The "good and experienced" developer is often working on the more difficult assignments. Any screw ups this developer makes are likely to be nasty indeed. Even more reason to write unit tests.

Also, unit tests aren't just for testing code. In TDD they help drive design. The also support refactoring. If you never plan on revisiting the code you're writing, then by all means, dispense with the unit tests. If someone is likely going to be following after you, not writing unit tests is just plain stupid.

This mythical "good and experienced" developer may exist, but I've yet to meet a developer that can work without unit tests.
anon
Tuesday, December 07, 2004
 
 
I agree with anon above. As previously mentioned unit tests are indispensable for regression testing. Additionally, unit tests make it explicit how a function is supposed to work.

As I see it, unit tests are like documentation: A pain to write, a pain to maintain and a necessity for any non-trivial project.
jz Send private email
Tuesday, December 07, 2004
 
 
>Additionally, unit tests make it explicit how
> a function is supposed to work.

They make explicit what the developer felt like testing. If your are using TDD then they aren't even real tests in the sense of traditional testing.

For knowing how a function is supposed to work you need to look at the code and the doc. For example, the lisp spec was relatively short when compared to the 100MBs of tests. I wouldn't want to read the tests to figure out what was going on.
son of parnas
Tuesday, December 07, 2004
 
 
Another use for unit tests is to provide an example of how your routines should be used.  Manuals are all well and good (if they ever get written), but nothing beats a working example.

I tend to unit test everything.  I write a routine or 3, write unit tests for them, repeat as needed.

Then again, I too am not much of a Rock Star.  Call me Jessica Simpson.
Snotnose
Tuesday, December 07, 2004
 
 
"So refreshing and so controversial! Yet I agree completely!"

I agree too.  Unit testing is very inefficient.  Actually, I'll go farther:  it's a crutch. 

People learn through incentives and risk-taking.  After you make a mistake a certain number of times and it bites your ass, you learn to avoid it in the future.

Why can't we take risks and learn from mistakes, rather than invoking a mistake-proof, puritanical methodology from the start?

Yes, it costs $$$ and ego, pain, and time; but trials by fire produce better developers in the end.
indeed Send private email
Tuesday, December 07, 2004
 
 
I am not interested in producing better developers. I am interested in producing software that our customers value and pay us for.

Even if the unit test tests for obvious things, you can look at it and see what's been tested for. If there is a bug, someone has the chance to amend the test while fixing the bug. And, for regression, they would be invaliable.

I would willing to pay the price of having coding take 30% longer if I could get a regression test out if it when it's finished. Coding is only a fraction of the time spent adding a new feature so it won't actually add 30% to the total time, only to the coding time.
Miles Archer
Tuesday, December 07, 2004
 
 
<quote>
Why can't we take risks and learn from mistakes, rather than invoking a mistake-proof, puritanical methodology from the start?
</quote>

In carpentry, you would "measure twice, cut once", and if doing many cuts you create a jig so you don;t even have to think about each cut.  Its great to learn from your mistakes, but if you can avoid them all together by learning from others mistakes your on the way to Rock Stardom!

I think some people are just masochistic ( or maybe sadistic since its the poor user who suffers the pain)
Honu
Tuesday, December 07, 2004
 
 
"I am not interested in producing better developers. I am interested in producing software that our customers value and pay us for."

...

Pardon me for being shrill, but: what the hell does that mean?  Do better developers not produce more software that customers value, by definition?

Do I have to qualify what "better" means to the Nth degree, for us to have a substantive discussion?

"If there is a bug, someone has the chance to amend the test while fixing the bug. And, for regression, they would be invaliable."

Like bugs are just innate qualities of code.  Somewhere along the line, the unit-testing methodology (and most modern software apologism) made the assumption that bugs are something OTHER than programmer error.

The value of which you speak, only exists if there are bugs to begin with.

Is it terribly out of the box to believe that if programmers learn not to make mistakes initially, by eventually going "without training wheels," we can avoid the need to actually fix those mistakes?

"I would willing to pay the price of having coding take 30% longer if I could get a regression test out if it when it's finished."

Why?  Is building a regression test so hard after-the-fact, if the code was well-written to begin with?

Any well-designed system has subsystem boundaries (well-defined interfaces) that can be easily wrapped up into a harness.  What part of good software design requires unit testing be done before-the-fact, or at a certain time, to ensure good regression tests at a later date?

Isn't easy testing mostly a function of having well-defined subsystems and interfaces, NOT the tests themselves?  In which case your position is a red herring--good design is doing the heavy lifting of testing, not unit tests.
indeed Send private email
Tuesday, December 07, 2004
 
 
Talk about a tempest in a teacup.
Do you *always* need to unit test?  No.  Are the times (plenty) when you should?  Yes, especially with new development.  There's nothing more to it than simply being reasonable.
Far, far too many people assume that all you need to do is hire smart people.  But if your smart people don't do (or worse, aren't allowed to do) smart things, what's the point?  Seriously: developing quality software is very, very hard.  Who gives a damn if one in a million developers can do it backwards and blindfolded?  Bully for them but the rest of us have to soldier on regardless.
John Haren Send private email
Tuesday, December 07, 2004
 
 
^

I am personally not opposed to unit testing per se, but I am opposed to the _methodology_ which implies that any code written without a unit test is sinful.  As I said, it's puritanical in nature; and the (illogical) evolution of so-called Test Driven Development proves this.
indeed Send private email
Tuesday, December 07, 2004
 
 
"I would willing to pay the price of having coding take 30% longer if I could get a regression test out if it when it's finished."

If you already have to build an acceptance test, which aren't the unit tests, then why build the unit tests?
son of parnas
Tuesday, December 07, 2004
 
 
What I've always wondered is: who unit tests the unit tests?  Do we have to write both good and bad functions for every unit test to test in order to make sure they correctly flag good and bad functions as such?

:P
David Kentley
Wednesday, December 08, 2004
 
 
There actually exists software to test your unit tests.  Jester.sourceforge.net for JUnit (along with Pester for PyUnit), and Nester.sourceforge.net for NUnit.

I haven't used it myself for more than cursory examination, but this is what I read about it.

It works by taking a copy of your source files (it doesn't automatically make a copy currently, you do it) and then systematically mutates your code, recompiling, and running the unit tests again.  Mutations that don't cause the tests to fail are flagged differently than ones that do cause your tests to fail.  All the mutations performed are spit out in a big XML file with the results.

Ideally, you would examine the mutations that did not cause your tests to fail and adjust your tests accordingly, since the test has proven to be flawed.

Jester is on 1.22, Pester is on 1.01, and Nester is on 0.2 right now.  I only fiddled with Nester, and it's not a slick process yet.  It also takes a while to run, since it recompiles your solution after each mutation.

The mutations it does by default include making if tests automatically succeed, automatically fail, change == to !=, != to ==, ++ to --, -- to ++, true to false, false to true, 0 to 1, 1 to 0.

They're working on handling critical errors within Nester like stack overflows or out of memory errors which you may run into due to the mutations.  Currently if one happens, the application crashes and you don't get any results.  I don't know if Jester or Pester have covered this yet.

It seems like a neat idea if you're heavily into TDD.
Neyah
Wednesday, December 08, 2004
 
 
> It seems like a neat idea if you're heavily into TDD

It does sound need, but not for TDD. In TDD tests drive the design. They aren't tests in the sense of trying to prove your code works.
son of parnas
Thursday, December 09, 2004
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz