The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

What do you look for in a (C/C++) unit testing framework?

I'm not too thrilled with the current crop of C++ unit testing frameworks. They're either too huge, with features I'll never use (Boost.Test), or require lots of code for even simple unit tests (CppUnit, unit++, etc.). I've begun creating my own, and will probably end up distributing it as open source (BSD or something similarly permissive).

My question to you is - what are you looking for? It's got all the normal features like suites, error handling, fixtures, etc. already. Anything else you have always wished you could get in a C++ unit testing framework?
Name
Saturday, September 23, 2006
 
 
It got to be lightweight, easy to use and integrate. Something just like http://unittest-cpp.sourceforge.net/ is.
Andrzej Haczewski Send private email
Saturday, September 23, 2006
 
 
Thread Safety. But that may just be me. CPPUnit is the only framework I could find that claims to be threadsafe.
Michael Suess Send private email
Saturday, September 23, 2006
 
 
+1 Andrzej for suggesting UnitTest++

If @Name makes something better than UnitTest++ then I would consider but the second it becomes more complicated I will revert back to easier options. Simplicity and functional for the task at hand - what I look for in any library.
Chris Send private email
Saturday, September 23, 2006
 
 
@Michael

As in, being able to run each test in a separate thread? I believe mine does that...in fact, I can't imagine how one could make a framework that isn't thread safe.
Name Send private email
Saturday, September 23, 2006
 
 
As in: being able to test a multithreaded program with it. When I write a test, I usually go parallel inside the test. It must therefore be safe to call the Asserts from multiple threads at the same time. Look in my blog for an example of such a test, e.g. here:
http://www.thinkingparallel.com/2006/09/20/how-to-do-it-once-in-openmp/
Michael Suess Send private email
Saturday, September 23, 2006
 
 
> require lots of code for even simple unit tests (CppUnit, unit++, etc.).

For CppUnit I have a minimal header file (a 'template' but not in the sense of a C++ template), which I copy/paste/edit to create a new test suite.

Most of the verbiage in the header, I think, is needed because C++ doesn't support reflection (which would e.g. let the unit test framework get the list of test case methods from a test suite class at run-time).

How do you better?

> My question to you is - what are you looking for?

I wanted something I could port to run on the Windows Mobile O/Ses.

The test suites (but not necessarily the framework which runs them) need to be buildable in the same environment as the code that's being tested (eg make files, or DevStudio).

Reporting is useful: ability to run tests interactively with a GUI, and/or as part of an unattended build.

Ability to select specific suites (instead of aways running all suites).

> It's got all the normal features like suites, error handling, fixtures, etc. already. Anything else you have always wished you could get in a C++ unit testing framework?

Sometimes the hard part of unit testing is setting up the environment in which the tests will run: e.g copying clean test data to a clean test directory, and so on. With CppUnit and NUnit I find myself coding these steps in the make files, and/or in the setup routines for specific suites.
Christopher Wells Send private email
Saturday, September 23, 2006
 
 
>Most of the verbiage in the header, I think, is needed because
>C++ doesn't support reflection (which would e.g. let the unit
>test framework get the list of test case methods from a test
>suite class at run-time).

>How do you better?

Creating a static version of the test, which registers itself with the suite. No reflection is needed, since tests don't get added or removed at runtime. There's a global list of suites, obtained in the same way.

I'm pretty sure Boost.Test and CPPUnitLite both work this way as well.
Name
Saturday, September 23, 2006
 
 
@ Michael

I've been doing some reading on your site, and can now give a better answer. The tests/assertions/etc. are thread safe. There's no shared data between them, except for a few const strings.

However, the output handlers are not thread safe. The two GUI handlers (GTK+ and Qt) both use their own threading APIs, which are probably not compatible with OpenMP. The console output handler doesn't bother with threads at all, but it's probably easy enough to make the relevant methods atomic.
Name
Saturday, September 23, 2006
 
 
I look for fewer of them.
son of parnas
Sunday, September 24, 2006
 
 
As far as  know, both QT and GTK+-Threads are only wrappers for the native thread-api that is used on the particular platform. OpenMP-directives usually work fine with the native threads on any platform where it is supported (it of course depends on your compiler).

Whether or not that is really enough to make your framework threadsafe is a different question altogether.
Michael Suess Send private email
Sunday, September 24, 2006
 
 
I like CxxTest -- the web site should be http://cxxtest.sf.net/. I discovered it from Noel Llopis series of articles on TUs: http://www.gamesfromwithin.com/articles/0412/000061.html

It is not as verbose, nor complex, as other frameworks as the reflection is not done in the C++ source (which can't do anything about that), but in an external perl tool.

There are a few points where it can be improved, but nevertheless I quite happy with it.
Luc Hermitte Send private email
Monday, September 25, 2006
 
 
I look for non-use of static registrar objects, syntactic convenience (short names, mainly), configurable output targets, output that shows both program text and actual value for any test failures, and messages that are clickable if you opt to have them printed to the Visual Studio debugger window.

(Once I've got my system working without exceptions, I'll add "no exceptions" to my list as well. But in their absence it will be hard to reuse test snippets by calling functions, unless you use longjmp, which is hardly better.)

I feel all of these are pretty basic, but I've seen them missed out, so I mention them anyway.
Tom_
Monday, September 25, 2006
 
 
Ideally we'd be looking to use a single framework for all languages we use (primarily C++ and C#, BTW). I feel that that is especially important when a project uses multiple languages.

Right now that means NUnit...when I have a bit of time I'm going to look at whether it's practical to test native C++ projects with NUnit using a C++/CLI wrapper.
Anna-Jayne Metcalfe Send private email
Tuesday, September 26, 2006
 
 
1. Ease of use.
2. Don't make me deal with test registration.
3. Don't make me link to special libraries.

In short, manage the tests for me, and otherwise stay out of the way.

I like TUT (http://tut-framework.sourceforge.net/).
Samuel Reynolds Send private email
Tuesday, September 26, 2006
 
 
I'm surprised to see, both from that Games from Within article and Tom_, that exceptions are not wanted. It would probably take a day or two to remove them.

Perhaps what I should have asked is "What DON'T people want?" My library currently requires exceptions for failure and error handling, template functions for typesafe assertions, STL lists, and stringstreams (for displaying user-defined types). All of these work correctly on modern compilers (tested on VC++6 and GCC 3.4). Is it important to not rely on these C++ features?
Name
Tuesday, September 26, 2006
 
 
"I like CxxTest"

That's what I use.
A. Nonymous
Tuesday, September 26, 2006
 
 
+1 to CxxTest
Bruce Rennie
Thursday, September 28, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz