The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

sticky tests

What are some of the things to avoid so that automated tests do not run only on a specific machine and can be moved around as we like?
Tuesday, July 11, 2006
Some points:
1. you application and tests should be rolled up in installable package that can be installed onto any of your machines, preferably automatically.
2. Your test should take its configuration from a configurable source like an XML file, command line, or database.
3. The test results should be written to a database or XML file so you can determine when the test has completed, know what the results are, and display them on your dashboard.
son of parnas
Tuesday, July 11, 2006

Establish a testing environment complete with all of the debugging tools and logging tools you'll need. Create a virtual machine image or a disk image. Pass that image around as you'd like.

In the long run, creating a test environment that "can be moved around as we like" is probably not as useful as one might think. For example, as soon as you introduce variances in the underlying machines, you will have to take those into consideration whether those variances are the cause of your bugs.

Let me take a step back.

For the purposes of this discussion, there's two kinds of automated testing - making sure that you've correctly built the application and ensuring that the application works correctly with other third party programs or hardware.

In general, you want to keep to a fairly consistent machine when doing build tests. If you're doing compatibility tests, then you essentially want to treat those tests as an application unto itself, not as a set of tools that piggyback on your primary program. (This is essentially son of parnas' suggestion.)

My suggestion is that you quite literally define your target platforms (Windows 2000 with 512MB of RAM), create virtual machines that mimic those platforms, and then you can test every platform configuration you have on any one machine you happen to have handy.
Tuesday, July 11, 2006
Yes, rebuilding a VM is the best way to go.  On one of my customer sites, they have a server that does this every hour when it detects a change in the SVN.  Even Tomcat and Maven are reinstalled into the default clean environment.

Oh, and it's called "kenny", so when the build is broken, it emails everyone with "You killed Kenny!"
KC Send private email
Wednesday, July 12, 2006
My favorite example of tests not being able to run on different machines:

A test would produce an alphabetical list of results, some of which started with the name of the computer the tests were run on, and compare that against a static list of results that just had the computer name substituted in. My computer had a name that started with an 'a', so the results weren't in the same order on my computer as on the original computer, and so the test failed.

So, the moral is to think about all the factors that could make an individual test be non-portable (timing, computer name, OS version, etc.). Also, +1 for all the comments about making it easy to set up the test environment on new computers.
Exception Guy Send private email
Wednesday, July 12, 2006

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz