* The Business of Software

A former community discussing the business of software, from the smallest shareware operation to Microsoft. A part of Joel on Software.

We're closed, folks!


» Business of Software FAQ
» The Business of Software Conference (held every fall, usually in Boston)
» Forum guidelines (Please read before posting!)


Andy Brice
Successful Software

Doug Nebeker ("Doug")

Jonathan Matthews
Creator of DeepTrawl, CloudTrawl, and LeapDoc

Nicholas Hebb
BreezeTree Software

Bob Walsh
host, Startup Success Podcast author of The Web Startup Success Guide and Micro-ISV: From Vision To Reality

Patrick McKenzie
Bingo Card Creator

Testing new product

hi all. New to forum here.

We getting ready to publish new mobile product and have following question about testing: how do you perform testing for your products before release?
We are very limited team (we only 2 developers) do you outsource testing to companies or hire inlab professional testers?

We have automated unit tests but like to know what others approach this aspect of software business. Thank you for feedback!
Xin Hue Send private email
Wednesday, December 03, 2014
The more different types of testing you can do, the better.

I do most of the testing myself, but also use customers to beta test major new releases and pay companies ( such as testlab2.com ) to do third party testing.

See also:
Andy Brice Send private email
Wednesday, December 03, 2014
First, you will want to have a lot of unit tests. You already have them, so that is a great start so don't overlook their importance.

Automated unit tests provide a first-line safety net for two reasons:
1) As you evolve your product, you will introduce bugs which they will catch (competent programmer thesis)
2) they can catch other satellite, unrelated, nasty and emergent bugs even if they were not designed for those (the coupling effect).

But you also don't want redundant unit tests. Just like having manual duplicate tests, redundant automated tests are correlated to high amounts of bugs in software. See

See my presentation here:

To beef up your automated tests, I recommend you utilize mutation testing techniques. These techniques allow you to design tests that catch concurrency issues, deadlocks, etc.
See here:

Dislaimer: I write a mutation testing tool called Mutator for Java, Ruby and JS.

I also agree with the previous poster that performing different types of testing is beneficial. Each approach/type of testing has a blind spot so you will want to mix the testing types.

ET has been shown to *not* outperform scripted manual testing, but it is more efficient in several ways (for instance. you do not have to write test cases).
On the other hand, if you utilize use cases for your development planning, then you already did most of the test case writing work and it's a simple matter to convert those. So I also recommend having scripted testing for repeatability.
See here:

My rule of thumb is: ET for usability and uesr experience (UX),  scripted for functionality, performance, security, etc.

For ET, I recommend the book "Exploratory Software Testing" by Whittaker:

Finally, be aware that there is simply no correlation between how complex a software product is vs the amount of bugs. You might need or want to release software that is buggy but good enough to get your MVP out.
See here:

I hope that helps.
Mario G.
Ortask founder and chief engineer
Mario G Send private email
Wednesday, December 03, 2014
Another disclaimer: I do research in software testing.

Mario G.
Mario G Send private email
Wednesday, December 03, 2014
Thank you for details reply! This already give us good direction
Xin Hue Send private email
Friday, December 05, 2014
I highly recommend How We Test Software at Microsoft.

At Microsoft, there's a tester for every single developer. So if they had 15,000 devs, they would have 15,000 testers. Buggy software obviously hurt their bottom line and they know it.

So that's how serious you can take it.

Hope you write a great bug-free app!
Li-fan Chen Send private email
Friday, December 19, 2014

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz