The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
TechInterview.org
CityDesk
FogBugz
Fog Creek Copilot


The Old Forum


Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

Mocking + Unit Testing

I feel that using a mocking mechanism for unit test isolation is largely a waste of time.

Here's why:

1. Having to 'record' & 'playback' adds a significant amount of overhead (coding) IMO.

2. Let's suppose you are mocking your DAL to avoid having to use a live database.  You can write tests until you're blue in the face, but only when you run your integration tests (against a DB) will you find out that someone changed a column to be non-nullable and your input validation isn't up to date.

I should note that I don't think that unit tests in isolation (using mocks) are totally useless.  I just think that their ROI is low compared to doing the same test excercise against a real live database. 

In this instance I'm talking about a database or some other 'external' system that you have under your control.  I'm totally on board with the notion of mocking a system that isn't in your control (a large claims processing system's DB, an SMTP server (you don't want to send real test emails!), and so on. 

I can't help but to feel that it's wrong that I should mock out every external dependency for my unit tests, given the additional amount of code that is required to be maintained.  These types of tests are inherently brittle and will break if you manage to change the order of a set of mocked calls, etc. 

On the other hand I feel like I'm "missing something" in that I don't grasp the value that mocking provides. 

What's your opinion?  This question/topic is geared towards .NET/Java developers...I understand this is alot easier in dynamically typed languages like Ruby.
GiorgioG Send private email
Monday, November 03, 2008
 
 
Per (2), this is why you should restrict it such that the component owns its tables. If anyone is able to bypass the component and manipulate the data directly it gets ugly really quick. If you own all management of the data, then changing the column type is done with regards to the component so it all works out cleanly.

Being able to mock the database, such as with HSQL, makes it much easier to break apart problems. Knowing that its a general failure of the component, rather than a quirk between different DB flavors, can be handy in resolving issues quickly.

Mocking, and unit testing, isn't that hard to do and usually saves time imho.
Benjamin Manes Send private email
Monday, November 03, 2008
 
 
"I feel that using a mocking mechanism for unit test isolation is largely a waste of time."

I think what you are really arguing is that unit tests are a waste and that you'd prefer to use integration tests.

integration testing != unit testing.

IMHO, developers shouldn't be worrying about integration tests.  Professional QA types should be.
LL
Monday, November 03, 2008
 
 
Of course developers should be worrying about integration tests.

If their stuff doesn't work, what good is it?

Many unit tests are useless and contrived.
Just Some Guy
Monday, November 03, 2008
 
 
Who needs testing. Just ship it.

Monday, November 03, 2008
 
 
I was one of the persons that totally ignored unit-testing until 2006.Now I do not feel confident unless I have at least 60 % Cobertura coverage .. 
For the moment we use JMock  (instead of EasyMock you mentinoned) but I believe mocking is worth the trouble.
I am not experienced with TDD but I can see the value in that as well.
My team (QA +developers) uses a lot of integration tests and functional tests and the right amount would be  70% of the tests unit-tests and the rest functional /integration.
As a general idea unit-testing should begin  when you start the project not when you are almost code-complete (some go as far as to write tests first :) ).
Hope this helps.
Mike L. Send private email
Monday, November 03, 2008
 
 
Integration and unit tests should be done as a joint effort with QA and Engineering. The bulk should be on unit testing (70-80 percent) and the rest in integration.

With TDD, you always write tests first. It's not only good practice, but it will also help you catch a LOT of problems early on that you will get burned on later.

I wrote a DAL for a product, and I used a real database, but just filled with dummy data. We were able to generate a lot of unit tests that covered everything we needed.

TDD requires effort up front, which is why a lot of people brush it off. Once you get over that initial hump, then things move much smoother down the road. Think of this as an investment in your software engineering process.
QADude Send private email
Tuesday, November 04, 2008
 
 
Unit tests are The Best Thing Since Sliced Bread, but there is one problem with them in Java(which I live with, since unit tests are TBTSSB): they have a potential to break encapsulation. Say I have an important private method which really needs to be unit tested. Up to "default" access it goes.
quant dev Send private email
Tuesday, November 04, 2008
 
 
Without mocking how do you know your passing the correct values to the DAL? Ever wasted countless hours debugging code and ended up tracing the problem back to the initial input.

Mocking can be used to verify interactions, not just to isolate chunks of code.
flukus
Tuesday, November 04, 2008
 
 
I run unit tests every few minutes. I run integration tests nightly. It's fine to have your integration tests take a long time to make database connections, talk over the Internet, drive your GUI, etc., but sitting around waiting for a couple of hundred DB queries kind of sucks.

At any rate, I don't think the primary benefit of writing unit tests is to catch bugs. To me, the biggest benefits are:

1. It forces you to write more modular code
2. It prevents bugs introduced through refactoring/change
3. It forces you to think through your assumptions
4. It makes other developers think you're cool.

Never underestimate the power of #4.
(User deleted) Send private email
Tuesday, November 04, 2008
 
 
It's more work but I prefer to set a table(s) and test with a real database, I thought I was doing the wrong thing until I googled and found out many people thought the same

regards
Jorge Diaz Tambley Send private email
Tuesday, November 04, 2008
 
 
<quote>
Unit tests are The Best Thing Since Sliced Bread, but there is one problem with them in Java(which I live with, since unit tests are TBTSSB): they have a potential to break encapsulation. Say I have an important private method which really needs to be unit tested. Up to "default" access it goes.
</quote>

What's more important to you? Testability or encapsulation?

There's nothing special about encapsulation. I'll sacrifice it for testability any day of the week and twice on Sundays.

That's one point...

The next point is: Why are you testing your private methods? Can't you test them through the public interface?

If not, is there something slightly off in the class design? Would a little refactoring create other classes where these methods would be public?
Bruce
Tuesday, November 04, 2008
 
 
<quote>
1. Having to 'record' & 'playback' adds a significant amount of overhead (coding) IMO.

2. Let's suppose you are mocking your DAL to avoid having to use a live database.  You can write tests until you're blue in the face, but only when you run your integration tests (against a DB) will you find out that someone changed a column to be non-nullable and your input validation isn't up to date.
</quote>

To answer:

1. What "record and playback" are you referring to? Also, I always get a little nervous when developers start worrying about overhead. The goal is to produce correct code. The overhead necessary to confirm that is the cost of doing business. If you don't want to pay that overhead, don't write such complicated software.

Project managers and other managers can usually be counted on to complain loud and long if you're taking too long to write software. If they're not complaining, then you're in a happy place. Forget about overhead and concentrate on producing good, low defect, clean code.

2. Maybe your database should have unit tests, too.

http://www.onjava.com/pub/a/onjava/2004/01/21/dbunit.html
Bruce
Tuesday, November 04, 2008
 
 
>>1. What "record and playback" are you referring to? Also, I always get a little nervous when developers start worrying about overhead. The goal is to produce correct code. The overhead necessary to confirm that is the cost of doing business. If you don't want to pay that overhead, don't write such complicated software.

Most mocking frameworks provide a record/playback mechanism for the tests.  Have a look @ Rhino Mocks for .NET and probably JMocks for Java.

The goal is not ONLY to produce correct code.  I always get a little nervous when developers completely ignore cost & basic economics.  The goal is to produce correct code in order to make money (for commercial software.)  If it costs more to make it, than is received, your business will fail. 

The criticality of code correctness is directly proportional to the amount of damage caused by code being incorrect.  DoD & medical equipment is tested to death - as it should be.

Correct code has to be balanced with profitability.  If you work for a huge corporation that can eat the cost of creating up to 50% of codebase that is unit tests...I'm happy for you.  Not everybody lives in that world.
GiorgioG Send private email
Tuesday, November 04, 2008
 
 
<quote>
Correct code has to be balanced with profitability.  If you work for a huge corporation that can eat the cost of creating up to 50% of codebase that is unit tests...I'm happy for you.  Not everybody lives in that world.
</quote>

You missed my point. There are untold legions of people in most organizations watching the bottom line. They don't need the help of developers to do this.

No one said go out and waste time. Of course, all testing should be commensurate with the risks involved. Do we really have to state the obvious?

A developers job is to defend the code. Define "done" for the organization that addresses the risk and then stick to it. If that includes full unit testing and mock objects then that's what it takes. Stop worrying about efficiency.

As they say, I can be REALLY fast if it doesn't have to work.

Btw, our last project had slightly more lines of code in the unit tests than in the actual product. 50% isn't all that much.
Bruce
Tuesday, November 04, 2008
 
 
What bothers me (in general) with automated unit (and even integration) testing is that all these testing frameworks, etc seem to be a result of something lacking in our development languages. 

I'm not saying I have the answer to the problem, all I know is that I feel like something's missing that would help alleviate a majority of what unit testing does for us.

And again, I'm not against automated testing, I just don't see the major value of mocking systems (classes) that will perform happily and without causing negative external impact in a unit test.
GiorgioG Send private email
Tuesday, November 04, 2008
 
 
<quote>
And again, I'm not against automated testing, I just don't see the major value of mocking systems (classes) that will perform happily and without causing negative external impact in a unit test.
</quote>

The value? Unit testing is about feedback.

On one project, we wrote nearly 8,000 unit tests. If each of those unit tests touched the network or file system or database, they would have taken quite a bit longer to execute than if they were mocked.

Let's say, for arguments sake that all of these unit tests touch some sort of external dependency. Let's say that, as a result, they each take about a second to run. That's almost 2.5 hours. Oops, you've just lost 99% of the value of having unit tests. Why? Because developers aren't going to run the test suite very often, if at all. Forget about test driven development. Soon, the tests are only being run during overnight builds and your feedback loop has ballooned from minutes to overnight.

Even with mock objects, with 8,000 unit tests there's an organizational burden. 8,000 units at a tenth of a second each is still over 10 minutes which is pushing the boundaries where a developer will run the entire suite frequently.
Bruce
Tuesday, November 04, 2008
 
 
>>>Say I have an important private method which really needs to be unit tested. Up to "default" access it goes.<<<

Try reflection?  It's not the prettiest solution but it'll work.


>>>Forget about overhead and concentrate on producing good, low defect, clean code.<<<

Easier said than done.  Sometimes poorly tested code has to ship.
Synodontis Send private email
Tuesday, November 04, 2008
 
 
<quote>
Easier said than done.  Sometimes poorly tested code has to ship.
</quote>

Hey, I get it. The console games industry could probably make a bug free game. The only problem is that games would cost $200 a copy.

A team/organization should identify "done" for a project before a single line of code is written. "Done" should represent your minimum safe code line, a line you won't cross.

If "done" includes, say, 100% path coverage in unit tests, then there's no arguing about efficiency and delivery. It's an atomic part of the task. Where teams get into trouble is when they quote code time and unit test time as two separate things. Code without unit tests is not an asset, it's a liability.
Bruce
Tuesday, November 04, 2008
 
 
Yeah I know, I always get mocked when I suggest unit testing!

yeah, I'll have the quesadilla
Gleeter
Tuesday, November 04, 2008
 
 
@Bruce

I'm testing private methods because they are important for how this class works. Of course I am also testing the public and package interface, but want to test this private method as well.

Why I want to keep it private? my class is part of a large codebase, and I want to be able to hide the method from other developers and expose its functionality only via the class interface, not directly. Also, including this method in the class interface would be confusing.


@reflection

Oh joy. I'd rather avoid using reflection when not absolutely necessary. It's a can of worms.
quant dev Send private email
Tuesday, November 04, 2008
 
 
In a mailing list when mocking came up, a member discussed the benefits of using the technique in custom/embedded hardware. He stated that most developers wait until the hardware is complete before starting their work, claiming that they can't begin until then. When they find hardware faults they get up in arms about it. In a few of his projects the teams had instead mocked the hardware and developed over that abstraction. He claimed that it took only a few days for integration testing, which did require changes of their assumptions, versus weeks as with prior projects. So mocking was extremely valuable in his scenario.

Unfortunately I can't find the thread, but I found his comments quite insightful.
Benjamin Manes Send private email
Tuesday, November 04, 2008
 
 
<quote>
I'm testing private methods because they are important for how this class works. Of course I am also testing the public and package interface, but want to test this private method as well.
</quote>

Here's an article that might be of some use: http://www.artima.com/suiterunner/private.html

Point 1 is interesting. Private methods may be a code smell that indicates the class has too many responsibilities. Refactoring the method into another class where it can be public, and therefore testable, may be an option.

Of course, it's complete guesswork at this point to say whether or not this applies to your context or not.
Bruce
Tuesday, November 04, 2008
 
 
Why are so many people confusing unit tests with integration tests and the other forms of testing????  It really is NOT that difficult of a concept.

As for mock classes, they should only be developed far enough and deep enough to support basic component functionality.  Unit testing is NOT meant to do anything but test basic functionality.  They are NOT intended to cover edge cases, stress testing, extended functionality, or error conditions.  That's why they are call UNIT TESTS.

Unit tests are there to tell you when you break something and that basic functionality is there as expected.  Software engineers are there to do development not QA.

Things like testing for edge cases, stress testing, extended functionality, or error conditions fall into the universe of the professional QA engineers.
<shaking head/>
Tuesday, November 04, 2008
 
 
I have two problems with unit testing and mocks.

In some cases the Mock would be a major development effort of its own, with multiple classes and methods to be defined before the test will even compile.

There is also the tendency to create unreadable spaghetti code in the mock methods with many "if" or "case" statements to determine what it should be doing for each test.  The other option is completely separate Mock classes for tests, which results in:

More time spent rewriting the Mock objects than doing actual work.  A lot of refactoring goes on, unit tests are supposed to help with this not make it impossible.  When the code that is being Mocked changes, all the test classes must also change or become worse than useless: actually deceptive.  This also seems to lead to even more useless abstraction interface classes written between development groups just to insulate unit tests from the other side's changes.

The second problem I have deals with network code.  Mocking just can't simulate the real life complications of actual networking and OS interaction.  Even *attempting* to write unit tests for networking code is bad practice I feel, because it gives the programmer a false feeling of confidence when he hasn't even begun to handle the possible problems.

For example, I recently ran into much code at work that comes to a complete halt without manual intervention when the server database restarts, reboots or fails over.  Yet it passed tests.  I also see a *ton* of SQL code that seems to believe query latency is 0 and when put on a WAN link both client and server CPU usage goes to 1% while both sides twiddle waiting for data packets.

Mocked up unit tests for OS/network interaction hide the programmer from the real world too much.
Jonathan Briggs Send private email
Wednesday, November 05, 2008
 
 
Two comments:

1. What language are you using? For Java and C# (and other languages) mock libraries work using interfaces. There's not nearly as much effort required to create mocks and not nearly as much maintenance (though there is some).

2. Networking.

Unit tests are should not touch the network because we need them to be fast. However, unit tests are not the only tests you should write. Functional level tests should be used to test networking issues.

Unit tests: Fast, fast, fast. Run them before every check in.
Functional tests: Slower. Run them overnight or on weekends.

You need them both.
Bruce
Wednesday, November 05, 2008
 
 
@Bruce

Thank you, this article was very interesting. I think my situation is similar to the author's situation, when he describes his parseArgsIntoLists method. I could separate my method into a separate class, but it would be class with one method, used by one class.

Though... as I'm writing it, I see another reason for separating the methods into another class. I will have to think about it.
quant dev Send private email
Wednesday, November 05, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz