These days testing interactions is getting very common. Mocking tools are becoming a necessary supplement to xUnit libraries precisely to facilitate testing interactions. I cannot imagine writing software without assert() (to test state) and verify() (to test interactions).
I’m about to show that it is very useful to distinguish two basic kinds of interactions: asking an object of data and telling an object to do something. Fair enough, it may sound obvious but most mocking frameworks treat all interactions equally and don’t take advantage of the distinction.
Let’s have a look at this piece of code:
Before mocking was invented we manually wrote stubs and tests looked like this:
Sometimes I miss those days before mocking. Test code looked so nice… The only problem was that hand writing every stub was quite cumbersome and produced tons of extra code.
Anyway, let’s try to extract a pattern from above test code:
1. when the repository is asked to find “foo” article, then return article
2. deleteByHeadline(“foo”);
3. make sure repository was told to delete article
So, to generalize it:
1. stub
2. run
3. assert
Now, let’s put it into words:
Interaction that is asking an object for data (or indirect input) is basically providing necessary input for processing. This kind of interaction is meant to be stubbed before running code under test (exercising the behaviour).
Interaction that is telling an object to do something (or indirect output) is basically the outcome of processing. This kind of interaction is meant to be asserted after running code under test.
Testing interactions like that is more natural and produces better code. Lessons learned from the old ways of testing interactions are implemented in Mockito:
Controversies
Above approach may seem controversial (quoting interesting para):
Then, when you do verify(…), you have to basically repeat the exact same expression you put in stub(…). This might work if you only have a couple calls to verify, but for anything else, it will be a lot of repeated code, I’m afraid
Basically, the controversy is about repeating the same expression in stub() and verify():
Do I really have to repeat the same expression? After all, stubbed interactions are verified implicitly. The execution flow of my own code does it completely for free. Aaron Jensen also noticed that:
If you’re verifying you don’t need to stub unless of course that method returns something that is critical to the flow of your test (or code), in which case you don’t really need to verify, because the flow would have verified.
Just to recap: there is no repeated code.
But what if an interesting interaction shares the characteristic of both asking and telling? Do I have to repeat interactions in stub() and verify()? Will I end up with duplicated code? Not really. In practice: If I stub then it is verified for free, so I don’t verify. If I verify then I don’t care about the return value, so I don’t stub. Either way, I don’t repeat myself. In theory though, I can imagine a rare case where I do verify a stubbed interaction, for example to make sure the stubbed interaction happened exactly n times. But this is a different aspect of behavior and apparently an interesting one. Therefore I want to be explicit and I am perfectly happy to sacrifice one extra line of code…
Finally…
Mockito recognises the difference between asking and telling, hence stubbing is separated from verification. Stubbing becomes a part of setup whereas verification a part of assertions. This produces cleaner code and gives more natural testing/TDD. It doesn’t increase code duplication as one might think. It’s just one of the ideas implemented in Mockito to make the mocking experience a little bit better. It’s not a silver bullet, though. Decent code is easily testable regardless of any mocking tools.
April 27, 2008 at 1:47 am |
Recently, I’ve been coaching new team members in the ways of mocking, and a common beginner mistake I see is that they verify everything. If it can be verified, then it will be verified, mostly because the tools make it so easy to do so. And, in an immediate sense, the test writer gets a feeling that “everything I just did works exactly like I think it does”.
But this quickly leads to over-specified tests. Making one little change in implementation breaks the test, even if the code is doing the correct thing. My mantra is to try not to use verify at all because we value side-effect free code. But if you must use verify, then be sure to verify only that which really matters. If the method under test performs two side effects that are of importance, then there is a high probability that the method is overloaded with too much functionality and should be broken up. Many side-effect bearing methods follow a “get indirect input, then pass it to the indirect output”, which should result in a stub->exercise->single_verify step you’re suggesting. But if a method follows a “get indirect input, then pass it to the indirect output and then pass it to the second indirect output” you’re stuck with verifying multiple collaborators. For me, it’s too complicated and a code smell. The answer is refactoring, not excessive verify calls.
Nice post!
April 27, 2008 at 2:19 pm |
The ‘verifying everything’ phenomena is quite interesting. The mocking tools offer a feature that is only seemingly cheap. Verifying everything is not cheap but what’s even more unfortunate, most mocking tools by default ‘verify everything’ implicitly (Just try not to specify some interaction then BOOM: UnspecifiedInteractionException).
Some feedback I received about Mockito:
For many, ‘verifying all’ makes the test safer and seemingly better. It’s not obvious but it is expensive, usually unnecessary, and will bite sooner or later when the tests become over-specified.
I’ll be looking in the code for your ‘verifying multiple collaborators smell’. It doesn’t sound that bad to me at first (given the test is still small & focused) but I guess it may indicate an overcomplicated code.
September 4, 2008 at 7:48 am |
Please remove the link ‘Israel’s right to exist’ from this page. It has nothing to do with your excellent mockito tool.
January 18, 2009 at 11:49 am |
[...] Once again, Mockito’s author has a great post on the stubbing or verifying debate. [...]
April 14, 2009 at 6:41 pm |
“If I stub then it is verified for free, so I don’t verify. If I verify then I don’t care about the return value, so I don’t stub.”
I don’t agree at all with what is being said here. What you’re basically proposing is to sacrifice correctness of your test code to make it more DRY, that just sounds terribly wrong.
The simple yet obvious reason being that stubbing is used to decouple tests from each other: You stub out stuff that is not under test, but still, invocations of those stubs may be totally relevant for the object under test.
Let’s say that you’re testing a method, say, A.foo, which you always want to produce some log output (i.e. call Logger.log). So you want to *verify* for A.foo that Logger.log is always invoked, but the object under test is A, not Logger, so you stub out Logger.log (to keep the test free of side effects of an object the behavior of which is not relevant for your test; you just want to verify that interaction takes place).
Stubbing out the method alone does not help: If the developer should accidentally remove the call to Logger.log from A.foo, you would have a regression in your code which your test is unable to detect. That’s a bad thing.
On the other hand, only verifying makes your test very brittle, because suddenly it relies on the correctness of the behavior exposed by Logger, which you didn’t intend to test at all. I.o.w. if Logger should suddenly behave incorrectly, e.g. throws an exception, then your test for A.foo fails, although A.foo might behave totally correct… that also can’t be right.
So, no, stubbing and verification are not mutually exclusive concepts, in fact, in most situations you will want to do both.
April 17, 2009 at 12:10 pm |
Ok, this is your point of view. The way I see it is I can increase readability and maintainability with the same level of validation. The only thing that I’m sacrificing is maybe error reporting when I used my stub incorrectly (e.g. wrong argument). However, this is something I work on at the moment.
About DRY – see, in this blog post I’m replaying to other great mocking framwework designer who writes “Mockito means duplicated code”. In fact, I believe that DRY is just one of quite a few benefits of separating stubbing from verification.
You say that because you come from the classic mocking world. Before mocking frameworks developers crafted fakes by hand. I’ve seen such codebases. Hardly any of those fakes did both: stubbing and verification. Developers did the most natural thing: stubbed if needed indirect input, verified for indirect output (using Meszaros nomenclature).
Also bear in mind that OOD seem to separate asking & telling (e.g. ‘tell don’t ask’ principle) therefore I don’t see anything wrong in doing the same in interaction testing.
Frankly, I don’t care too much if they are exclusive concepts. I just think testing is better if you acknowledge the difference so I build a framework that facilitates it
April 17, 2009 at 12:32 pm |
To that I agree. There is definitely a difference (which was also my point), and I don’t think it’s a bad thing that Mockito distinguishes between both in a clear manner (just to prevent misunderstandings, I prefer Mockito over all other Java based mocking frameworks I have used, you already did an outstanding job on that).
However I think it’s dangerous to propose that one only ever needs to use one, not both (to be frank, I’d like to see that passage gone from the Mockito docs, but maybe it’s just me seeing a problem with it). So on that I still disagree.
Cheers
Matthias
April 17, 2009 at 12:55 pm |
Great!
Mockito docs say: “Although it is possible to verify a stubbed invocation, usually it’s just redundant”. If you have ideas how to make javadocs clearer don’t hesitate to write to our mailing list.
May 12, 2009 at 2:44 pm |
@stub XOR verfiy:
I am not sure if this is the point, but consider this example: You write a unit test for a “business logic” class (somewhere between UI and low-level persistence) which uses a DAO. This BL class should use methods of the DAO, depending on your test input. In some cases the parameters will be passed through 1:1, but sometimes these parameters may be altered by the BL class or completely new parameters will be computed. (you dont want to / cannot read the entire source code, but you know the rough specs)
In many cases, assert() on the retrun value will fail, if the method call (of the dao) is wrong, but imho a failed verify is more expressive then a failed assert.
examples:
1) invoking a deleteBiggest…() twice instead of once
=> return was correct: deleted obj, but due to a wrong if-block, delete was called again
2) getting the parameter sequence wrong (creating a 1-way relationsship between 2 obj the wrong way)
=> return of the relation was mixed, so data base and program state did not match anymore
1) no chance in hell finding that w/o verify() or some code coverage tool
2) a more rigrid stub may have uncovered this as well
I know especially 2) is more like additional debugging, but if you need to look into someone else’s code / test it, every little bit helps.
verify seems paranoid / senseless when testing good code, but I prefer writing 1 more verify to reading 10 lines of ugly code.
May 12, 2009 at 3:04 pm |
I’m not too worried about this one. I’m putting it in a separate test method that verifies that the biggest is deleted. It’s an interesting behavior and I’m happy to create a specific method that describes it.
I agree that if you make a mistake in calling a stubbed method (e.g. wrong args) it’s not going to be immediately visible what’s wrong and some extra debugging might help. I respect you prefer to have a verify() that helps out in this matter. However, there is a feature I’d like to introduce in Mockito that will try to solve this problem. Hear me out let me know what you think
I’d like Mockito to provide extra debugging info only when the test fails. Exception message could potentially contain information like: you stubbed this method this way but you called it with different arguments. Or: you stubbed this method but this stub was not really used at all. The interesting part is that I’d like this extra debugging info to be added to any exception, regardless if it’s NullPointerException from the code or AssertionError from the test. Basically, any exception/error that makes the test fail.
I guess if Mockito had this, you wouldn’t need to look for too long at 10 lines of ugly code, neither you would need to verify your stubs, right?
July 27, 2009 at 1:56 pm |
Interesting idea about the messages. But why is it that you want them only to show when the test fails?
When can such an error occur?
1) functionality is missing: in that case I would like to see, you stubbed a method but it is not called yet
2) you wrongly stubbed something: in that case I want to see the error too
3) you stubbed something, but it is not needed: an error too please
I might have overlooked something, but I think it is handy if the test would fail on stub configuration errors.
August 1, 2009 at 3:13 pm |
You might be right.
My reason is somewhere between keeping Mockito not very aggressive and pushing tests to be smarter. Smarter, meaning failing for the ‘right’ reasons. Example: if the functionality is missing, then the failure should somehow point out that… the functionality is missing (e.g. central assertion fails). If the functionality is missing I guess I don’t want a failure like: ‘you didn’t stub me correctly’. I’d prefer: ‘this assertion fails because functionality is missing. By the way: this stub was called with wrong arguments.’
Unless I’m wrong
Anyway, if you use Mockito 1.8.0 check out VerboseMockitoJUnitRunner