I have a feeling too many people get it wrong so let me stress:
@Test(expected = SomeException.class) // IS BAD FOR YOU
I do enjoy jUnit a lot, thanks guys for sharing! Yet, I don’t like @Test(expected) feature of jUnit. In my TDD classes I simply teach to avoid using it. I admit the feature can potentially make few tests cleaner. However, it tends to bring more trouble than value. Newbies too often misuse it which leads to poor quality tests. Therefore:
Rule 1: don’t use @Test(expected)
My colleague (cheers Bartek!) pursues Master at the Uni. They teach him testing. One day a lecturer was presenting some test code:
@Test(expected=Exception.class)
public void testSomething() {
//line 1: throws NullPointerException
//lines 2-30: some crappy test code
}
The test is useless. Due to some bug in the setup the test method breaks in line 1. Therefore most of the test code is not even executed. The expected exception does great job in hiding this issue; the test is green! Therefore:
Rule 2: If you violate Rule #1 then at least make the exception very specific. Example:
@Test(expected=BreadOutOfStockException.class)
public void shouldBuyBread() {}
Even if you use specific exception then the test does not document exactly where the exception is thrown. Sometimes it may lead to subtle bugs. Therefore:
Rule 3: Don’t violate Rule #1 when the test method has multiple lines.
Instead you can simply write:
@Test
public void shouldBuyBread() throws Exception {
//given
me.goToBakery(bakery);
bakery.setBreadAmount(0);
try {
//when
me.buyBread();
//then
fail();
} catch (BreadOutOfStockException e) {}
}
Above is my preferred way of documenting ‘exceptional’ behavior. There you go, it is The Ultimate Test Template for ‘exceptional’ behavior.
JUnit is getting better. New goodies for exception handling in tests arrive. @Rule ExpectedException better documents what action triggers the exception. Also, it nicely maps to //given //when //then style. The downside is that you need some extra code if you want to interrogate the exception, for example if you want assert if exception message contains a substring (api as of 07-2010). Nevertheless:
Rule 4: Favor @Rule ExpectedException over @Test(expected) when test method has multiple lines.
Rule 5: Keep an eye on new versions of your testing libraries.
If you are already at RI in TDD than consider my Rule #1 as a safety net. You’ll never know when a new junior team member joins. He may easily misuse the patterns he sees in the codebase. Javadoc for JUnit does not say how to safely use @Test(expected) (at the moment).
If you are learning testing don’t get too upset with my dogmatic rules. If only you write any tests then I am smiling already. You smile too, write tests and be happy. Soon you’ll get your RI rank in TDD.
Rule 6: Write tests & smile!
I use similar pattern for testing exceptions (try/fail/catch). First thing to notice is to always remember about adding fail(“Expected SomeException”) right after the execution of “when” method. If you forget about it, the test will remain green even when the method didn’t threw the exception.
Also I am not certain about the place you put “then” comment. This section is meant to have assertions and, more generally, some testing logic. So, IMHO, “then” should be placed in catch block body. And then this catch body might even have assertions on exception message or other error codes – to narrow the possibility of “false positive” test even more.
I like the idea of ‘then’ in the catch block! But wouldn’t it look weird if catch block is empty (like in my example)?
Good point that fail() is a must.
I write “then” comment above catch block. My rationale is that “when” I invoke the tested method “then” I expect the exception to be caught, not the test failure.
As for missing fail() I’ve seen so many tests that got it wrong! For me it’s another reason to do test-first because if you write the test before the implementation you immediately spot such mistake.
[...] Szczepan post about expected (exception) in ours [...]
Hello Szczepan,
I never had any problems with misuse of expected exceptions (and I have done enough of code reviews) so I’m little surprised that you consider it to be a serious problem worth blogging about.
Anyway, I think I don’t agree. :) If my method throws sensible exceptions then they are a part of its API (yes, they are) and should be tested. Thus expected exceptions is one of many tools in my testing toolbox.
BTW. Why do you use jUnit and not TestNG? Yeah, I know – curiosity killed the cat – but it still surprises me that people stick to old, good jUnit.
–
Cheers,
Tomek Kaczanowski
I wrote this article after someone (again) posted a code snippet to the mockito mailing list with glaring misuse of @Test(expected). I found myself teaching the same thing over and over again so I blogged about it. Trust me, I’m lazy. I wouldn’t write anything if there wasn’t an incentive for me – now I can simply pass on the link!
I’m not sure what you disagree with… I don’t say you should not test exceptions. I try to give generic guidelines on how to test exceptions *successfully* =)
Why don’t I use TestNG? I find jUnit slightly better integrated with tools I use and there’s very little extra in TestNG api that I need. Both tools are ok =)
Szczepan, read the beginning of your post again – you say there that “(expected = SomeException.class) // IS BAD FOR YOU” (notice the CAPITAL LETTERS) and that you “don’t like @Test(expected) feature of jUnit.”
This is in contradiction to what you say now:
“I don’t say you should not test exception”.
So is it bad or not? Is it bad or is it a useful feature which can be misused? Should you test your exception or you shouldn’t because this “is bad for you”?
Sounds schizophrenic to me. ;)
–
Cheers,
Tomek Kaczanowski
There are number of ways of testing exceptions and this post tries to rank them. I don’t see a contradiction but if you see it… I guess I will have to live with this somehow =)
I’m dogmatic, every coach is. I don’t like features that on average day lead to poor code. I’m sure you are RI enough to see the big picture 8-)
Ok, so I have to comment on this. Your proposed “solution” doesn’t alleviate the problem at all. You’re still dependent on the original developer knowing how to write a good test. You properly catch BreadOutOfStockException in Rule 3, but had the developer caught Exception instead you’d be right back to square one. In short, Rule 3 is just Rule 2 but with more bloat.
Hi Szczepan
I found your blog through the Mockito homepage.
Have you seen Antony Marcano’s way of verifying thrown exceptions (http://antonymarcano.com/blog/2010/07/old-favourite-expected-exceptions/)? I find it realy good because it follows the given/when/then pattern so nicely.
In essence, this is his pattern (slightly edited by me):
@Testpublic void shouldVerifyAThrownExampleException() throws Exception {
// given
// ...
Exception thrown = null;
try {
// when
// ...
} catch (Exception caught) {
thrown = caught;
}
// then
assertThat(thrown, is(instanceOf(ExampleException.class)));
}
Perhaps a bit more lengthy, but that’s easily fixed by a code template.
Hi Szczepan,
as you I felt quite uncomfortable with JUnit’s approach of testing expected exceptions. So I used Mockito to implement a little helper that addresses the issues: http://code.google.com/p/catch-exception/
I would like to here your opinion about that.
Rod
I like your library :) It’s a bit too late for me to use it since I’m mostly with groovy these days doing a lot of spock testing.
The api looks good, the name of the library is good. Still, there’re always of ways you can play with the API so you should have a lot fun with it :), for example:
when(foo).bar();
thenThrown(RuntimeException.class);
or:
//given
…
//then
thrown(RuntimeException.class).when(foo).bar();
//sort of consistent with Mockito api:
verifyThrown(RuntimeException.class).when(foo).bar();
verifyThrown(messageContains(“foo”)).when(foo).bar();
Also regarding the API, you might want to lean towards consistency with other frameworks, e.g. fest assert – build the api starting with assertThat, or mockito – make sure the method names don’t clash and do not overlap, etc. This way your framework will be nicely pluggable to existing testing toolkits many devs use.
Hope that helps and good luck!
Hi Szczepan, I liked your ideas and I made them part of the latest version of catch-exception. Thank you for your suggestions :-)
Rod
This style is steller! You definitely know how to maintain a reader entertained. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Amazing job. I genuinely enjoyed what you had to say, and far more than that, how you presented it. Too cool!