expected exception in tests

July 26, 2010

I have a feeling too many people get it wrong so let me stress:

@Test(expected = SomeException.class) // IS BAD FOR YOU

I do enjoy jUnit a lot, thanks guys for sharing! Yet, I don’t like @Test(expected) feature of jUnit. In my TDD classes I simply teach to avoid using it. I admit the feature can potentially make few tests cleaner. However, it tends to bring more trouble than value. Newbies too often misuse it which leads to poor quality tests. Therefore:

Rule 1: don’t use @Test(expected)

My colleague (cheers Bartek!) pursues Master at the Uni. They teach him testing. One day a lecturer was presenting some test code:

@Test(expected=Exception.class)
public void testSomething() {
  //line 1: throws NullPointerException
  //lines 2-30: some crappy test code
}

The test is useless. Due to some bug in the setup the test method breaks in line 1. Therefore most of the test code is not even executed. The expected exception does great job in hiding this issue; the test is green! Therefore:

Rule 2: If you violate Rule #1 then at least make the exception very specific. Example:

@Test(expected=BreadOutOfStockException.class)
public void shouldBuyBread() {}

Even if you use specific exception then the test does not document exactly where the exception is thrown. Sometimes it may lead to subtle bugs. Therefore:

Rule 3: Don’t violate Rule #1 when the test method has multiple lines.

Instead you can simply write:

@Test
public void shouldBuyBread() throws Exception {
  //given
  me.goToBakery(bakery);
  bakery.setBreadAmount(0);

  try {
    //when
    me.buyBread();
    //then
    fail();
  } catch (BreadOutOfStockException e) {}
}

Above is my preferred way of documenting ‘exceptional’ behavior. There you go, it is The Ultimate Test Template for ‘exceptional’ behavior.

JUnit is getting better. New goodies for exception handling in tests arrive. @Rule ExpectedException better documents what action triggers the exception. Also, it nicely maps to //given //when //then style. The downside is that you need some extra code if you want to interrogate the exception, for example if you want assert if exception message contains a substring (api as of 07-2010). Nevertheless:

Rule 4: Favor @Rule ExpectedException over @Test(expected) when test method has multiple lines.

Rule 5: Keep an eye on new versions of your testing libraries.

If you are already at RI in TDD than consider my Rule #1 as a safety net. You’ll never know when a new junior team member joins. He may easily misuse the patterns he sees in the codebase. Javadoc for JUnit does not say how to safely use @Test(expected) (at the moment).

If you are learning testing don’t get too upset with my dogmatic rules. If only you write any tests then I am smiling already. You smile too, write tests and be happy. Soon you’ll get your RI rank in TDD.

Rule 6: Write tests & smile!


the net value of tests

July 8, 2010

[Please, read me carefully. If at any point in time you think I’m preaching to stop writing tests then STOP reading further and FORGET what read so far. Thanks!]

One might think that tests only bring value, that is muchos dolares. If this is the case then why many teams stop maintaining their tests at some point? Or they @Ignore a test forever when it is too difficult to fix? Isn’t it like throwing away ‘value’. Why on earth you are throwing away $$$!?

Thing is that tests bear maintenance cost: keeping them green, refactoring along with the code, etc. Let’s talk about net value of a test: Net = Value – Cost. If you write crappy tests then the net value might be negative. Since I visited quite a few archeological sites digging in legacy code I saw many tests with negative net value.

When half of your team wants to run ‘svn delete‘ on the ‘FitnesseRoot‘ folder it might mean your tests’ net value is plummeting down. Obviously, fitnesse is just an example, any tool can be a blessing or a curse depending on your skill in misusing it. (Dear god of Agile, I swear I love fitnesse… but only with GivWenZen).

Let’s think about it. I’ve got 3 tests with a respective value I conjured using cosmic energy.

Test 1. Adds an article to the content db. Value: 50 $, Cost: 20 $
Test 2. Adds an article & edits the article. Value: 50 $, Cost: 20 $
Test 3. Adds an article & edits it & deletes it. Value: 50 $, Cost: 20 $

The cost is relatively high when compared with value because those tests are deep functional tests that require the whole system to be set up; they use funky frameworks to drive the web browser; they are kind of painful to debug & maintain.

Total cost is easy, just sum up the cost and be scared by 60$.

Total value is tricky. To your great surprise (hopefully ;)) it is not 150$… but 55$ !!! Why? Read again test #3 – it covers the same cases as test #1 & #2 so the value of those tests does not really sum up! I threw 5$ extra because by having the functionality in separate tests we have better defect localization when tests fail. Hence 55$.

This means the net value of tests is: -5$ (minus five dollars).

Funny enough, the tests mentioned are a real case from a gig I was few years ago. I argued with the QA lead that keeping those tests separate does not make sense. To my despair, I failed. 6 months later nobody was maintaining functional suite of web-clicking tests in this project.

My message distilled:

1. Understand the net value of tests.
2. Look after your tests as if they are your newborn daughter!

Now go back to the wilderness and keep writing sane tests!


Sweetest

February 22, 2010

In a constant battle for sweet tests (readable & maintainable) I developed another weapon. Have a look at this tool for acceptance testing: Sweetest. What do you think?


//given //when //then forever

December 7, 2009
@Test
public void shouldDoSomethingCool() throws Exception {
    //given
	 
    //when
 
    //then

}

I like to call it the Ultimate Test Template. I’m so fond of those 3 little comments but surprisingly, I didn’t buy it at first. Micheal (it’s you, tapestry-maven-iPhone fan boy =), friend from ThoughtWorks showed me it a couple of years ago. Actually, he didn’t show it to me – I just overheard him coaching a young dev about it. That day I thought I didn’t need any hip comments because my tests were great anyway. It was foolish.

Don’t be a fool like me and start writing //given //when //then today. Life is too short for messing around – you want to get level 85 in software craftsmanship soon, right? Here’s the deal: use the template for 1 iteration and if you don’t like the results then I will give you your money back. Seriously, no matter what you think about it – buy it! BTW. If you need to document some ‘exceptional’ behavior in your test somewhere there is the template for tests with exceptions.

Lately, I’ve been selling //given //when //then quite relentlessly. I even try to sell it via Mockito api. (The link also shows how to install the template in Eclipse so don’t miss it!)

I tried to lobby for the Church of given-when-then in Krakow, Warsaw & Kiev. I heard rumors that Wroclaw develops a growing number of brothers and sisters in faith =)

I think I forgot to thank Dan North for given-when-then and Liz Keogh for the idea of BDD aliases in Mockito. There you go!


TDD unlimited

November 26, 2009

So I read a short rant that questions TDD (or unit testing – it’s hard to figure). Here’s my message:

If you want to learn your trade don’t get too excited about gurus that outsmarted the Universe and found cracks in TDD. There are way more IT celebrities that successfully practice TDD. They blog about, they teach and you know what? Their code kicks ass big time. So… Write loads of tests, try TDD, taste BDD. Learn it yourself and at some point you won’t be needing blogs to decide what side to take (Alliance or the Horde? =).

Now, if you already know your trade please try to influence others to write more tests and learn TDD. How much inspirational is this or that! Anyway, it’s just my humble request to the world. Of course, you can write whatever you want, dear influential blogger!


Mockito in Zurich

June 21, 2009

I will have a pleasure presenting Mockito at Jazoon in Zurich. My session is on Thursday – make sure you don’t miss it. During the session I’m going to show few slides and do live TDD with Mockito & other mocking frameworks. I’m going to use Infinitest. Guys, it’s huge. Brett Schuchert wrote about it, Bartek told me to try it several times. Finally I did and I must say I’m impressed. Infinitest is like transcending to a real TDD.


is debugger a modern translator?

May 19, 2009

Couple of days ago we interviewed a young adept of the craft. We let him code but he got stressed out a bit and wrote very complicated code. At some point he stopped understanding the code he wrote. So he started using debugger to figure out the meaning of the code. This is when debugger became sort of a modern electronic translator.

The thing is we probably don’t want to use debugger as a translator. We prefer to understand the code we write. Instead of debugging I usually recommend writing a small test that covers the logic in question. You can try this exercise next time you are about to launch the debugger.

Don’t get me wrong – debugger is a handy tool in every developer’s kit. It’s just the less you use it the better you are in producing high quality code. Actually, it’s probably other way around – when you get fluency with TDD/simple design you will soon discover how rarely you use debugger.