the net value of tests

[Please, read me carefully. If at any point in time you think I’m preaching to stop writing tests then STOP reading further and FORGET what read so far. Thanks!]

One might think that tests only bring value, that is muchos dolares. If this is the case then why many teams stop maintaining their tests at some point? Or they @Ignore a test forever when it is too difficult to fix? Isn’t it like throwing away ‘value’. Why on earth you are throwing away $$$!?

Thing is that tests bear maintenance cost: keeping them green, refactoring along with the code, etc. Let’s talk about net value of a test: Net = Value – Cost. If you write crappy tests then the net value might be negative. Since I visited quite a few archeological sites digging in legacy code I saw many tests with negative net value.

When half of your team wants to run ‘svn delete‘ on the ‘FitnesseRoot‘ folder it might mean your tests’ net value is plummeting down. Obviously, fitnesse is just an example, any tool can be a blessing or a curse depending on your skill in misusing it. (Dear god of Agile, I swear I love fitnesse… but only with GivWenZen).

Let’s think about it. I’ve got 3 tests with a respective value I conjured using cosmic energy.

Test 1. Adds an article to the content db. Value: 50 $, Cost: 20 $
Test 2. Adds an article & edits the article. Value: 50 $, Cost: 20 $
Test 3. Adds an article & edits it & deletes it. Value: 50 $, Cost: 20 $

The cost is relatively high when compared with value because those tests are deep functional tests that require the whole system to be set up; they use funky frameworks to drive the web browser; they are kind of painful to debug & maintain.

Total cost is easy, just sum up the cost and be scared by 60$.

Total value is tricky. To your great surprise (hopefully ;)) it is not 150$… but 55$ !!! Why? Read again test #3 – it covers the same cases as test #1 & #2 so the value of those tests does not really sum up! I threw 5$ extra because by having the functionality in separate tests we have better defect localization when tests fail. Hence 55$.

This means the net value of tests is: -5$ (minus five dollars).

Funny enough, the tests mentioned are a real case from a gig I was few years ago. I argued with the QA lead that keeping those tests separate does not make sense. To my despair, I failed. 6 months later nobody was maintaining functional suite of web-clicking tests in this project.

My message distilled:

1. Understand the net value of tests.
2. Look after your tests as if they are your newborn daughter!

Now go back to the wilderness and keep writing sane tests!

4 Responses to the net value of tests

  1. this is a great post Szczepan. I just google searched for it specifically so that I could send the link to my new team.

  2. Dierk König says:

    You are discounting the value to the marginal value but not the costs to the marginal costs.
    Even with duplication, the costs of initially writing the tests would go down massively. With proper factoring of the tests also the maintenance costs would.

    I would expect something like 55 – 25.

  3. szczepiq says:

    Great point Dierk, I missed that. Yet, I think that in my particular situation the discount for cost is a bit smaller, e.g: 55 – 35. Therefore I still prefer 50 – 20 (30 net value) than 55 – 35 (20 net value).

    I will update my post to accommodate cost discount.

  4. Leo says:

    Draq 发表于2012年02月21日 19:30上个体验版没有解决这个问题,也没人联系我,我就想知道你们到底在干什么。就因为这个体验版,我装上去后在覆盖正式版的都运行不起来,最后卸载了,把文件都删了才装上正式版的,我的设置皮肤都没了,你们敢有个回答吗?我交年费就换来这服务?广告都还没说你们的事呢。我再发一遍,诺顿安全特警2012正式版报的警,我买的正版激活码,别说是用的破解杀毒软件完整路径: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\minizip.dll威胁: WS.Reputation.1________________________________________________________在电脑上的创建时间 不可用上次使用时间 2012/2/17 ( 12:52:03 )启动项目 否已启动 否________________________________________________________未知诺顿社区中使用此文件的用户数量: 未知____________________________未知此文件版本当前未知。____________________________中此文件具有中等程度风险。____________________________威胁详细信息威胁类型: 智能网络威胁。 很多迹象表明此文件不可信任,不安全________________________________________________________文件操作文件: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\minizip.dll已删除____________________________文件指纹 – SHA:e612f29d671235b5d83bc9701390d659001a22a4b43261d6e8e59d8237ceb665____________________________文件指纹 – MD5:45e651145bf7880cccb640bf8a2bf8ff____________________________完整路径: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\zlib1.dll威胁: WS.Reputation.1________________________________________________________在电脑上的创建时间 不可用上次使用时间 2012/2/17 ( 12:52:59 )启动项目 否已启动 否________________________________________________________未知诺顿社区中使用此文件的用户数量: 未知____________________________未知此文件版本当前未知。____________________________中此文件具有中等程度风险。____________________________威胁详细信息威胁类型: 智能网络威胁。 很多迹象表明此文件不可信任,不安全________________________________________________________文件操作文件: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\zlib1.dll已删除____________________________文件指纹 – SHA:a25fbe3d11e9a386dd88829d1ef37b7c0b26ce2943b9734b67d7021d2c09d2c1____________________________文件指纹 – MD5:bf9275314e9dd2ef1a65e9240b7b49d7____________________________完整路径: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\libexpat.dll威胁: WS.Reputation.1________________________________________________________在电脑上的创建时间 不可用上次使用时间 2012/2/17 ( 12:55:16 )启动项目 否已启动 否________________________________________________________未知诺顿社区中使用此文件的用户数量: 未知____________________________未知此文件版本当前未知。____________________________中此文件具有中等程度风险。____________________________威胁详细信息威胁类型: 智能网络威胁。 很多迹象表明此文件不可信任,不安全________________________________________________________文件操作文件: c:\program files (x86)\common files\thunder neotwrk\tp\ver1\1.1.2.101_1111\libexpat.dll已删除____________________________文件指纹 – SHA:353e7be5cfa58f86d35cd146aef08982a807224ec67c56ab49d5de9a77ef24ed____________________________文件指纹 – MD5:91741355fd2ce25ddef5c6318769471c____________________________希望你们查明是怎么回事。QQ:2709****||

%d bloggers like this: