This is the big question : how much ?

Cheap people only calculate cost with money.  But if you want well-tested software you’ve got to see the bigger picture.

Here is how I see it. But since it’s quite a new matter for me, I’d appreciate to debate about it and have your experience in comment.

  1. Your company’s goal is to make money 
  2. To make money you must produce good useful software (to sell or to use internally)
  3. To produce stable tested and useful software you need a good team (who cares about tests and quality more than having a 1000 features)
  4. And to have a good team you need more than money

A Comparison

Let’s compare 3 kind of tests I hope you already practice in your company : unit tests, manual integration tests and manual regression tests.

[table id=1 /]

Alright, that’s my vision, and you can guess in favor of what I preach. But when talking to fellow devs and seeing different projects, I can tell this table is pretty accurate.

An Example

This week I spent 2 days automating every test I could think of in the service I’m developing.

I though my application was simple and ok before putting unit tests everywhere, even on the most trivial classes.

But I discovered bugs. Sometimes simple copy/paste bugs. Sometimes bigger ones.

So I started to automatize those 208 manual tests. It became clear that I saved more than 2 days.

Also I saved money by increasing my understanding of the product and my competences in mocking and JUnit.

The project leader was ok to spend 2 days improving code by refactoring and unit testing after I asked him. And I’m sure he took the right decision. Of course we’ll never know exactly but I think we saved debugging days.

The Cost Of Debugging

Sometimes debugging could seem faster than implementing unit tests. And in absolute time it could be true. But be honest : if you have many procedures and many people in the bug hunting sessions, your costs are going to crash the ceiling.

Necessary but costly heavy testing sessions go like this :

  1. Prepare an environment for the testers to tests
  2. Ensure that testers test against the right version of specifications
  3. When a tester find a bug (sometimes that could be avoided simply by 3 lines of JUnit) he fills a report
  4. The project manager gets the bug report, test again and attribute it to a developer if needed
  5. The developer stops everything he is doing to try to reproduce, correct and test again
  6. If the developer is a good one he implements immediately a unit test to avoid finding this bug manually ever again
  7. Back to the project manager for validation
  8. Back to the tester for validation
Don’t you think some bugs are more expensive than a complete unit test batch ?

TODO

  • Go on, hit some numbers in Excel and try to find out the cost of letting your people think instead of just passing some manual tests.
  • Come back here and tell us where you’ll invest more in your project. We’d be delighted to debate about it here.
  • Ask yourself honestly : What’s the cost of NOT doing unit tests ?

For the developers out there

What’s your opinion/experience on this ? Do you feel managers don’t allocate enough budget on unit tests ? Are they taking to many risks by only paying for manual tests ?