Test Culture Episode 3 : Can You Unit Test Everything ?

No.

But you can auto-test a lot more than you think.

Here is a quick tour of what can or cannot be tested easily.

Database Access

No. When a method you unit test calls database, then it should not be unit tested anymore. It’s integration testing.

Why ?

  • Accessing the db every time tests runs could be dangerous;
  • Unit testing is an atomic process and need stable state and know variables. Db could change.
So what’s the solution ?
  1. Divide your method in small methods that do no db access and test them instead;
  2. Mock db query result by manually building objects with the data you want;
Besides, implementing this solution forces the dev team to refactor in any small modules. Hence the maintenance and reading of the code is facilitated.

Network Communications

No.  Same as database acess. Your unit tests must be in a closed environement. You cannot rely on network state.

If you are testing acess to another application you have the same problem. It’s integration testing.

What can you do ?

  • Unit test package reception by mocking them;
  • Place some sensor and indicators somewhere that polls connections regularly and send alerts the team when it’s broken.

User Interface (GUI)

Yes. But with some limitations. You’ll need a framework to be able to test your user interface. The graphical part.

The principle is the following :

  1. Capture/record some GUI actions or code them in a specific language and capture screenshot;
  2. Run automated testing that will reproduce the steps you indicated and take screenshot;
  3. Compare automatically both screenshot. If differences exists, then your GUI is not consistent.
Problem with this approach : it’s very long to install unit testing, and it’s subject to change.

User Experience (UX)

This is a tricky one.

First what is UX ? User experience is a more complete set of parameters than GUI. For example : speed of use, consistency, ease of use, paradigm employed and so on. Go to ux-fr for more info.

It is critical for user adoption. Unfortunately like unit tests it’s often despised as useless, or gimmicky. It’ not.  It should be considered very early in the project conception. Because after all : that’s the interface with the user. That’s what he sees and uses.

It can’t be unit tested. But be sure to allocate resources and time to test different options. Better : put your software in the hand of your users and see how they react. Capture data and analyse what should be changed in your UX.

Internationalization

No.  If your software is in two languages, how can you be sure that all strings have their versions in each languages ?

For this you won’t need unit testing, but regular inspection with a tool like Eclipse. It will tell you what is internationlized or not.

Besides, code review is an excellent way to find non-internationalized strings.

Configuration

 No. Configuration can vary from a single variable changing to a whole lot of libraries to switch. It’s hard to test.

Of course all configurations should be tested. In some cases dedicated machines with different configurations should be maintained. You can also have each developer run a special conf on is machine. This way if one is broken you’ll know it quickly.

Framework and third-party library

No. But it’s not important.

If you use tier software, you should choose them for their quality. Hence you assume that they are tested.

For example : I used Eclipse RCP as a base on two projects. Whenever I’m testing acess to RCP I assume it’s always right.
Problem : sometimes you need to load a lot of tier libraries to test a single method. In this case you cannot unit test it properly, because it’s to heavy.

Accessibility

Accessibility is the capacity for a disabled person to use your software. For example color blindness.

This could be tested. But not unit tested.

You can do final user test sessions, or use standards to ensure your software is accessible.

Regressions

Yes, yes and yes ! 

In my opinion this is one of the main reasons to put unit tests in place.

If you have solid tests in place since the start of your project, you can be sure they will by useful when you product is changing. In so many cases did we have to test manually if a feature still behave the same way as before. This could have been automatized.

Know Bugs

This one is a special one for sure.
Yes, you can have unit tests that determine whether or not a known bug is still there.
But the trick is : the know bugs is not from your software, but from a library you use.
In the excellent book Pragmatic programmer: from journeyman to master you’ll find the following recommendation : “Find bugs once : One a human tester finds a bug, it should be the last time a human tester finds that bug. Automated tests should check for it from then on”.

TODO

  • Ask your team : what do we test exactly ? Can’t we test more ? If  so what’s the cost compared to still using manual testing or no testing ?
  • Go in your bug tracking tool and search for a  bug you have been correcting twice. If you find one : unit test it NOW!
  • Revise your planning to include more unit testing.

For the developers out there

What do you unit test ? Only code or other config files for example ?
If you unit test GUI, which framework do you use ?

Author: Jean-Baptiste Rieu

Trained software engineer and now product manager. I ❤️ #space #architecture #typography #books #games #verticalfarming. I do #productmanagement #software #abtesting #data. I work on #payment @sundayapp_ Blogging mostly to practice writing, and to engage with others on life in Korea, products, engineering, books and anything worth geeking about.