Value of JUnit?

I just wrapped up my part in a project which has now gone into testing. It went pretty well. I didn't do any JUnit stuff. I haven't done any JUnit in more than a year.

I actually like JUnit. It's a good way to generate automated tests. It is pretty easy to learn and use. So if there was more time then I'd be motivated to use it.

What should you do if there isn't quite enough time for JUnit? Don't get me wrong, I test and the code I write tends to have a low defect rate.

JUnit is kind of the "formal" test for the code. This type of test is a permanent type of test involving planning, source control of test code, potential peer review, ongoing maintenance. So there's overhead involved.

For me, JUnit is the third level of testing. The first level is to print the code after I write it, then go over it carefully looking for errors. I find most errors through this static self inspection. For schedule reasons we don't do peer review, which is unfortunate because I am a fan of peer review.

The second level is informally running the code. Create a scratch J2SE client program that exercises the new code. Or click on the screen that will invoke the new code and check that it works as expected. The important thing is to exercise the new code in scenarios that are as close to possible to how it will be invoked in production. This will give you confidence that the code will be OK in black box integration testing.

The third level is formally testing the code within a controlled environment like JUnit, where the formal tests go into source control and are maintained along with the actual code.

So the JUnit level is basically a formal retest of code that you know works. My experience has been that the effort for levels 1 and 2 (static code inspection, informal scratch testing) is a fraction of that required to do JUnit. So it is possible to ship robust, tested code without requiring the overheads of JUnit. So focus on checking your code and scratch testing, and only do the JUnit at the end if there is time.

An issue I do have with JUnit is that it tests code in unrealistic scenarios that will not happen in production, while missing scenarios that will happen in production. Earlier this year, I was team lead on a project. This project involved a Tomcat WAR that processed a formatted URL and sent a message over JMS to a J2EE server that invoked an EJB call that did a bunch of stuff. The code was written and ready to test.

The developer responsible for that section wanted to set up JUnit for the different pieces. I said don't bother with JUnit, do an end to end test instead. Set up the components, invoke the Tomcat URL, and verify from the database and the logs that it works end to end. Well he did what I asked, then he spent several days sorting out deployment issues, permission issues and glitches between the components. But he got it working and it worked when we went to production. We never did do the JUnit for that, and I'm glad we didn't. The issues the developer found would only have been uncovered by doing an end to end test of the running system. If we'd done JUnit instead, we would have looked bad as the problems would have been found by the test team.

In my opinion the bulk of value in JUnit is for refactoring. It's also valuable for automated test of working code. The automated test is basically a form of refactoring test. You take something that is verified to work with JUnit. Then if someone inadvertantly breaks your code by changing something you call, then the automated test will catch it. But code has to exist and work before it can be refactored. And refactoring of new code generally only happens after the current project ends. Also let the person doing the refactoring absorb the cost of JUnit, since he is the one who benefits from it.

My observation is that there is no link at the individual level between use of JUnit and defect rate. Some mediocre developers are JUnit fans, yet it doesn't seem to improve the quality of code they produce. Some developers write lots of good, low defect code while spending little time on JUnit.

My theory is that strong developers will write good code and if they do JUnit then they'll do a good job of that too. But for the good developers the JUnit is not really necessary as the code is high quality regardless. On the other hand, weak developers will write mediocre code. The JUnit will be of similar lower quality and there will be more defects regardless of how much time they spend on JUnit.