I’ve noticed for a while our unit tests run very fast through my IDE but take forever on our build box. At first I attributed this to our severely overloaded build box, but I was wrong.
In our particular case the tests take 5 minutes 3 seconds to run through Ant while forking for each test class. Very not cool.
If you’re using fork to run your JUnit tests in Ant, there are two attributes to be concerned with: fork and forkmode.
The forkmode attribute is the big one here. Possible values are “perTest” (default), “perBatch”, and “once”.
It turns out that because “perTest” is the default; meaning forking is done for each test class. While that might be what you want, it can make your tests significantly slower. Using the “once” option instead means forking happens just once; all your tests will be ran together in a single JVM.
After switching the forkmode to “once”, our test running time plummeted to 33 seconds for a net gain of 4.5 minutes. As Borat would say, “very niiiice”. (In this case 33 seconds is still longer than it should be, but that’s for another post).
There’s some very good technical reasons to run all your tests in the same JVM. As Chris at nominet.org.uk pointed out, it might reveal some issues with your production code or test code:
Having the tests run each in its own JVM also covers up problems. You can create whatever sort of mess you like and it’ll all be swept away before the next test runs. While going through a setting the forkmode to “once” I found a database connection leak in some test code. This sort of problem would be much more visible if all the test code was running in a single JVM and reminded me of Martin Fowler’s advice on testing resource pools.
Of course Martin Fowler comes through again with good advice.