I’ve been trying to write unit tests for any iPhone apps that I write. With Xcode, that’s much much easier said than done (it should not be this difficult!).

I’ve tired a couple of frameworks and tried to follow several steps found on Google. So far nothing is doing the trick.

My latest effort followed instructions for using OCUnit written by Sen:te. After following the steps outlined, I get this error when compiling and running the project:

error: Testing rig /Somepath/iPhoneSDKVersion/otest exited abnormally with code 139 (it may have crashed).

Naturally I turned to trusty Google which came up with nothing helpful.

Thinking that it was a failure to follow the directions, I downloaded the sample test application, loaded the project in Xcode, then built and ran the project. Same error.

I tweeted to Michael Galpin, a programmer at eBay, asking for any ideas on unit testing iPhone apps. Since he had just given two presentations on mobile development [1, 2] for the Silicon Valley Code Camp 09, I thought he might have some ideas.

His initial response does not give me good feelings about iPhone development:

I’ve only seen homebrewed stuff. XCode is not exactly extensible.

He did turn me to the google-toolbox-for-mac which has some promising instructions on doing iPhone unit testing.

Using Xcode feels something like this...

What bothers me about doing iPhone development is that I am leaving all my very excellent development tools for iffy tools.

Sun wants Java on the iPhone and is willing to make it happen. Java has one hell of a community behind it and Apple could really benefit from it.

Unfortunately someone at Apple has some bad information on Java.

Perhaps Apple is just interested in keeping it’s pet language alive and kicking. After all, Apple seems to have problems with Not Invented Here.

This was not really meant to turn this into a bit of an Apple hate-fest. It’s just that the tools and community that Java has is light years ahead of what Apple is offering anyone interested in iPhone app development.

When I’m effectively boxed into Xcode to do iPhone development, that’s a problem. I would like an IDE that doesn’t seem to break at least one thing every update. Chris Hanson’s steps to setting up unit tests has required updating between various Xcode updates.

More importantly, the barrier to entry for unit testing iPhone apps is painfully high. I have almost given up and I’m already sold on the idea of unit testing. Imagine if I wasn’t already sold.

google-toolbox-for-mac is next up for me to try. Wish me luck, I’ll need it.

I ran into an issue while trying to test some innocent looking GWT-EXT code for a Google Web Toolkit app:

    @Override
    public void onActivate( com.gwtext.client.widgets.Panel panel )
    {
        if ( !( panel instanceof MyCustomGwtExtPanel ) )
        {
            throw new IllegalArgumentException( "Panel must be a MyCustomGwtPanel." );
        }

        ... snip ...
    }

The test

I had one test to test that an exception is thrown:

    @Test(expectedExceptions = IllegalArgumentException.class)
    public void testOnActivate_unexpectedPanel()
    {
        Panel panel = new Panel();
        new TripReportsPanelListener().onActivate( panel );
    }

Test failures

It was so simple, I guess it just had to fail:

Caused by: java.lang.UnsupportedOperationException: ERROR: GWT.create() is only usable in client code! It cannot be called, for example, from server code. If you are running a unit test, check that your test case extends GWTTestCase and that GWT.create() is not called from within an initializer or constructor.
at com.google.gwt.core.client.GWT.create(GWT.java:85)
at com.google.gwt.user.client.ui.UIObject.(UIObject.java:140)
… 23 more

By using GWTMockUtilities.disarm in the setup and GWTMockUtilities.restore in the teardown, I made some progress by ended up with this error instead:

java.lang.UnsatisfiedLinkError: checkExtVer
at com.gwtext.client.widgets.Component.checkExtVer(Native Method)
at com.gwtext.client.widgets.Component.(Component.java:108)

The solution

The solution is right there in the stacktrace from Google.

I extended GWTTestCase but continued to get the same errors. The reason is simple enough, but has bitten me a a couple of times before. GWTTestCase is written using the old JUnit 3 style tests. My tests were using TestNG annotations which tells TestNG to run the tests as TestNG tests. Consequently the setup and teardown methods in GWTTestCase never get called. JUnit acts the same way if you mix and match JUnit 3 and 4 style test declarations.

It’s been a frustrating experience with UnsatisfiedLinkErrors and NoClassDefFoundErrors, but I eventually got what should be a simple unit test working. And it sucks that my seemingly innocent unit test have so many dependencies. It also stinks that GWT must be started up to run the tests resulting in a feedback loop increase of 10-15 seconds.

When it comes to unit testing you might find yourself wanting to test private methods. Here’s four solutions, some much better than others.

1. Don’t test private methods (refactor!)

If you find yourself needing to test private methods, you’re code is trying to tell you something – listen up!

Unit tests should test the behavior of classes and not the implementation details. Unit tests should be able to naturally cover your private methods. If you find this difficult or impossible to do, consider it a code smell.

Testing implementation will become a barrier to refactoring since you will be unable to change the implementation without updating tests. As Charles Miller put it “The amount of work you have to do to improve your code becomes the amount needed to change the private methods, _plus_ that required to change the tests. As such, you’re less likely to make the improvement.”

If you choose to test private methods keep in mind you are likely taking on some technical debt.

Pros: You’re not testing private methods. Best solution. Results in better design.
Cons: Can sometimes be tricky to implement, especially in legacy applications.

2. Use reflection

Thanks to reflection, access levels are more of a suggestion than a requirement. With a little bit of code, we can access any private method (or field for that matter).

First use reflection to change the access level:

MySuperCoolDao mySuperCoolDao = new MySuperCoolDao();
// get method by name "isThisCool" and signature
Method isThisCoolMethod = mySuperCoolDao.getClass().
getDeclaredMethod( "isThisCool", String.class );
// make method accessible
getCoordiantesMethod.setAccessible( true );

Then we use the newly accessible method:

boolean isWindowsCool = isThisCoolMethod.invoke(mySuperCoolDao, "Windows");
boolean isUbuntuCool = isThisCoolMethod.invoke(mySuperCoolDao, "Ubuntu");

assertFalse("Windows is not cool...", isWindowsCool);
assertTrue("Ubuntu is very cool", isUbuntuCool);

A big negative to this method is maintainability. In fact this just happened to me today: I refactored out a private method and as expected the test failed. Unlike other solutions (like the package level solution below), the failure was at runtime – not compile time.

Pros: Relatively easy and quick. Does not break encapsulation.
Cons: Tests become busier. More difficult to keep tests updated with production code. Security settings could prevent this technique from working.

3. Use package level access

This solution is very easy, simply remove private to get the (default) package level access. As long as your tests are in the same package as the production code, no problem. Since it’s good practice for your unit tests to be in the same package (but in a separate source folder), this works well.

Right out of college, this was the first way I learned to handle these situations. Almost everything method was package level. It never really smelled quite right, but the postives out weighed the negatives. In hindsight it might not have been the best solution, but there are far worse things.

Pros: Quick and easy.
Cons: Breaks encapsulation. Classes become a little noisier (although not as bad as using public).

4. Mix production and test code

This is the worst of all the solutions, so I hesitate to even bring it up. By making your test class an inner class in your production code, your tests will have access to private methods. Unfortunately this solution leaves your production code dependent on your test code.

Challenge: is there a legitimate reason to do this? I can’t think of one.

Pros: At least you’re writing tests! Weaksauce, I know.
Cons: Production code contains more than production code.

Quick update: Chad Bradley informed me that it is possible to test private methods using Groovy. I would be hesitant to use the technique without knowing for certain if it’s a bug that will get fixed or an actual language feature.

Turtles are surprisingly fast.

I’ve noticed for a while our unit tests run very fast through my IDE but take forever on our build box. At first I attributed this to our severely overloaded build box, but I was wrong.

In our particular case the tests take 5 minutes 3 seconds to run through Ant while forking for each test class. Very not cool.

If you’re using fork to run your JUnit tests in Ant, there are two attributes to be concerned with: fork and forkmode.

The forkmode attribute is the big one here. Possible values are “perTest” (default), “perBatch”, and “once”.

It turns out that because “perTest” is the default; meaning forking is done for each test class. While that might be what you want, it can make your tests significantly slower. Using the “once” option instead means forking happens just once; all your tests will be ran together in a single JVM.

After switching the forkmode to “once”, our test running time plummeted to 33 seconds for a net gain of 4.5 minutes. As Borat would say, “very niiiice”. (In this case 33 seconds is still longer than it should be, but that’s for another post).

There’s some very good technical reasons to run all your tests in the same JVM. As Chris at nominet.org.uk pointed out, it might reveal some issues with your production code or test code:

Having the tests run each in its own JVM also covers up problems. You can create whatever sort of mess you like and it’ll all be swept away before the next test runs. While going through a setting the forkmode to “once” I found a database connection leak in some test code. This sort of problem would be much more visible if all the test code was running in a single JVM and reminded me of Martin Fowler’s advice on testing resource pools.

Of course Martin Fowler comes through again with good advice.