Nowadays Android developers are offered two major types of unit testing. On the one hand, local tests that run on the Java Virtual Machine, and on the other hand instrumented tests that run on a device. For this purpose we would either test via emulator, or would use an Android-powererd hardware device. As a group of developers with primarily JEE background, we believe that continuous integration is crucial for successful software development. When I started at Triona, our Android projects were already using both types. The resulting question is: Which type of framework is to be picked under which circumstances?
The following blog entry will deal with this particular question by comparing two of the most commonly used frameworks Robolectric and Espresso.
How do instrumentation unit tests differ from local ones?
Instrumented tests depend on the Android API that contains all the major Android classes (e.g. Activity, Fragment,…). Hence Espresso unit tests can be executed directly via emulator or hardware device. In contrast to local unit-test frameworks like Robolectric that cannot simply be executed by the JVM due to the missing API. Therefore the Robolectric framework provides these Android classes itself.
1) Robolectric (local test)
Since we needed a simple and fast way to test our code on a build server, we pretty early decided to integrate Robolectric in our project. Below I will list some of the significant pros and cons about Robolectric.
Advantages
- The most significant advantage of running tests on the JVM is the higher speed of execution compared to instrumented tests. As a result of not having to launch an emulator or a device, Espresso works recognisably slower than Robolectric.
- Robolectric tests can easily be executed on a continuous integration server. We experienced that it’s somewhat complicated and laborious to set up a working environment for instrumented tests on build servers.
- The unit tests are executed by referring to a third party implementation of the Android API. Developers therefore tend to adopt a sceptical attitude towards frameworks like Robolectric arguing that the tests are not based on the actual Android API but on a reflection of it. As we worked along with Robolectric, we didn’t notice any limitation due to missing Android API features. Almost every Android class is available.
- Robolectric also stands out when it comes to multi threaded code (e.g. background work with AsyncTask).
Disadvantages
- Like already mentioned, Robolectric covers almost every feature of the Android API but not all of them. Every so often Google publishes new Android versions including new features and Java classes. As a result, we need to keep in mind that functionalities of newer API versions might not be available at the very same time they appear.
- Though Robolectric facilitates most of the unit tests we are not getting around performing UI and integration tests (e.g. testing multi touch,…). In fact, it would be grossly negligent just to rely on Robolectric tests.
2) Espresso (instrumented test)
For many years Android lacked a good approach to writing tests. In Oct. 2013 Google eventually released a test framework called Espresso. Unlike Robolectric the big advantage of Espresso is it’s test reliability.
Advantages
- Espresso tests are much more reliable. This is due to the fact that the unit tests run on an instance of Android on a device or an emulator with access to the real device and its resources. As a result Espresso test are more suitable for executing integration tests.
- Writing Espresso tests is very uncomplicated which means that you most likely won’t run into major problems. In chapter 3, I will try illustrate the problems we were confronted with while using the Robolectric framework.
Disadvantages
- Since instrumented tests run on the Dalvik VM instead of running on the JVM, the speed of execution is noticeably slower compared to local unit tests.
- Complicated to set up on a build server
3) The frustrating part of Robolectric
After writing our first Robolectric tests, any attempt to run our unit tests resulted in a compilation failure. At first we had to deal with following error:
java.lang.IllegalStateException: You need to use a Theme.AppCompat theme (or descendant) with this activity…
The reason we were having this problem is because our activities are extending ActionBarActivity which is not bundled with Robolectric. This error seems to be common among many other users of Robolectric. After fiddling around with lots of configurations, we noticed that Robolectric 3.+ comes with more than one test runner. We were using the test runner that was also used by Robolectric in one of the main samples.
Nevertheless, switching to the RobolectricGradleTestRunner in our test classes solved all our problems.
@RunWith(RobolectricGradleTestRunner.class) public class ProbeActivityTestRobo {...
Conclusion
In our case we decided to go along with both testing frameworks Robolectric and Espresso. Keep in mind that it’s not really an “either…or” question. We do believe that Espresso is the most reliable and secure way of testing but if you’re part of a group with several developers and a central repository, I would highly recommend the usage of a continuous integration server executing Robolectric unit tests. Continuous integration testing is key when it comes to meet a certain level of quality standards.