Android Tests

I saw a question on /r/androiddev that made me pause:

Are these terms synonymous? instrumentation testinstrumented testinstrumented unit test– Fr4nkWh1te1

Naming things is hard in software, but when I looked around at the Android docs I was surprised to find several separate references to these, some of which conflict with each other. Here's my take on Android tests and how to apply generic software testing definitions to them.

Tests

Tests are small programs you write to test your actual program. Most modern languages and frameworks support writing, running, and reporting on tests in a variety of ways. Android is no different and really just builds on an existing Maven project structure for Java modules - this is why some tests exist in the src/test/java directory. We'll get to the src/androidTest/java directory later.

Unit Tests

One of the first types of tests that we learn about when getting started in testing is unit tests which validate each unit of the software performs as expected. A unit is the smallest testable piece of your code, usually with some inputs and one deterministic output.

The Android doc makes a reference to local unit tests:

These are tests that run on your machine's local Java Virtual Machine (JVM). Use these tests to minimize execution time when your tests have no Android framework dependencies or when you can mock the Android framework dependencies.

At runtime, these tests are executed against a modified version of android.jar where all final modifiers have been stripped off. This lets you use popular mocking libraries, like Mockito.

Those bits are specific to Android development and writing unit tests for Android apps. The idea behind this is that actually running an android.jar requires either an emulator or device which has significant overhead. And since the Android Framework has its own tests and is not your code, unit tests should not be concerned with it.

Their documentation states that unit tests should be in the test directory, and this jives well with other project setups in most other languages. I will point out that other types of tests like integration tests that don't require the Android Framework could benefit from being in this directory, for example Robolectric (also why don't they have https?).

Instrumented Tests

The word instrumented means running a program with code tracing, debugging, monitoring, profiling, logging, and performance counters. The Android docs in my opinion abuse this definition a little bit when they say instrumented unit test where they define it as you are running a unit test on some unit of your code but instead of a stubbed out android.jar you're running on a real device or emulator and exist in the androidTest directory instead of test. I believe this is immediately an integration test (where individual units are combined and tested together) because you're running a full Android environment with background services and apps as well as your entire application. And their own documentation conflicts with that page, as you can see here they refer to the same thing as instrumented tests. I believe the correct way to refer to them is instrumented tests because their docs state:

These are tests that run on a hardware device or emulator. These tests have access to Instrumentation APIs, give you access to information such as the Context of the app you are testing, and let you control the app under test from your test code. Use these tests when writing integration and functional UI tests to automate user interaction, or when your tests have Android dependencies that mock objects cannot satisfy.

Instrumentation test is defined as either just a UI test written in Espresso or UIAutomator or a special test execution environment as launched via the am instrument command depending on what you read. I believe the AOSP docs have the most correct definition and Firebase has a typo (I filed a support ticket to fix this).

Summary

Android's test folders just split up functionality in running tests against a stubbed android.jar vs an actual Android environment. The names used in the documentation could be better.