android junit report

Introduction

The Android JUnit report test runner is a custom instrumentation test runner for Android that creates XML test reports. It's the glue between your Android unit tests and the rest of your toolkit. By exporting your test results in a format mostly-compatible with that used by the AntJUnit task, the runner allows you to leverage any tools that support this format for your Android projects.

For the Impatient

Just want to get started? Refer to the quick start instructions on the project home page.

Motivation

I originally created the runner to integrate my own Android projects with our Pulse continuos integration server. There is nothing Pulse-specific about the runner, though. Hence I've open sourced it so other Android developers can benefit and (perhaps) contribute.

License

This code is licensed under the Apache License, Version 2.0. See the LICENSE file for details.

How It Works

This test runner may be used as a drop-in replacement for the standardInstrumentationTestRunner provided in the Android SDK. It is in fact an extension of that runner, supporting the same functionality with the addition of XML report generation.

To be consistent with existing SDK support for generating coverage reports, this runner outputs its XML report in the file storage area for the application under test. By default, the report file is produced at:

/data/data/<application under test package>/files/junit-report.xml

The file may be retrieved from the device using adb pull (see below for more details). You may also customise the output location, file name and even choose to produce multiple output files (one per suite). Again, details may be found below.

Note: by default only the application under test has full access to the directory where the report is stored. You will not be able, for example, to list the contents of the directory unless you have a rooted device. The runner marks the file as world-readable, though, so knowing the full path you are able to pull it from the device.

Using the Runner

Obtaining the Jar

First you will need to obtain a release jar file containing the runner. You can build from source (see below), but it's probably easier to just grab the latest jar from the downloads page. You can also grab the latest release build directly from our Pulse build server (click on "jar" in the "featured artifacts" table on the right of the page.)

Using with Ant Builds

To use this runner with an Ant build (based on the Android SDK support for Ant):

Add android-junit-report-<version>.jar to your test project's libs/directory.
Edit your test project's AndroidManifest.xml to set android:name in the<instrumentation> tag to:com.zutubi.android.junitreport.JUnitReportTestRunner.

Edit your test project's ant.properties to add the line:

test.runner=com.zutubi.android.junitreport.JUnitReportTestRunner

These steps will cause your tests to be executed with the custom runner, which will produce the junit-report.xml file in the internal storage area for the project under test. To retrieve the file from the device, you can use Ant to run an adb pull, for example:

<target name="fetch-test-report">
  <xpath input="${tested.project.dir}/AndroidManifest.xml"
         expression="/manifest/@package" output="tested.package"/>

  <echo>Downloading XML test report...</echo>
  <mkdir dir="${reports.dir}"/>
  <exec executable="${adb}" failonerror="true">
  <arg line="${adb.device.arg}"/>
    <arg value="pull" />
    <arg value="/data/data/${tested.package}/files/junit-report.xml"/>
    <arg value="${reports.dir}/junit-report.xml"/>
  </exec>
</target>

The runner comes with an example project that includes custom Ant rules intests/custom_rules.xml.

Using with Eclipse

When running within Eclipse, there is not much call for XML test reports. However, assuming you have edited your test project's manifest to specify this custom runner (as described above), you may also need to tell Eclipse to use this runner to avoid problems. You can do this by:

Adding android-junit-report-<version>.jar to the build path of your test project in Eclipse. Note with newer versions of the ADT this should be done automatically for you.
Ensuring all existing run configurations for unit tests specify an Instrumentation runner of:
```
test.runner=com.zutubi.android.junitreport.JUnitReportTestRunner
```

If you see an error about not being able to find a target package forandroid.test.InstrumentationTestRunner, it is likely you have an existing run configuration that is set to use the default runner. You can recreate any such configurations and the issue will disappear.

Customising Via Arguments

The runner supports various arguments, summarised below, to customise its behaviour.

Supported Arguments

multiFile

If set to true, a new report file is generated for each test suite. If false, a single report is generated containing all suites.

Default: false

reportDir

If specified, absolute path to a directory in which to write the report file(s). May begin with__external__, which will be replaced with the path for the external storage area of the application under test. This requires external storage to be available andWRITE_EXTERNAL_STORAGE permission in the application under test.

Default: unspecified (reports are written to the internal storage of the application under test)

reportFile

The name of the report file to generate (single file mode) or a pattern for the name of the files to generate (multiple file mode). In the latter case the string __suite__ will be substituted with the test suite name to produce the file name for each suite. See the reportDirargument for the destination of the report files.

Default: junit-report.xml in single file mode, junit-report-__suite__.xml in multiple file mode.

filterTraces

If true, stack traces in the report will be filtered to remove common noise (e.g. framework methods).

Default: true

Specifying Arguments

To specify arguments, use the -e flag to adb shell am instrument, for example:

$ adb shell am instrument -w -e reportFile my-report.xml \
  com.example.test/com.zutubi.android.junitreport.JUnitReportTestRunner

If you need to pass arguments in your Ant build, you must override the built-in run-tests target. As an example, if you prefer multiple output files (which are more compatible with some tools), you need to set the multiFile argument to true. In this case you will also need to pull the whole directory of files from the device after running your tests:

<target name="custom-test">
  <xpath input="${tested.project.dir}/AndroidManifest.xml"
         expression="/manifest/@package" output="tested.package"/>

  <echo>Running tests...</echo>
  <exec executable="${adb}" failonerror="true">
    <arg line="${adb.device.arg}"/>
    <arg value="shell"/>
    <arg value="am"/>
    <arg value="instrument"/>
    <arg value="-w"/>
    <arg value="-e"/>
    <arg value="reportDir"/>
    <arg value="__external__/my-reports"/>
    <arg value="-e"/>
    <arg value="multiFile"/>
    <arg value="true"/>
    <arg value="${tested.package}/${test.runner}"/>
  </exec>
<target>

The example project includes a sample target of this type in tests/custom_rules.xml.

Report Format

As stated above, the XML report format is based on the Ant JUnit task's XML formatter. A few caveats apply:

In the default single file mode, multiple suites are all placed in a single file under a root <testsuites> element. In multiple file mode each XML file contains a single suite.
Redundant information about the number of nested cases within a suite is omitted.
Durations are present on cases, but omitted from suites. Adding them to suites would require buffering that the runner specifically avoids due to memory usage concerns.
Neither standard output nor system properties are included.

The differences mainly revolve around making this reporting as lightweight as possible. The report is streamed as the tests run, making it impossible to, e.g. include the case count in a<testsuite> element. If required this information can be added to the reports by post-processing.

The format is intended to be "compatible enough" for integration with existing tools. In my particular case I use the reports with our own Pulse continuous integration server, so compatibility with it was my first target. If you find an incompatibility with another tool, let me know (see below) and I will see what can be done.

Building From Source

If you would like to modify the runner, or build it yourself for any other reason, you will need:

A JDK, version 1.5 or later.
The Android SDK (or at least a stub android.jar as provided in the SDK).
Apache Ant version 1.7 or later.

To run a build:

Create a file local.properties in the top-level directory. In this file, define the location of an android.jar to build against, for example:
```
android.jar=/opt/android/platforms/android-14/android.jar
```
where /opt/android is the root of an Android SDK.
Run ant in this same directory:
```
$ ant
```
The jar will be created at build/android-junit-report-dev.jar.