Skip to main content

Deploying and viewing test results

View your test results in one place after a Bitrise CI build. All tests will be available on the build page, even if you have multiple testing Steps in your Workflows. Our goal is to make sure you have accessible and actionable test results on Bitrise, leading to faster time to recover from test failures and a better overall developer experience.

Deploying test resultsClick to copy link​

Deploy your test results on the build page with minimal configuration. The requirements depend on your setup:

  • Using the official Bitrise testing Steps to run tests
  • Using other Steps to run tests

Deploying results from the official Bitrise testing StepsClick to copy link​

The supported testing Steps are the following:

If you use any of these Steps, make sure you have the Deploy to Bitrise.io Step in your Workflow. Bitrise will automatically deploy your test results to the build page.

Attachments and flaky tests

If your tests generate attachments or you want Bitrise to show flaky tests, the Deploy to Bitrise.io Step must be of version 2.19.1 or newer. To link attachments to test cases by filename, it must be version 2.26.0 or newer: Naming attachments after their test case.

Deploying results from other StepsClick to copy link​

If you run your tests using other Steps (for example, you can use Script Steps for a fully custom testing solution), you need some additional configuration.

  1. Add the Step running your tests to your Workflow.

  2. Add the Export test results to the Test Reports Step to your Workflow.

  3. Configure the Step: Using the Export test results to the Test Reports Step.

  4. Make sure you have the Deploy to Bitrise.io Step in your Workflow.

    Attachments and flaky tests

    If your tests generate attachments or you want Bitrise to show flaky tests, the Deploy to Bitrise.io Step must be of version 2.19.1 or newer. To link attachments to test cases by filename, it must be version 2.26.0 or newer: Naming attachments after their test case.

Viewing test resultsClick to copy link​

Rich HTML reporting

If you use rich HTML test reports, you can still find those on the Artifacts tab.

To view your results, open the build page and select the Tests tab. Your tests are sorted into different tabs based on their status:

  • Failed
  • Passed
  • Skipped
  • Error
  • Flaky

By default, Bitrise shows the list of failed tests. You can view test run details within each of these categories.

Click any test to check the details: the duration of the test, the output, and any attachments that the test generated. For easier debugging of failures, test reports display attached image files in line with their associated test cases.

Supported file formats

Currently, we support the following file formats for attachments:

.jpg, .jpeg, .png, .txt, .log, .mp4, .webm, .ogg

You can view flaky tests on the Flaky tab. Flaky tests are tests that sometimes fail and sometimes succeed without any changes in the code. If a test fails and then succeeds on an automatic retry, Bitrise marks the test as flaky, and you can view it on the Flaky tab. It also displays the details and attachments for each retry of the test. You can find more information on how to detect flaky tests and how to quarantine them, if needed, in Detecting and quarantining flaky tests.

test-run.png

Test reports also support the use of the system-out tag in JUnit. If a test case includes this tag, we display its contents on the Test output tab. If a test case has both a system-out tag and produces an error message, we display both.

Collating test attachments with test resultsClick to copy link​

Many tests generate image files: screenshot tests, XCUITests, and so on. Bitrise can show these attachments under their associated test cases, so you can view them on the Tests tab of your build page. How to collate the test attachments with test results depends on your test tool and whether it generates an xcresult file:

Supported file formats

Currently, we support the following file formats for attachments:

.jpg, .jpeg, .png, .txt, .log, .mp4, .webm, .ogg

Naming attachments after their test caseClick to copy link​

Name a screenshot, recording, or log after the test case that produced it, and Bitrise shows it under that test on the Tests tab.

Step versions

This requires the Deploy to Bitrise.io Step version 2.26.0 or newer. If you use the Export test results to Test Reports Step, it must be version 1.3.0 or newer.

You need to:

About the filename structureClick to copy link​

The filename should have the following structure:

<classname>__<name>[__run<k>]__<label>.<ext>

For example, com.example.LoginTest__emptyState__1.png.

To get the correct filename:

  • Copy classname and name from the <testcase> element of your JUnit XML, and replace any of the " * / : < > ? \ | characters with _. Bitrise applies the same replacement to the names in the JUnit XML, so either spelling matches.
  • Always add a label, such as a counter. The label can't contain __ or start with _.
  • If the same tests run on more than one device, put the device in the label, for example api34_1. Otherwise, files with the same name are skipped, and the attachments don't appear.
  • To attach a file to a later run of the same test case, add __run2, __run3, and so on before the label.

Setting up your test toolClick to copy link​

Set up the naming once, in your test configuration. The following examples do this for some popular test tools:

Register a RunListener that takes a screenshot when an Espresso check fails:

// app/build.gradle.kts
android {
    defaultConfig {
        testInstrumentationRunnerArguments["listener"] = "com.example.testing.BitriseScreenshotListener"
        testInstrumentationRunnerArguments["useTestStorageService"] = "true"
    }
}

dependencies {
    androidTestUtil("androidx.test.services:test-services:1.6.0")
}
// app/src/androidTest/java/com/example/testing/BitriseScreenshotListener.kt
package com.example.testing

import android.content.res.Resources
import android.os.Build
import androidx.test.core.app.takeScreenshot
import androidx.test.core.graphics.writeToTestStorage
import androidx.test.espresso.Espresso
import androidx.test.espresso.base.DefaultFailureHandler
import androidx.test.platform.app.InstrumentationRegistry
import org.junit.runner.Description
import org.junit.runner.notification.RunListener

private fun clean(value: String) = value.replace(Regex("""["*/:<>?\\|]"""), "_")

class BitriseScreenshotListener : RunListener() {
    override fun testStarted(test: Description) {
        val default = DefaultFailureHandler(InstrumentationRegistry.getInstrumentation().targetContext)
        Espresso.setFailureHandler { error, viewMatcher ->
            val screen = Resources.getSystem().displayMetrics.run { "${widthPixels}x$heightPixels" }
            val name = "${clean(test.className)}__${clean(test.methodName)}__api${Build.VERSION.SDK_INT}_${screen}_failure"
            takeScreenshot().writeToTestStorage(name)
            default.handle(error, viewMatcher)
        }
    }
}

Plain JUnit assertions don't get a screenshot.

Exporting the attachmentsClick to copy link​

Step versions

This requires the Deploy to Bitrise.io Step version 2.26.0 or newer. If you use the Export test results to Test Reports Step, it must be version 1.3.0 or newer.

The Deploy to Bitrise.io Step is always required: it deploys the correctly named attachments alongside your test results. What else you need depends on the Step that runs your tests:

  • Official Bitrise testing Steps: No further configuration is needed. The Deploy to Bitrise.io Step exports the attachments together with the test results. For the list of Steps, see Deploying results from the official Bitrise testing Steps.
  • Other Steps, such as Script or npm Steps: Add the Export test results to Test Reports Step after the Step that runs your tests. It finds the attachments and moves them to the location the Deploy to Bitrise.io Step deploys from. Set its Test result base path input to a folder that contains both your test results and your attachments, for example app/build/. See Using the Export test results to Test Reports Step.

Troubleshooting missing attachmentsClick to copy link​

If an attachment doesn't show up, check the build log:

  • The Export test results to Test Reports Step logs a warning for names that match more than one test or file, or that have an invalid __run number or no label. Other skipped files appear only with verbose logging: set its Enable verbose logging? input to yes.
  • The Deploy to Bitrise.io Step logs a warning for every file in the report that it can't link to a test case.

Referencing attachments in the JUnit XMLClick to copy link​

If you can't name the files after their test cases, reference them in the JUnit XML instead:

  1. Generate a JUnit.xml file from your tests.

    The general file structure should look something like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuites time="15.682687">
        <testsuite name="Tests.Registration" time="6.605871">
            <testcase name="testCase1" classname="Tests.Registration" time="2.113871" />
        </testsuite>
    </testsuites>

  2. Add a <properties> element to associate an attachment with a testcase.

    The name attribute of the element must be set with a value attachment_# where # is the ordered index of the attachment file, and the value is the filename. Bitrise will show attachments for any test with the attachment properties, but it's most common to only attach screenshots to failed tests. Mark a test as failed with a <failure> element.

    <testcase name="testCase1" classname="Tests.Registration" time="2.113871" />
      <failure message="Assertion error message" type="AssertionError">
        Call stack printed here
      </failure>
      <properties>
        <property name="attachment_1" value="pp1.jpg" />
        <property name="attachment_2" value="pp2.jpg" />
      </properties>            
    </testcase>

  3. Add the Export test results to Test Reports Step to your Bitrise Workflow. It should be version 1.1.0 or higher.

  4. Set a test name in the The name of the test input.

  5. Set the path to your JUnit XML file in the Test result search pattern input.

  6. Add the Deploy to Bitrise.io Step to the end of your Workflow.

    You don't have to change the default input values.

Using the Export test results to Test Reports StepClick to copy link​

You can use the Export test results to Test Reports Step to make sure your test results appear on the Tests tab, even if you use Steps that don’t automatically export their results.

With the correct configuration, the Step finds the test results in your project’s repository and puts them in the export directory.

  1. Log in to Bitrise and select Bitrise CI on the left, then select your project.

  2. Click the Workflows button on the main page.

  3. Add the Export test results to Test Reports Step to your Workflow after the Step that runs your tests.

  4. In the The name of the test input, set the name of the test run.

  5. In the Test result base path input, set the path where your test results can be found.

    Managing_an_app_s_bitrise.png

    We recommend setting a folder here, though you can also set a specific filepath. The default value is the source directory of your project. Example patterns:

    • If your project’s root directory is app: app/build/test-results/testDemoDebugUnitTest/
    • If your test results are within a project folder but app is not the root directory: ./app/build/test-results/testDemoDebugUnitTest/

    To export attachments named after their test case, set a folder that contains both your test results and your attachments, for example app/build/.

  6. In the Test result search pattern input, set a pattern that matches your test result file.

    This search pattern is used to search every file and folder of the path that was set in the Path where custom test results reside input.

    If there is more than one match, the Step will export the first match with a warning in the logs. If you set a specific filepath in the previous input, just set * here.

    Example patterns:

    • Matching all files within the base path: *
    • Matching all files within a given directory of the base path: */build/test-results/testDemoDebugUnitTest/*
    • Matching a single JUnit XML when the base path is a wider folder, such as app/build/: */test-results/testDemoDebugUnitTest/TEST-com.example.ScreenshotTest.xml

  7. In the Step’s test result directory input, make sure the path is correct.

    Do NOT modify this input’s value: this is the folder where the Deploy to Bitrise.io Step will look for the test results to export them. It should be set to the $BITRISE_TEST_RESULT_DIR Env Var.

  8. Make sure you have a Deploy to Bitrise.io Step in your Workflow.

    If your tests generate attachments, make sure the Step is of version 2.19.x or newer.

See also