Naming attachments after their test case

Name a screenshot, recording, or log after the test case that produced it, and Bitrise shows it under that test on the Tests tab.

Step versions

This requires the Deploy to Bitrise.io Step version 2.26.0 or newer. If you use the Export test results to Test Reports Step, it must be version 1.3.0 or newer.

You need to:

• Use the correct filename structure.
• Set up your test tool to name the attachments correctly.
• Export the attachments with a Bitrise Step.

About the filename structure

The filename should have the following structure:

<classname>__<name>[__run<k>]__<label>.<ext>

For example, com.example.LoginTest__emptyState__1.png.

To get the correct filename:

• Copy classname and name from the <testcase> element of your JUnit XML, and replace any of the " * / : < > ? \ | characters with _. Bitrise applies the same replacement to the names in the JUnit XML, so either spelling matches.
• Always add a label, such as a counter. The label can't contain __ or start with _.
• If the same tests run on more than one device, put the device in the label, for example api34_1. Otherwise, files with the same name are skipped, and the attachments don't appear.
• To attach a file to a later run of the same test case, add __run2, __run3, and so on before the label.

Setting up your test tool

Set up the naming once, in your test configuration. The following examples do this for some popular test tools:

Espresso

Register a RunListener that takes a screenshot when an Espresso check fails:

// app/build.gradle.kts
android {
    defaultConfig {
        testInstrumentationRunnerArguments["listener"] = "com.example.testing.BitriseScreenshotListener"
        testInstrumentationRunnerArguments["useTestStorageService"] = "true"
    }
}

dependencies {
    androidTestUtil("androidx.test.services:test-services:1.6.0")
}

// app/src/androidTest/java/com/example/testing/BitriseScreenshotListener.kt
package com.example.testing


private fun clean(value: String) = value.replace(Regex("""["*/:<>?\\|]"""), "_")

class BitriseScreenshotListener : RunListener() {
    override fun testStarted(test: Description) {
        val default = DefaultFailureHandler(InstrumentationRegistry.getInstrumentation().targetContext)
        Espresso.setFailureHandler { error, viewMatcher ->
            val screen = Resources.getSystem().displayMetrics.run { "${widthPixels}x$heightPixels" }
            val name = "${clean(test.className)}__${clean(test.methodName)}__api${Build.VERSION.SDK_INT}_${screen}_failure"
            takeScreenshot().writeToTestStorage(name)
            default.handle(error, viewMatcher)
        }
    }
}

Plain JUnit assertions don't get a screenshot.

Paparazzi

Create your Paparazzi rules through a shared function that adds a snapshot handler. The handler saves every snapshot under the attachment name as well:

// src/test/java/com/example/testing/BitrisePaparazzi.kt
package com.example.testing


private fun clean(value: String) = value.replace(Regex("""["*/:<>?\\|]"""), "_")

fun bitrisePaparazzi(deviceConfig: DeviceConfig = DeviceConfig.PIXEL_5) =
    Paparazzi(deviceConfig = deviceConfig, snapshotHandler = BitriseSnapshotHandler())

class BitriseSnapshotHandler(
    maxPercentDifference: Double = 0.1,
    private val delegate: SnapshotHandler =
        if (System.getProperty("paparazzi.test.verify") == "true") SnapshotVerifier(maxPercentDifference)
        else HtmlReportWriter(),
) : SnapshotHandler by delegate {
    override fun newFrameHandler(snapshot: Snapshot, frameCount: Int, fps: Int): SnapshotHandler.FrameHandler {
        val frames = delegate.newFrameHandler(snapshot, frameCount, fps)
        val test = snapshot.testName
        val label = snapshot.name?.replace(Regex("_+"), "_")?.removePrefix("_") ?: "snapshot"
        val name = "${clean("${test.packageName}.${test.className}")}__${clean(test.methodName)}__$label"
        val file = File(System.getProperty("paparazzi.build.dir"), "bitrise-snapshots/${System.getProperty("bitrise.test.task")}/$name.png")
        return object : SnapshotHandler.FrameHandler by frames {
            override fun handle(image: BufferedImage) {
                file.parentFile.mkdirs()
                ImageIO.write(image, "png", file)
                frames.handle(image)
            }
        }
    }
}

In your tests, replace Paparazzi(...) with bitrisePaparazzi(...). Then add the following to the module's build.gradle.kts. The first block lets the handler use java.awt on Android Gradle plugin 8, and the second one gives each test task its own folder, removes the snapshots of the previous run, and keeps the current ones in the build cache:

tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinJvmCompile>().configureEach {
    if ("UnitTest" in name) compilerOptions.noJdk.set(false)
}

tasks.withType<Test>().configureEach {
    val snapshots = layout.buildDirectory.dir("bitrise-snapshots/$name")
    systemProperty("bitrise.test.task", name)
    outputs.dir(snapshots)
    doFirst { snapshots.get().asFile.deleteRecursively() }
}

The example is for Paparazzi 1.3.5. On Paparazzi 2.0, create the report writer with HtmlReportWriter(maxPercentDifference = maxPercentDifference).

Roborazzi

Copy Roborazzi's screenshots under the attachment name after each Roborazzi run:

// app/build.gradle.kts
val bitriseScreenshots by tasks.registering(Sync::class) {
    from(layout.buildDirectory.dir("outputs/roborazzi")) { include("*.png") }
    into(layout.buildDirectory.dir("bitrise-screenshots"))
    rename("""(.+)\.([^.]+?)(?:_(\d+))?(_actual|_compare)?\.png""", "$1__$2__screenshot$3$4.png")
}
tasks.named { it == "finalizeTestRoborazziDebug" }.configureEach { finalizedBy(bitriseScreenshots) }

Roborazzi writes screenshots only when you run recordRoborazziDebug, verifyRoborazziDebug, or compareRoborazziDebug. Parameterized tests don't get attachments.

Detox

Point artifacts.pathBuilder in your Detox configuration to this file. It names every screenshot, video, and log the way jest-junit names the test with its default settings:

// e2e/bitrisePathBuilder.js
const path = require('path');

const clean = (value) => value.replace(/["*/:<>?\\|]/g, '_');

module.exports = class BitrisePathBuilder {
  constructor({ rootDir }) {
    this.rootDir = rootDir;
  }

  buildPathForTestArtifact(artifactName, testSummary) {
    if (!testSummary) return path.join(this.rootDir, artifactName);

    const { title, fullName } = testSummary;
    const key = clean(`${fullName.slice(0, -title.length).trim()} ${title}`);
    const ext = path.extname(artifactName);
    const label = path.basename(artifactName, ext).replace(/_+/g, '_').replace(/^_/, '');
    return path.join(this.rootDir, `${key}__${key}__${label}${ext}`);
  }
};

Playwright

Turn on includeProjectInTestName for the junit reporter, and add a reporter that copies the screenshots and videos of each test under the attachment name:

// playwright.config.ts
reporter: [
  ['junit', { outputFile: 'test-results/junit.xml', includeProjectInTestName: true }],
  ['./bitrise-reporter.ts'],
],
use: { screenshot: 'only-on-failure', video: 'retain-on-failure' },

// bitrise-reporter.ts

const clean = (value: string) => value.replace(/["*/:<>?\\|]/g, '_');
const supported = /\.(png|jpe?g|mp4|webm|ogg|txt|log)$/i;

export default class BitriseAttachments implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    const [, project, file, ...titles] = test.titlePath();
    const key = `${clean(file)}__${clean((project ? `[${project}] ` : '') + titles.join(' › '))}`;
    const files = result.attachments.filter((a) => a.path && supported.test(a.path));
    files.forEach((a, i) => {
      const label = (result.retry ? `retry${result.retry}_` : '') + (i + 1);
      const target = path.join('test-attachments', `${key}__${label}${path.extname(a.path!)}`);
      fs.mkdirSync(path.dirname(target), { recursive: true });
      fs.copyFileSync(a.path!, target);
    });
  }
}

Exporting the attachments

Step versions

This requires the Deploy to Bitrise.io Step version 2.26.0 or newer. If you use the Export test results to Test Reports Step, it must be version 1.3.0 or newer.

The Deploy to Bitrise.io Step is always required: it deploys the correctly named attachments alongside your test results. What else you need depends on the Step that runs your tests:

• Official Bitrise testing Steps: No further configuration is needed. The Deploy to Bitrise.io Step exports the attachments together with the test results. For the list of Steps, see Deploying results from the official Bitrise testing Steps.
• Other Steps, such as Script or npm Steps: Add the Export test results to Test Reports Step after the Step that runs your tests. It finds the attachments and moves them to the location the Deploy to Bitrise.io Step deploys from. Set its Test result base path input to a folder that contains both your test results and your attachments, for example app/build/. See Using the Export test results to Test Reports Step.

Troubleshooting missing attachments

If an attachment doesn't show up, check the build log:

• The Export test results to Test Reports Step logs a warning for names that match more than one test or file, or that have an invalid __run number or no label. Other skipped files appear only with verbose logging: set its Enable verbose logging? input to yes.
• The Deploy to Bitrise.io Step logs a warning for every file in the report that it can't link to a test case.