Detecting and quarantining flaky tests

Flaky tests block PRs even when the code is fine. A single transient failure forces you to dig through logs, hit rerun, and wait, slowing merges and masking real regressions.

Detecting flaky tests

Configure specific build Steps to automatically rerun any failed tests. If the tests pass on retry, the build can go green, and the tests are marked flaky, so you can fix it later. You can also quarantine it if you have an Enterprise plan.

Steps allowing automatic reruns for failed tests

The inputs in these testing Steps allow you to specify the number of retry attempts for failed tests. When a test fails during a run, the Step will automatically retry the tests.

If the test passes on a subsequent attempt, it is marked as flaky; however, the Step itself will still succeed, allowing your build and PR check to turn green.

This allows you to merge your changes without being blocked, while still providing the data you need to identify and fix unstable tests later.

For example, a UI test might occasionally fail because a network request times out or an animation hasn't completed. Instead of failing the entire build, you can now configure it to retry failed tests up to 3 times. If it passes on the second try, your PR is unblocked, and you can see in the test report that this specific test was flaky.

Note that if you use either the iOS Device Testing Step or the Virtual Device Testing for Android Step, then the entire test suite will be rerun, and not just the tests that failed.

Configuring flaky test detection

Configure flaky test detection in the input fields of these testing Steps:

Xcode Test for iOS and Xcode Test without building

Set below input fields under Test repetition in the Step:
- Test Repetition Mode: Defines the repetition mode. For example, retry_on_failure will only rerun the tests that failed.
- Maximum Test Repetitions: The maximum number of times to repeat a test.
- Relaunch Tests For Each Repetition: It controls whether the app is relaunched for each repetition.
iOS device testing and Virtual device testing for Android

Set below input field in the Step:
- Number of times a test execution is reattempted: Specifies the number of times to retry a failed test execution. An execution that fails initially but succeeds on a reattempt is reported as flaky. The maximum value is 10. The default is 0 (no reruns).
  
  Running tests with Virtual Device Testing for Android and iOS Device Testing Steps
  
  Both Steps will always rerun all tests (successful and failed) as many times as is configured in the input.
Android Unit Test

This functionality is configured directly through the Gradle plugin. You can enable retries for failed tests within your project's build.gradle file.

Quarantining flaky tests

It is possible to quarantine flaky tests that disrupt your builds to run until you can fix the issue. The above mentioned Steps will automatically skip the quarantined tests as you specify some quarantine details.

Environment Variable for flaky test detection

Testing Steps now automatically populate BITRISE_FLAKY_TEST_CASES when retries are enabled, identifying which specific tests are behaving inconsistently.

When any of these Steps run with retries enabled, they automatically track individual test results:

Xcode Test for iOS (ID: xcode-test)
Android Unit Test (ID: android-unit-test)
iOS Device Testing (ID: virtual-device-testing-for-ios)
Virtual Device Testing for Android (ID: virtual-device-testing-for-android)

These Steps populate the BITRISE_FLAKY_TEST_CASES Env Var with tests that failed at least once but also passed at least once during retry attempts.

The BITRISE_FLAKY_TEST_CASES Env Var contains a newline-separated list of flaky test identifiers:

- TestTarget_1.TestClass_1.TestMethod_1
- TestTarget_1.TestClass_1.TestMethod_2  
- TestTarget_1.TestClass_2.TestMethod_1
- TestTarget_2.TestClass_1.TestMethod_1

Examples of when to use the flaky test Env Var

Here are a couple of use cases when using this Env Var comes handy:

The test fails if flaky tests are detected:

steps:
...
- script:    
    run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'    
    inputs:    
     - content: |-        
     #!/bin/env bash        
     echo "Build failed due to flaky tests:"        
     echo "$BITRISE_FLAKY_TEST_CASES"       
     exit 1

To send Slack notification if flaky tests are detected:

steps:
...
- slack:    
    run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'    
    inputs:    
    - webhook_url: $SLACK_WEBHOOK_URL    
    - text: |-        
        ⚠️ Flaky tests detected in build $BITRISE_BUILD_NUMBER:       
        $BITRISE_FLAKY_TEST_CASES

Creates a GitHub issue automatically:

steps:
...
- script:    
    run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'    
    inputs:    
    - content: |-       
      #!/bin/env bash        
      ISSUE_BODY="Flaky tests detected in build 
$BITRISE_BUILD_NUMBER:\n\n\`\`\`\n$BITRISE_FLAKY_TEST_CASES\n\`\`\`"        
      curl -X POST \          
       -H "Authorization: token $GITHUB_TOKEN" \          
       -H "Accept: application/vnd.github.v3+json" \         
       https://api.github.com/repos/owner/repo/issues \          
       -d "{\"title\":\"Flaky tests - Build $BITRISE_BUILD_NUMBER\",\"body\":\"$ISSUE_BODY\"}"

You can also log to the analytics service, save to build artifacts, count and categorize by test target, and set the Environment Variable for downstream Steps.

Test results and outcomes after retries

If a test fails, the Step reruns the tests up to the configured limit. If a previously failing test passes on retry, it’s marked flaky in the Bitrise Tests tab. If tests still fail after retries, the Step fails as usual.

If all tests pass after retry, the build passes and flaky tests get highlighted in the new test report. If any test fails after retries, the Step fails and the build status says FAILED.

Flaky test case with failed and successful runs

Even if a test case has successful and failed runs, it will be marked as flaky. The number and order of status do not matter. For example: 1 failed, 1 success, 1 failed is marked as flaky as well as 1 success and 2 failed runs (in this order).

Quarantining flaky tests

This feature lets you temporarily isolate problematic tests without disrupting your entire test suite. You can mark flaky tests as quarantined so Bitrise will skip them in future builds.

Quarantined tests are tracked in a separate Test Quarantine tab and will continue being skipped from your builds until you remove them from quarantine.

Use of Test Quarantine is limited

Please note that this feature is only available with the Enterprise plan. Additionally, only an Admin, Platform Engineer and a Developer can have full read, write and edit access to quarantined tests. Testers and QA people have read-only access.

It’s recommended to quarantine tests if, for example:

A test passes locally but fails intermittently in CI.
You've identified a flaky test, but don't have time to fix it immediately.
You want to maintain green builds while investigating test stability issues.
You need to prevent automatic reruns from consuming build minutes on known problematic tests.

Adding tests to quarantine

You can quarantine tests from the Tests tab or from Insights to prevent unexpected build failures:

From the Tests tab

Navigate to the Flaky tab on the Tests tab.
Click Add to quarantine next to any flaky test.
Define the quarantine scope (workflows, pipelines, or branches) by clicking Edit quarantine scope.
Add an optional comment for future reference.

From Insights

Go to Insights → Tests → Flaky tests → Test cases.
Apply the Project filter.
Click Add to quarantine next to any test in the breakdown view.
Configure the quarantine scope and conditions by clicking Edit quarantine scope.
Submit to exclude the test from future runs.

Defining the scope of quarantine

By default, quarantine applies to all builds on your project, unless you specify otherwise. Leave fields empty to skip the test on all branches and all targets (workflows and pipelines).

You can limit the scope by branches and targets:

Branch (optional): enter space-separated branch patterns with wildcard support (*). Examples: main , release-*, feature/auth-*
Targets (optional): enter space-separated workflow and pipeline names. Examples: ci, nightly-tests, release-pipeline

Combining branches and target to narrow the scope

If you set both branches and targets, the test is quarantined only when both the branch AND the target match. Keep patterns tight to avoid silencing too much. Prefer release-* over * and target the specific workflow that runs the flaky test.

You can use the Comment field to note the quarantine reason and include a ticket link so you remember why you quarantined the test.

You can edit or remove the scope later from the quarantine dashboard.

Managing quarantined tests

Access the quarantine dashboard from the Test Quarantine tab on your CI home page.

Accessing Quarantined tests

Please note that only an Admin, Platform Engineer and a Developer can have full read, write and edit access to quarantined tests. Testers and QA people have read-only access.

The dashboard shows:

Test name and module path.
The build where the test was quarantined.
Timestamp and user who quarantined it.
Direct links to insights for each test.

Quarantined tests that were not executed in a build will not show up on your Tests tab. Quarantined tests remain in the dashboard until you remove them from the quarantine.

We recommend that you scope narrowly; quarantine on the branches/workflows where the flakiness occurs. It is also helpful to leave a comment for future cleanup and set regular reviews for long-lived quarantines and fixed tests.

Environment Variable for quarantined tests

During builds, Bitrise injects the scoped list of quarantined tests into the $BITRISE_QUARANTINED_TESTS_JSON Environment Variable. Supported Steps read this variable to skip those tests and exclude them from automatic reruns. If you use custom Steps or direct runner calls, parse $BITRISE_QUARANTINED_TESTS_JSON and filter your test runner arguments accordingly. Here is an example of the value in the Env Var:

Here is the value of the $BITRISE_QUARANTINED_TESTS_JSON Env Var:

BITRISE_QUARANTINED_TESTS_JSON = "[
  {
     testCaseName: 'EnableQuarantiningDisabledByPlan',
     testSuiteName: ['UITests', 'CI', 'Settings'],
     className: 'BuildSettings'
  },
  {
     testCaseName: 'EnableQuarantiningDisabledByLDFlag',
     testSuiteName: ['UITests', 'CI', 'Settings'],
     className: 'BuildSettings'
  }
  ...
]"

Steps supporting quarantined tests

The following steps will automatically skip the quarantined tests within the specified scope. If you use custom steps or direct runner calls, parse $BITRISE_QUARANTINED_TESTS_JSON and filter your test runner arguments accordingly.

In this section:

Detecting and quarantining flaky tests

Detecting flaky tests

Steps allowing automatic reruns for failed tests

Configuring flaky test detection

Running tests with Virtual Device Testing for Android and iOS Device Testing Steps

Quarantining flaky tests

Environment Variable for flaky test detection

Examples of when to use the flaky test Env Var

Test results and outcomes after retries

Flaky test case with failed and successful runs

Quarantining flaky tests

Use of Test Quarantine is limited

Adding tests to quarantine

From the Tests tab

From Insights

Defining the scope of quarantine

Combining branches and target to narrow the scope

Managing quarantined tests

Accessing Quarantined tests

Environment Variable for quarantined tests

Steps supporting quarantined tests

Search results