Skip to main content

Pipelines with stages

Abstract

Pipelines with stages are the older way of configuring Pipelines. We recommend switching to the new way where you can define Workflow dependencies to create more flexible configurations, and you no longer have to waste time waiting for a stage to finish.

Pipelines with stages are the older way of configuring Pipelines. We recommend switching to the new way where you can define Workflow dependencies to create more flexible configurations, and you no longer have to waste time waiting for a stage to finish.

You can convert a Pipeline with stages into the new format: Converting a Pipeline with stages into a graph Pipeline.

However, if you need to, you can keep using Pipelines with stages. Such Pipelines have three main building blocks:

  • Steps: Blocks of script execution, each defining a single task in a CI/CD process.

  • Workflows: Collections of Steps. When a build of an app is running, the Steps will be executed in the order that is defined in the Workflow.

  • Stages: Collections of Workflows. A Stage can contain multiple Workflows which all run in parallel in the same Stage. If all Workflows are successful in a Stage, the Pipeline moves on to the next Stage. If any of the Workflows fail, the Pipeline ends without running the other Stages unless you configure a given Stage to always run.

Configuring a Pipeline with stages

Configuring a Pipeline that contains stages is only possible by directly editing the bitrise.yml file. You can create and modify Workflows in the graphical Workflow Editor but you need to define Pipelines and Stages in YAML format.

Defining Pipelines with stages

The bitrise.yml file contains the full configuration of your Pipelines. As with all bitrise.yml files, first you need to define the format version and the project type.

---
format_version: '8'
default_step_lib_source: 
project_type: android

This is a bare minimum bitrise.yml configuration. To define your Pipelines, you will need to use the pipelines attribute.

pipelines:  
  pipeline-successful:
    stages:
    - stage-successful-1: {}
    - stage-successful-2: {}
    - stage-successful-3: {}

In this example, we have a Pipeline called pipeline-successful, with three Stages that will run consecutively. This means that if stage-successful-1 finishes successfully, stage-successful-2 starts. If any of the Stages fail, the subsequent Stage will not start: instead, the Pipeline will be aborted and marked as failed.

Each Stage has to be defined separately under the stages attribute. Defining a Stage means specifying the Workflows that are part of the Stage.

stages:
  stage-successful-1:
    workflows:
    - test-1: {}
  stage-successful-2:
    workflows:
    - build-1: {}
    - build-2: {}
  stage-successful-3:
    workflows:
    - deploy-1: {}
    - deploy-2: {}

In this example, the Stages run the test-1, build-1, build-2, deploy-1, and deploy-2 Workflows.

Configuring a Stage to always run

By default, if a Stage fails - because one of its Workflows failed -, any other subsequent Stages of the Pipeline will not run. However, you can configure your Pipeline to run certain Stages unless the Pipeline is aborted.

To do so, you just need to set the should_always_run attribute of the Stage to true:

stages:
  stage-always-run-successful-1:
    should_always_run: true
    workflows:
    - deploy-1: {}
    - deploy-2: {}

In the example above, the Stage called stage-always-run-successful-1 will always run, regardless of the status of previous Stages. The only way these Stages will not run is if the Pipeline build is aborted by the user.

Aborting the Workflows of a failed Stage

By default, if a Workflow in a particular Stage fails, the other Workflows in the same Stage aren’t automatically aborted: these Workflows will run but the next Stage won’t start. However, you can change this behavior to immediately and automatically abort all other Workflows in the same Stage.

To do so, you need to set the abort_on_fail attribute to true:

stages:
  stage-abort-on-fail-1:
    abort_on_fail: true
    workflows:
    - deploy-1: {}
    - deploy-2: {}

Sharing Env Vars between Pipeline Stages

You can reuse any environment variable from a Workflow and reuse it in subsequent Workflows using the Share Pipeline variables Step.

Optional Workflows using run_if conditions

You can easily combine the Share Pipeline variables Step with run_if expressions to create Pipelines with optional Workflows. For more information, check out Enabling or disabling a Step conditionally.

To do so:

  1. Add the Share Pipeline variables Step to the Workflow.

  2. Optionally, you can define additional run conditions in the Additional run conditions input. The Step will only run if the conditions you specify here are true.

  3. Add the Env Var(s) you would like to use in subsequent Workflows in the Variables to share between Pipeline Workflows input.

    Using environment variable keys

    You can define Env Vars using a {key}={value} syntax. For example, MY_ENV_KEY=value, or INSTALL_PAGE_URL=$BITRISE_PUBLIC_PAGE_URL.

    If you want to use the default environment variable keys, you can use a shorthand syntax. For example, EXISTING_ENV_KEY.

    Sharing Env Vars using this Step does not override existing Env Vars defined in the app.

That's it! You can now use the Env Var in any subsequent Stage!

Configuring a Pipeline with stages

Configuring a Pipeline that contains stages is only possible by directly editing the bitrise.yml file. You can create and modify Workflows in the graphical Workflow Editor but you need to define Pipelines and Stages in YAML format.

Defining Pipelines with stages

The bitrise.yml file contains the full configuration of your Pipelines. As with all bitrise.yml files, first you need to define the format version and the project type.

---
format_version: '8'
default_step_lib_source: 
project_type: android

This is a bare minimum bitrise.yml configuration. To define your Pipelines, you will need to use the pipelines attribute.

pipelines:  
  pipeline-successful:
    stages:
    - stage-successful-1: {}
    - stage-successful-2: {}
    - stage-successful-3: {}

In this example, we have a Pipeline called pipeline-successful, with three Stages that will run consecutively. This means that if stage-successful-1 finishes successfully, stage-successful-2 starts. If any of the Stages fail, the subsequent Stage will not start: instead, the Pipeline will be aborted and marked as failed.

Each Stage has to be defined separately under the stages attribute. Defining a Stage means specifying the Workflows that are part of the Stage.

stages:
  stage-successful-1:
    workflows:
    - test-1: {}
  stage-successful-2:
    workflows:
    - build-1: {}
    - build-2: {}
  stage-successful-3:
    workflows:
    - deploy-1: {}
    - deploy-2: {}

In this example, the Stages run the test-1, build-1, build-2, deploy-1, and deploy-2 Workflows.

Configuring a Stage to always run

By default, if a Stage fails - because one of its Workflows failed -, any other subsequent Stages of the Pipeline will not run. However, you can configure your Pipeline to run certain Stages unless the Pipeline is aborted.

To do so, you just need to set the should_always_run attribute of the Stage to true:

stages:
  stage-always-run-successful-1:
    should_always_run: true
    workflows:
    - deploy-1: {}
    - deploy-2: {}

In the example above, the Stage called stage-always-run-successful-1 will always run, regardless of the status of previous Stages. The only way these Stages will not run is if the Pipeline build is aborted by the user.

Aborting the Workflows of a failed Stage

By default, if a Workflow in a particular Stage fails, the other Workflows in the same Stage aren’t automatically aborted: these Workflows will run but the next Stage won’t start. However, you can change this behavior to immediately and automatically abort all other Workflows in the same Stage.

To do so, you need to set the abort_on_fail attribute to true:

stages:
  stage-abort-on-fail-1:
    abort_on_fail: true
    workflows:
    - deploy-1: {}
    - deploy-2: {}

Sharing Env Vars between Pipeline Stages

You can reuse any environment variable from a Workflow and reuse it in subsequent Workflows using the Share Pipeline variables Step.

Optional Workflows using run_if conditions

You can easily combine the Share Pipeline variables Step with run_if expressions to create Pipelines with optional Workflows. For more information, check out Enabling or disabling a Step conditionally.

To do so:

  1. Add the Share Pipeline variables Step to the Workflow.

  2. Optionally, you can define additional run conditions in the Additional run conditions input. The Step will only run if the conditions you specify here are true.

  3. Add the Env Var(s) you would like to use in subsequent Workflows in the Variables to share between Pipeline Workflows input.

    Using environment variable keys

    You can define Env Vars using a {key}={value} syntax. For example, MY_ENV_KEY=value, or INSTALL_PAGE_URL=$BITRISE_PUBLIC_PAGE_URL.

    If you want to use the default environment variable keys, you can use a shorthand syntax. For example, EXISTING_ENV_KEY.

    Sharing Env Vars using this Step does not override existing Env Vars defined in the app.

That's it! You can now use the Env Var in any subsequent Stage!

Currently supported use cases for the iOS platform

We have prepared some Pipeline recipes based on common iOS use cases. These contain some of the most frequent tasks our users need to run.

(iOS) Run tests in parallel on multiple simulators

Description

This example uses the sample-swift-project-with-parallel-ui-test iOS Open Source sample app, which has some example Unit and UI tests and uses Test Plans to group the tests.

The example Pipeline config showcases how to run all the test cases of the project on different iOS simulators.

run_tests_on_simulators Pipeline runs two Stages sequentially:

  1. build_tests stage that runs the build_tests Workflow. This Workflow git clones the sample project and runs the xcode-build-for-test Step to build the target and associated tests. The built test bundle is transferred to the next Stage (run_tests_on_simulators) via the deploy-to-bitrise-io Step.

    The build test bundle is compressed

    xcode-build-for-test Step compresses the built test bundle and moves the generated zip to the $BITRISE_DEPLOY_DIR. That directory’s content is deployed to the Workflow artifacts by default via the deploy-to-bitrise-io Step.

    Artifact file size limitation

    There is no limitation on the number of files deployed to Artifacts per build. There is a limitation, however, on the file size which is 2GB per file.

  2. run_tests_on_simulators Stage runs three Workflows in parallel: run_tests_iPad, run_tests_iPhone, and run_tests_iPod. Both of these Workflows use the new xcode-test-without-building Step, which executes the tests based on the previous stage built test bundle. The pre-built test bundle is pulled by the _pull_test_bundle utility Workflow.

    iOS_example_modified.png

Instructions

To test the configuration in a new Bitrise example project, do the following:

  1. Visit the Create New App page to create a new App.

  2. When prompted to select a git repository, choose Other/Manual and paste the sample project repository URL (https://github.com/bitrise-io/sample-swift-project-with-parallel-ui-test) in the Git repository (clone) URL field.

  3. Confirm that this is a public repository in the resulting pop-up.

  4. Select the master branch to scan.

  5. Wait for the project scanner to complete.

  6. Select any of the offered Distribution methods (for example development, it does not really matter as now we are focusing on testing).

  7. Confirm the offered stack, skip choosing the app icon and the webhook registration and kick off the first build.

  8. Open the new Bitrise project’s Workflow Editor.

  9. Go to the bitrise.yml tab and replace the existing bitrise.yml with the contents of the example bitrise.yml file.

  10. Click the Start/Schedule a Build button, and select the run_tests_on_simulators option in the “Workflow, Pipeline” dropdown menu at the bottom of the popup.

bitrise.yml

GitHub link: https://github.com/bitrise-io/workflow-recipes/blob/main/recipes/ios-run-tests-in-parallel-on-multiple-simulators.md#bitriseyml

(iOS) Run test groups in parallel

Description

This example uses the sample-swift-project-with-parallel-ui-test iOS Open Source sample app, which has some example Unit and UI tests and uses Test Plans to group the tests.

XCode test plans

Xcode Test Plans provide a way to run a collection of tests with different test configurations. raywenderlich.com has a great tutorial on how to get started with Xcode Test Plans.

The example Pipeline config showcases how to run different test groups in parallel.

run_tests_groups Pipeline runs two Stages sequentially:

  1. build_tests Stage that runs the build_tests Workflow. This Workflow git clones the sample project and runs the xcode-build-for-test Step to build the target and associated tests. The built test bundle is transferred to the next Stage (run_tests_groups) via the deploy-to-bitrise-io Step.

  2. run_tests_groups Stage runs two Workflows in parallel: run_ui_tests and run_unit_tests. Both of these Workflows use the new xcode-test-without-building Step, which executes the tests based on the previous Stage built test bundle. The pre-built test bundle is pulled by the _pull_test_bundle utility Workflow.

    iOS_example_2.png

Instructions

  1. Visit the Create New App page to create a new App.

  2. When prompted to select a git repository, choose Other/Manual and paste the sample project repository URL (https://github.com/bitrise-io/sample-swift-project-with-parallel-ui-test) in the Git repository (clone) URL field.

  3. Confirm that this is a public repository in the resulting pop-up.

  4. Select the master branch to scan.

  5. Wait for the project scanner to complete.

  6. Select any of the offered Distribution methods (for example development, it does not really matter as now we are focusing on testing).

  7. Confirm the offered stack, skip choosing the app icon and the webhook registration and kick off the first build.

  8. Open the new Bitrise project’s Workflow Editor.

  9. Go to the bitrise.yml tab and replace the existing bitrise.yml with the contents of the example bitrise.yml file.

  10. Click the Start/Schedule a Build button, and select the run_tests_groups option in the “Workflow, Pipeline” dropdown menu at the bottom of the popup.

bitrise.yml

GitHub link: https://github.com/bitrise-io/workflow-recipes/blob/main/recipes/ios-run-test-groups-in-parallel.md#bitriseyml

Currently supported use cases for the Android platform

We have prepared some Pipeline recipes based on common Android use cases. These contain some of the most frequent tasks our users need to run. The examples include entire Workflows that can be copied and pasted for the most part.

(Android) Run UI tests in parallel on multiple devices or shards

Description

Running the UI (instrumentation) tests of a single module in parallel Workflows utilizing pipelines. You can run the tests in parallel by shards or by devices.

The Pipeline contains two Stages that are run serially:

  1. build_for_ui_testing: This Stage executes a Workflow — also named build_for_ui_testing — that runs the android-build-for-ui-testing Step to build APKs for use in testing, and runs the deploy-to-bitrise-io Step to save those APKs for use in the later Stages. Performing this Stage separately from the actual testing allows for each test Stage to use these pre-built APKs rather than having to rebuild them for each test Stage.

  2. run_ui_tests_on_devices: This Stage executes three UI test Workflows in parallel — ui_test_on_phone, ui_test_on_tablet, ui_test_on_foldable — which use the android-instrumented-test Step to run the UI tests on the APKs built in the previous Workflow on each specific device type.

    android_example.png

Instructions

To test this configuration in a new Bitrise example project, do the following:

  1. Visit the Create New App page to create a new App.

  2. When prompted to select a git repository, choose Other/Manual and paste the sample project repository URL (https://github.com/bitrise-io/Bitrise-Android-Modules-Sample.git) in the Git repository (clone) URL field.

  3. Confirm that this is a public repository in the resulting pop-up.

  4. Select the main branch to scan.

  5. Wait for the project scanner to complete.

  6. Enter app as the specified module.

  7. Enter debug as the specified variant.

  8. Continue through the prompts as normal — no changes are needed.

  9. Open the new Bitrise project’s Workflow Editor.

  10. Go to the bitrise.yml tab, and replace the existing yaml contents with the contents of the example bitrise.yml.

  11. Click the Start/Schedule a Build button, and select the ui_test_on_multiple_devices option in the Workflow, Pipeline dropdown menu at the bottom of the popup.

bitrise.yml

GitHub link: https://github.com/bitrise-io/workflow-recipes/blob/main/recipes/android-parallel-ui-tests-on-multiple-devices.md#bitriseyml

(Android) Running unit and UI tests in parallel

Description

Run unit tests and UI tests in parallel utilizing Pipelines.

This Pipeline contains one Stage — stage_unit_and_ui_test — that executes two Workflows in parallel:

  1. unit_tests: This Workflow simply runs the unit tests of the given module and variant using the android-unit-test Step.

  2. ui_tests: This Workflow builds the given module and variant using the android-build-for-ui-testing Step, spins up an emulator using the avd-manager Step, waits for the emulator to boot using the wait-for-android-emulator Step, and runs the UI tests using the android-instrumented-test Step.

    android_example2.png

Instructions

  1. Visit the Create New App page to create a new App.

  2. When prompted to select a git repository, choose Other/Manual and paste the sample project repository URL (https://github.com/bitrise-io/Bitrise-Android-Modules-Sample.git) in the Git repository (clone) URL field.

  3. Confirm that this is a public repository in the resulting pop-up.

  4. Select the main branch to scan.

  5. Wait for the project scanner to complete.

  6. Enter app as the specified module.

  7. Enter debug as the specified variant.

  8. Continue through the prompts as normal — no changes are needed.

  9. Open the new Bitrise project’s Workflow Editor.

  10. Go to the bitrise.yml tab, and replace the existing yaml contents with the contents of the example bitrise.yml.

  11. Click the Start/Schedule a Build button, and select the pipeline_unit_and_ui_test option in the Workflow, Pipeline dropdown menu at the bottom of the popup.

bitrise.yml

GitHub link: https://github.com/bitrise-io/workflow-recipes/blob/main/recipes/android-parallel-unit-and-ui-tests.md#bitriseyml

(Android) Unit test sharding by module

Description

Run the unit tests of a modularized app in parallel Workflows utilizing Pipelines.

This Pipeline contains one Stage — stage_unit_test — that executes two Workflows in parallel:

  1. unit_test_app: This Workflow runs the unit tests of the app module using the android-unit-test Step.

  2. unit_test_library: This Workflow runs the unit tests of the lib-example module using the android-unit-test Step.

    android_example_3.png

Instructions

  1. Visit the Create New App page to create a new App.

  2. When prompted to select a git repository, choose Other/Manual and paste the sample project repository URL (https://github.com/bitrise-io/Bitrise-Android-Modules-Sample.git) in the Git repository (clone) URL field.

  3. Confirm that this is a public repository in the resulting pop-up.

  4. Select the main branch to scan.

  5. Wait for the project scanner to complete.

  6. Enter app as the specified module.

  7. Enter debug as the specified variant.

  8. Continue through the prompts as normal — no changes are needed.

  9. Open the new Bitrise project’s Workflow Editor.

  10. Go to the bitrise.yml tab, and replace the existing yaml contents with the contents of the example bitrise.yml.

  11. Click the Start/Schedule a Build button, and select the pipeline_unit_test option in the Workflow, Pipeline dropdown menu at the bottom of the popup.

bitrise.yml

GitHub link: https://github.com/bitrise-io/workflow-recipes/blob/main/recipes/android-parallel-testing-unit-test-shards.md#bitriseyml

In this section:

A Step is a block of script execution that encapsulates a build task on Bitrise: the code to perform that task, the inputs and parameters you can define for the task, and the outputs the task generates.

A Workflow is a collection of Steps, Environment Variables, and other configurations. When Bitrise starts a build, it runs one or more Workflows according to the configuration defined in the bitrise.yml file.

A Stage is a collection of Workflows. Stages are the top-level building blocks of Pipelines. A Stage can contain multiple Workflows, which all run in parallel in the same Stage. If all Workflows are successful in a Stage, the Pipeline moves on to the next Stage. If any of the Workflows fail, the Pipeline ends without running the other Stages unless you configure a given Stage to always run.

A Bitrise Pipeline is the top level of the Bitrise CI/CD configuration. Pipelines can be used to organize the entire CI/CD process and to set up advanced configurations with multiple different tasks running parallel and/or sequentially.

appapp