In Gitlab CI: Up and running we focused on a simple CI flow (I would recommend reading this first if you’re new to Gitlab CI).
The same project was responsible for building, testing and deploying.
Sometimes, your release flow may involve different projects living in other gitlab repositories (end to end tests, deployment…).
In this blog post we will see how to setup a Gitlab CI flow that handle multiple repositories.

As for the first example, we will do a step by step guide.

Steps

Firstly we will re-use the existing CI YML that we used in the last article and add a pipeline trigger to our second project CI.
Then, we will setup our second project to run specific jobs depending on the variables passed by our first pipeline.
Finally, we will show a full setup to show re-entrance of a multi-pipeline CI flow (i.e we will have 2 pipeline triggering each other).

Step 1: Triggering a second project

Pipeline triggers

A pipeline trigger is a programmatic way to start a piepline on a project. In this tutorial we will rely on it to have a multi-project pipeline as well as a semi re-entrance setup.

Projects setup

Project 1

Our first project will have the following .gitlab-ci.yml:

image: alpine

stages:
  - build
  - test
  - trigger_deploy_ci

build:
  stage: build
  script:
    - echo

test:
  stage: test
  script:
    - echo

trigger_deploy_ci:
  stage: trigger_deploy_ci
  script:
    - apk update && apk add curl
    - curl -s -X POST -F token=$CI_JOB_TOKEN -F ref=master -F "variables[TRIGGERER_PIPELINE_ID]=${CI_PIPELINE_ID}" https://gitlab.com/api/v4/projects/9857328/trigger/pipeline | tee res.json
  artifacts:
    paths:
      - res.json

As you might see our CI is just a skeleton and does not actually do anything on the build and test stage. What is important is the last job: trigger_deploy_qa. This job will be responsible for triggering the second project pipeline.

Let’s decompose the curl command:

curl \
    -s \
    -X POST \
    -F token=$CI_JOB_TOKEN \
    -F ref=master \
    -F "variables[TRIGGERER_PIPELINE_ID]=${CI_PIPELINE_ID}" \
    https://gitlab.com/api/v4/projects/9857328/trigger/pipeline \
    | tee res.json

We are using the Job token. This token allows the job to trigger another project pipeline (if the user responsible for the first pipeline as the right to trigger pipeline on the second project). (You can see all the available environment variable here)
We want to run the triggered pipeline on the master branch
We pass a variable TRIGGERER_PIPELINE_ID to the second pipeline with the value of our pipeline ID (will be used for the re-entrance)
The URL is the one of our second project (see below) (our second project ID is 9857328)
We save the curl output (result from gitlab) to a file name res.json

We also leverage gitlab artifacts to save the res.json file (will be used for the re-entrance).

Project 2

On the second project:

image: alpine

stages:
  - deploy_ci

deploy_ci:
  stage: deploy_ci
  script:
    - echo
  only:
    refs:
      - pipelines

This setup is really simple, we just execute a single job (its also a skeleton but let’s pretend that it deploys the CI env).

By using the only directive we are limiting our pipeline to only triggers from other pipeline. This will never get run for a commit, a tag or any other event that would normally start a pipeline.

CI results on Project 1

This is the resulting CI on Project 1. First CI

This is a pretty good start, we now have a CI env deploy on another project.

You need to automatically test your CI environment to see if there are any regression.
With Gitlab CI you have two ways to do it:

Have another project that contains your end to end tests that will get trigger to test the newly deployed CI env
Find a way to “re-enter” the first CI and run end to end test from here

In this tutorial, we are going to focus on the second one.

Step 2: Re-entrance

Having a way to re-enter the first CI will allow us to chain multiple events:

We build our app (Project 1)
We deploy it on the CI environment (Project 2)
We test the newly deployed env (Project 1)
We deploy it to the next environment (Project 2)

Here is a graph of what we will have at the end.

First CI

Remarkably, the flow automatically stops if any of its steps fails.
For instance, if your end to end tests fail it will not automatically deploy the second environment. But you can bypass them by running the manual job yourself. This is a good solution to quickly bypass faulty tests and still deploy your app.

Pipeline setup

First we need to create an API key that will be used by our pipelines to call the Gitlab API.
Go to User Settings / Access Tokens and create token with the scope api.

Then we will put in our CI secret variables.
In each project go to Settings / CI/CD / Environment variables and create the variable GITLAB_API_KEY with the token created earlier.

Project 1

image: alpine

stages:
  - build
  - test
  - trigger_deploy_ci
  - test_e2e
  - trigger_deploy

build:
  stage: build
  script:
    - echo

test:
  stage: test
  script:
    - echo

trigger_deploy_ci:
  stage: trigger_deploy_ci
  script:
    - apk update && apk add curl
    - curl -s -X POST -F token=$CI_JOB_TOKEN -F ref=master -F "variables[TRIGGERER_PIPELINE_ID]=${CI_PIPELINE_ID}" https://gitlab.com/api/v4/projects/9857328/trigger/pipeline | tee res.json
  artifacts:
    paths:
      - res.json

test_e2e:
  stage: test_e2e
  script:
    - echo 3
  when: manual
  allow_failure: false

trigger_deploy:
  stage: trigger_deploy
  script:
    - apk add curl jq
    - cat res.json
    - >
        GITLAB_MANUAL_JOB=$(curl -s --header "PRIVATE-TOKEN: $GITLAB_API_KEY" "https://gitlab.com/api/v4/projects/9857328/pipelines/$(cat res.json | jq '.id')/jobs" | jq -r 'map(select(.status == "manual"))[].id');
        echo $GITLAB_MANUAL_JOB;
        curl -s -X POST --header "PRIVATE-TOKEN: $GITLAB_API_KEY" "https://gitlab.com/api/v4/projects/9857328/jobs/${GITLAB_MANUAL_JOB}/play" | jq;
  dependencies:
    - trigger_deploy_ci

We added:

a manual job test_e2e, responsible to test the environment that our other project deploys. It’s the responsability of the second project to start this job when the deploy is done (see below)
a trigger_deploy job that will play a manual job on the second project (this is to deploy the next env if the first one passes our tests)

Project 2

image: alpine

stages:
  - deploy_ci
  - post_deploy_ci
  - deploy

deploy_ci:
  stage: deploy_ci
  script:
    - echo
  only:
    refs:
      - pipelines

post_deploy_ci:
  stage: post_deploy_ci
  script:
    - echo 2
    - apk add curl jq
    - echo "${TRIGGERER_PIPELINE_ID}"
    - >
      GITLAB_MANUAL_JOB=$(curl -s --header "PRIVATE-TOKEN: $GITLAB_API_KEY" "https://gitlab.com/api/v4/projects/9857321/pipelines/${TRIGGERER_PIPELINE_ID}/jobs" | jq -r 'map(select(.status == "manual"))[].id');
      echo $GITLAB_MANUAL_JOB;
      curl -s -X POST --header "PRIVATE-TOKEN: $GITLAB_API_KEY" "https://gitlab.com/api/v4/projects/9857321/jobs/${GITLAB_MANUAL_JOB}/play" | jq;
  only:
    refs:
      - pipelines

deploy:
  stage: deploy
  script:
    - echo 2
  only:
    refs:
      - pipelines
  when: manual
  allow_failure: false

We added:

a post_deploy_ci that will be run if the deploy_ci job succeed, it will be responsible to start the manual job on Project 1 (it will starts the test_e2e job)
a deploy that will be run only if: the deploy of the CI environment passes and the end to end tests passes

Result

How it works

Project 1 is responsible for starting the pipeline on Project 2 and does so in the trigger_deploy_ci jobs.
It saves the output of the trigger call to a file It then “waits” on the job test_e2e for a manual action
Project 2 gets triggered by Project 1 (trigger_deploy_ci) and deploy the CI environment.
It “plays” the test_e2e job of Project 1. It “waits” on the job deploy for a manual action
Project 1 job’s test_e2e gets played and when it succeed the trigger_deploy “plays” the job deploy on Project 2
Project 2 job’s deploy get played

Conclusion

This is a simple example but with some simple tweak you could use it in your projects.

Gitlab CI: Multi-project pipelines

Steps

Step 1: Triggering a second project

Pipeline triggers

Projects setup

Project 1

Project 2

CI results on Project 1

Step 2: Re-entrance

Pipeline setup

Project 1

Project 2

How it works

Conclusion