b.stefanuk.ca

Appearance

Using GitLab CI ​

Example

Rust-based GitLab CI file starting point

In Continuous Integration/Continuous Development (CI/CD) parlance, pipelines automate the steps required to develop code. This typically involves steps that initiate code builds, run tests, deploy staging environments, and puplish to production.

Conventionally, pipelines are triggered by a code commit, after the commit tests are conducted code quality checkers can be done here and unit tests, next is build once the code is tested and integrated the code needs to be built, next the Testing stage, this stage checks the interaction between builds and ensures the app is working correctly, last is the deploy stage where the code is deployed to production. (ABOVE PARAGRAPH NEEDS FIXING)

What you need to use GitLab CI/CD:

  1. A Git codebase
  2. A YAML file in the root path of your repo named .gitlab-ci.yml

In the gitlab-ci.yml you can define scripts to run, define include and cache dependencies where you want to deploy application, etc. This file determines the structure of the pipeline, what steps to execute, and make decisions on conditions.

GitLab Python API ​

See https://python-gitlab.readthedocs.io/en/stable/api-usage.html and https://python-gitlab.readthedocs.io/en/stable/index.html

How to write a YAML file for CI/CD in GitLab ​

  1. Define the Docker image being used to run the scripts
  2. Define your stages: build, test, deploy, etc.
  • Define your image
  • Define the elements of the various stages in sequence... everything in this .gitlab-ci.yml is run in sequence.

Pro Strats ​

If it helps modularity or reusability a project's gitlab-ci.yml file can point to gitlab-ci.yml files in other projects, using the other project's pipeline files within their own. To do this, add the following items (adapted for your specific case).

yaml
include:
  - project: "<group>/<directory>/<target-project>"
    ref: "master" # this can also be a tag, which can be better practice
    file: "<custom-gitlab-ci>.yml"
include:
  - project: "<group>/<directory>/<target-project>"
    ref: "master" # this can also be a tag, which can be better practice
    file: "<custom-gitlab-ci>.yml"

You can skip the GitLab pipeline by adding [skip ci] to any commit message.

A good way of reusing code in any YAML file is the use YAML Anchors.

yaml
.reusable_script: &reusable_script
  - echo "Your custom reusable script"

build:
  script:
    - *reusable_script
.reusable_script: &reusable_script
  - echo "Your custom reusable script"

build:
  script:
    - *reusable_script

For additional details on using YAML anchors see the additional resources at the bottom of this page.

How to Setup a GitLab Runner ​

Assumption: Linux

  1. Install gitlab-runner

Add the gitlab-runner repository

sh
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

Install the GitLab runner

sh
sudo apt-get install gitlab-runner
sudo apt-get install gitlab-runner
  1. Register the runner
    • To register a group runner go to your group and select CI/CD > Runners
    • There should be a button Register a group runner with instruction on how the correctly register the runner
    • Follow the instructions
    • Select shell to run on the raw Linux machine and docker to run your pipelines in a docker image
    • You may need to update the sudoers file so that the gitlab-runner user can call commands without sudo access

Debugging the GitLab Runner ​

If your GitLab runner is a shell executor and you have root access to the device, then you can login as the gitlab-runner user (or whatever you named the user). To change the password for the GitLab runner use

sh
sudo su
passwd gitlab-runner
# Follow the prompts to provide a new password
sudo login gitlab-runner
sudo su
passwd gitlab-runner
# Follow the prompts to provide a new password
sudo login gitlab-runner

GitLab Pages: Code Coverage Reports ​

Hosting code coverage reports on GitLab pages is straightforward. To begin you have to generate the coverage report in HTML. This is typically provided in some way by your testing library or by a separate code coverage library such as lcov for C++.

To begin, you need to generate the coverage report during the test stage of your CI pipeline. Then, upload the path as an artifact in the .gitlab-ci.yml file.

yaml
test:
	stage: test
	...
    artifacts:
        when: always
        paths:
            - build/coverage/
test:
	stage: test
	...
    artifacts:
        when: always
        paths:
            - build/coverage/

Next, add a pages job to your pipeline and move the coverage report into a folder named public/. Below is a simple working job; there are additional options available.

yaml
pages:
  stage: deploy
  dependencies:
    - test
  script:
    - mv build/coverage/ public/
  artifacts:
    paths:
      - public
    expire_in: 30 days
  only:
    - master
pages:
  stage: deploy
  dependencies:
    - test
  script:
    - mv build/coverage/ public/
  artifacts:
    paths:
      - public
    expire_in: 30 days
  only:
    - master

Once the pipeline completes you should be able to access the coverage report. The URL where it was uploaded uses, by default, the following convention:

Making Releases ​

A common practice is to use a master/main branch for production code, everytime a feature branch is merged into it, the GitLab CI pipeline creates a release. The below workflow does three things:

  1. Uploads relevant packages to the package registry
  2. Create a tag for the merge commit
  3. Create an entry on the releases page

To begin, during the build stage of your pipeline, add the relevant packages and version information to a variables.env file. Ensure this is added as an artifact as a reports:dotenv field.

yaml
build:
    stage: build
    needs: []
    only:
        - main
    script:
        - *build-and-package
        - echo "PACKAGE=$PACKAGE" >> variables.env
        - echo "VERSION=$VERSION" >> variables.env
    artifacts:
        paths:
            - "build/$PACKAGE"
        reports:
            dotenv: variables.env
build:
    stage: build
    needs: []
    only:
        - main
    script:
        - *build-and-package
        - echo "PACKAGE=$PACKAGE" >> variables.env
        - echo "VERSION=$VERSION" >> variables.env
    artifacts:
        paths:
            - "build/$PACKAGE"
        reports:
            dotenv: variables.env

Next, create a release job. This job requires specific stage and image fields, and a user defined release field. Also ensure you add the job from the build stage that exports the variables.env file.

yaml
release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest # <- Alpine Linux
  needs:
    - job: "build"
      artifacts: true
  only:
    - master
  before_script: []
  script:
    - apk add curl
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file "build/$PACKAGE" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"'
  release:
    tag_name: "$VERSION"
    name: "$CI_PROJECT_NAME v$VERSION"
    description: "Release for $CI_PROJECT_NAME"
    assets:
      links:
        - name: "$PACKAGE"
          url: "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"
release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest # <- Alpine Linux
  needs:
    - job: "build"
      artifacts: true
  only:
    - master
  before_script: []
  script:
    - apk add curl
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file "build/$PACKAGE" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"'
  release:
    tag_name: "$VERSION"
    name: "$CI_PROJECT_NAME v$VERSION"
    description: "Release for $CI_PROJECT_NAME"
    assets:
      links:
        - name: "$PACKAGE"
          url: "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"

Viola! Once this is prepared, every merge time a Merge Request into master/main completes and the pipeline execuetes, a new release will be created.

Examples ​

Basic

yaml
stages:
  - lint-css
  - lint-js
  - unit-test

image: node:6.11.2

lint css:
  stage: lint-css
  before_script:
    - npm install
  cache:
    untracked: true
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js lint-css

lint js:
  stage: lint-js
  cache:
    untracked: true
    policy: pull
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js lint-js

run unit test:
  stage: unit-test
  cache:
    untracked: true
    policy: pull
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js test
stages:
  - lint-css
  - lint-js
  - unit-test

image: node:6.11.2

lint css:
  stage: lint-css
  before_script:
    - npm install
  cache:
    untracked: true
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js lint-css

lint js:
  stage: lint-js
  cache:
    untracked: true
    policy: pull
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js lint-js

run unit test:
  stage: unit-test
  cache:
    untracked: true
    policy: pull
  only:
    - master
  script:
    - ./node_modules/gulp/bin/gulp.js test

Create a tag in GitLab CI ​

If you want to create a tag and a release, the best option is to use the following example.

sh
release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    needs:
        - job: "test"
          artifacts: true
    only:
        - master
    before_script: []
    script:
        - apk add curl
        - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file "build/$PACKAGE" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"'

    release:
        tag_name: "${VERSION}"
        name: "Version ${VERSION}"
        description: "Release version ${VERSION} for ${CI_PROJECT_NAME}"
        assets:
            links:
                - name: "${PACKAGE}"
                  url: "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"
release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    needs:
        - job: "test"
          artifacts: true
    only:
        - master
    before_script: []
    script:
        - apk add curl
        - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file "build/$PACKAGE" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"'

    release:
        tag_name: "${VERSION}"
        name: "Version ${VERSION}"
        description: "Release version ${VERSION} for ${CI_PROJECT_NAME}"
        assets:
            links:
                - name: "${PACKAGE}"
                  url: "$CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/$VERSION/$PACKAGE"

Otherwise, if you only want to create a tag, use something like

sh
tag job:
  stage: tag
  only:
    - master
  needs:
    - test
  script:
    - git remote set-url origin ${PROJECT_URL}
    - git config --global user.name "${CI_USERNAME}"
    - git config --global user.email "${CI_EMAIL}"
    - git fetch origin ${CI_COMMIT_REF_NAME}
    - git checkout ${CI_COMMIT_REF_NAME}
    - DATE_VERSION=$(date +%x)
    - git tag ${DATE_VERSION}
    - git push ${DATE_VERSION}
tag job:
  stage: tag
  only:
    - master
  needs:
    - test
  script:
    - git remote set-url origin ${PROJECT_URL}
    - git config --global user.name "${CI_USERNAME}"
    - git config --global user.email "${CI_EMAIL}"
    - git fetch origin ${CI_COMMIT_REF_NAME}
    - git checkout ${CI_COMMIT_REF_NAME}
    - DATE_VERSION=$(date +%x)
    - git tag ${DATE_VERSION}
    - git push ${DATE_VERSION}

Note here that you need to create the variable ${PROJECT_URI}, this involves creating a CI user with the appropriate credentials and setting the variable earlier in the CI pipeline.

sh
PROJECT_URI: "https://${CI_GIT_USERNAME}:${CI_GIT_ACCESS_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git"
PROJECT_URI: "https://${CI_GIT_USERNAME}:${CI_GIT_ACCESS_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git"

Rust Cargo Caching ​

You can cache between pipelines by adding the cache:key: (see the gitlab-ci.yml reference). See: https://stackoverflow.com/questions/74881598/how-to-speed-up-gitlab-ci-job-with-cache-and-artifacts

ADDITIONAL RESOURCES Code Coverare with GitLab Pages