Continuous Integration

Overview

NUbots uses Buildkite for its continuous integration pipeline. The Buildkite server listens for code changes on GitHub and issues build jobs to our build agents (the iMacs in the lab), as they become available.

This CI process works in combination with the Git Feature Branch workflow we use for development:

1. Developers branch off main, creating a new branch with a name of the form <lastname>/<purpose-of-branch>. For example:

   Copy
   git checkout -b biddulph/ball-detector

2. Once development on the branch is complete, or reaches a point where it can be reviewed, the developer makes a pull request (PR) on GitHub. This triggers a webhook to Buildkite to build and test the PR.

3. Buildkite receives the webhook and creates separate build jobs for different parts of the PR (build, check code format, test, etc) and sends different jobs to different agents based on their availability.

4. Agents receive a job, clone the code, pull the existing main image for DockerHub, and build any new Docker changes. They then mount the code and run the job in a container based on the image, reporting the results back to Buildkite.

5. Buildkite receives the results and updates the build status in the Buildkite UI as well as on the PR page on GitHub.

6. When the build passes, the PR is reviewed by another member of the team. If approved, the PR can be merged into main.

7. Merging into main then triggers another CI build. This time, in addition to building the code, a separate job is created to rebuild the main image and push it to DockerHub. This job is handled by agents that have the DockerHub login credentials, indicated by the dockerhub=true tag on the agent.

CI Overview PRCI Overview Master

The Build Pipeline

The build pipeline is configured in the .buildkite/pipeline.yml file in the NUbots repo. This file specifies a number of "build steps", which are separate jobs such as "Build for NUC" and "Validate C++ and Protobuf formatting".

To add, remove, or change build jobs, we edit the pipeline file. The Buildkite pipeline documentation has everything you need to know to make changes.

Monitoring and Troubleshooting Builds

Buildkite has live-updating logs of each build, which you can use to monitor the build progress or troubleshoot errors when a build fails.

If for example your PR build fails due to formatting errors, you can check the logs to see which files are incorrectly formatted. To fix the build, you would reformat the offending files and push a new commit, which will trigger a rebuild.

To view the logs for a build, scroll to the build checks at the bottom of the PR page and click the Details link on the buildkite/nubots/pr check.

Pacman Package Cache

When Buildkite receives a webhook to build and test a PR, the Buildkite agents pull down the code and start building a Docker image based off the Dockerfile in that branch.

In the process of building our Docker image, several pacman packages are downloaded from public mirrors (if that specific layer itself is not cached). Because these packages are downloaded numerous times, across multiple PRs, caching them locally can speed up build times and save network bandwidth.

It is also possible that the signature on a package that is located on a public mirror changes. This change of signature can cause issues for us because we download date-locked packages and still expect an older signature. Caching such packages locally may alleviate these issues.

To cache packages locally, a pacman package cache is setup on each Buildkite agent. This cache uses the nroi/flexo Docker image, and exposes itself on:

Copy
http://localhost:7878/$repo/os/$arch

In order for pacman to use this package cache, the cache must be added to /etc/pacman.d/mirrorlist. Because our build happens inside a buildx container, we cannot use localhost as the address for the cache. Instead, we alias the IP address of the host machine as host.docker.internal. Therefore, the following is added to the mirrorlist here in the Dockerfile:

Copy
Server = http://host.docker.internal:7878/$repo/os/$arch

Setting Up a New Agent (Ubuntu)

The following instructions show how to setup and configure a new Buildkite agent with Docker and Docker Compose. These steps have been collected into a single script you can download here. Before running the script, ensure you have set the Buildkite agent token and DockerHub password correctly in the env variables at the top of the script. Also, make sure to run the script with sudo.

Install Docker

Steps from https://docs.docker.com/install/linux/docker-ce/ubuntu/.

Copy
# Remove old Docker
sudo apt-get remove docker docker-engine docker.io containerd runc

# Install packages to allow apt to use a repository over HTTPS
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add Docker's stable repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
​
# Install latest Docker CE
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Install Docker Compose

Copy
# Install Python 3 and pip
sudo apt-get update
sudo apt-get install python3.6 python3-pip

# Install Docker Compose
sudo -H pip3 install docker-compose

Install and Configure Buildkite

Steps from https://buildkite.com/docs/agent/v3/ubuntu.

Copy
# Add the signed Buildkite apt repository
echo "deb https://apt.buildkite.com/buildkite-agent stable main" | sudo tee /etc/apt/sources.list.d/buildkite-agent.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 32A37959C2FA5C3C99EFBC32A79206696452D198

# Install the Buildkite agent
sudo apt-get update
sudo apt-get install -y buildkite-agent

# Add the agent token to the configuration file.
# Replace INSERT-YOUR-AGENT-TOKEN-HERE with the token from Buildkite.
sudo sed -i "s/xxx/INSERT-YOUR-AGENT-TOKEN-HERE/g" /etc/buildkite-agent/buildkite-agent.cfg

# Start the agent and configure it to run on system startup
sudo systemctl enable buildkite-agent && sudo systemctl start buildkite-agent

Configure Docker

Steps from https://docs.docker.com/install/linux/linux-postinstall/.

Copy
# Create a 'docker' user group and add local users to allow for use without sudo
sudo groupadd docker                     # Create user group
sudo usermod -aG docker nubots           # Add nubots to the group
sudo usermod -aG docker buildkite-agent  # Add buildkite-agent to the group
newgrp docker                            # Activate group changes

# Configure Docker to run on system startup
sudo systemctl enable docker

Add DockerHub Credentials to Buildkite

For agents to push built main images to DockerHub, they need the service account login credentials. The username of this account is nubotsdocker, and is specified in the Buildkite pipeline file. The password needs to be made available to the agent when it runs, via the DOCKER_LOGIN_PASSWORD env variable.

Credentials are provided to the agent via env variables using agent hooks. Specifically using the environment hook. The default path for this hook is /etc/buildkite-agent/hooks/environment.

After the credentials are added, the agent needs to be marked as having the credentials by setting the dockerhub=true tag in the agent configuration file. The default path for this file is /etc/buildkite-agent/buildkite-agent.cfg.

For more information about agent configuration and hooks, see the following pages in the Buildkite documentation:

• https://buildkite.com/docs/agent/v3/hooks
• https://buildkite.com/docs/agent/v3/configuration

Setup Pacman Package Cache

1. Create a directory to store the cache config

   Copy
   mkdir ~/pacman-cache && cd ~/pacman-cache

2. Download the install script

   Copy
   wget https://raw.githubusercontent.com/NUbots/NUbots/main/doc/PacmanCache/install.sh

3. Make the install script executable

   Copy
   chmod +x ./install.sh

4. Run the install script

   Copy
   ./install.sh