Jenkins CI
From Linux NFS
Contents |
Jenkins Setup
Before doing anything, make sure the required environment variables are set to correct values (see the section "Required Environment Variables" below).
Begin by cloning the jenkins-nfs git repository (git clone <LOCATION>). Once you have the latest version of the scripts, cd into jenkins-nfs and run ./setup to download Jenkins, required plugins, and configure the jenkins environment. In addition, a launcher script called start_jenkins will be placed in your home directory. Jenkins is then started by running ./start_jenkins. If you ever move the jenkins.war file, be sure to update start_jenkins to reflect the new location.
Before you can use Jenkins to test changes, you must configure a git repo, a config file, and a virtual machine to run tests on. See the sections below for hints on how to do this.
Once everything is configured, you can begin a new run by starting the job "Linux - All", or by selecting any of the "Linux - Config - *" jobs.
Required Environment Variables
These need to be set before starting Jenkins
- NFS_WORKSPACE
- The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory. This directory should be accessable through NFS.
- NFS_SSH_USER
- The username of the virtual machine user that will install the compiled kernel and run the tests. SSH should be configured to avoid prompting for a password.
- NFS_SERVER
- This is the ip address of the server that makes NFS_WORKSPACE available on the network. The server is mounted to install compiled kernels and run the various tests.
- NFS_MOUNT
- The path on the NFS_SERVER to mount.
- NFS_MOUNT_POINT
- The path on the virtual machine that NFS_SERVER is mounted on.
- NFS_PYTHON_PATH
- The path to a python interpreter to use when executing python scripts. (Scripts have been written for Python 2.7)
The final mount command run will look similar to this:
mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT
Connectathon and Architectures
When adding a virtual machine, one required argument is the architecture of the processor. This is required for properly selecting the correct compiled connectathon tests. The tests are compiled with -m32 and -m64 as compiler flags and placed in the $NFS_WORKSPACE/cthon/$ARCHITECTURE directory, where $ARCHITECTURE is either "i386" or "x86_64". You can add a virtual machine with a different architecture, but you will be responsible for compiling the connectathon tests and placing them in an appropriate directory.
$NFS_WORKSPACE/scripts/functions
This file provides many convenient functions for controlling virtual machines. You can boot, halt, and reboot machines. You can also do more complicated actions like sending files, mounting the nfs server and running remote commands. Before using any of these functions, you have to tell your script what machine everything should be run on. This is done through the set_machine function. If you want to ls the $NFS_SSH_USER's home directory on the machine jenkins-tests, you could do the following:
#!/bin/bash . functions set_machine jenkins-tests boot run_cmd "ls" halt
Virtual machine requirements and initialization
There are a few requirements for a virtual machine to work correctly for the NFS Jenkins setup.
- Must have NFS client running
- Must be able to run connectathon tests
- Must be able to run xfs tests
- Passwordless sudo for NFS_SSH_USER user
- Passwordless ssh configured
The sample virtual machines come preconfigured with a valid user. Log in with:
username: jenkins password: jenkins
If you find you need root access:
username: root password: jenkins
Be sure to change the root password before hosting virtual machines publicly! You can download sample machines here:
Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm or by manually starting the job "Admin - Initialize VM". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from.
Virtual machines should be named based on the job it will be used to test (See "Adding a git repo" below).
If you intend to use NFS v3, you will have to an entry to /etc/hosts on both the client and the server so that each machine is aware of the other.
If you intend to use NFS v2, you will need to set the environment variable MNTOPTIONS so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server.
Archlinux and Fedora virtual machines are known to work properly.
Adding a git repo
Jenkins needs to be told what git repo contains the Linux source you want to compile. This is done by running $NFS_WORKSPACE/scripts/admin/add_git or by starting the job "Admin - Add Git" from the Jenkins web interface.
./add_git name git_url
$name is the name used to identify this build job. In the Jenkins web interface, this will be expanded to "Linux - $name".
$git_url is the url where the git repo can be found. For example, git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
When adding virtual machines for testing, be sure to add a "-$name" to the name of the VM. This will prevent conflicts later on, when installing or testing a kernel. If a VM is running tests, you don't want it to reboot after installing a different kernel!
Adding a Linux configuration file
Before Linux can be compiled, Jenkins needs to be told what .config files to use. This is done by running the script $NFS_WORKSPACE/scripts/admin/add_config or by starting the "Admin - Add Config" job from the Jenkins web interface.
./add_config name config_file architecture virtual_machine_name
$name is the name of this particular configuration. In the Jenkins web interface, this will be expanded to "Linux - Config - $name"
$config_file is the configuration file used when compiling. This will be copied to $NFS_WORKSPACE/config. Try not to add two configuration files with the same name.
$architecture is the architecture of the virtual machine that will run the kernel and various tests. Use "i386" for a 32-bit machine and "x86_64" for a 64-bit machine. (See "Connectathon and Architectures" for more information)
$virtual_machine_name is the name of the virtual machine that will boot the resulting kernel and run the tests. The git repo name will be appended automatically, so it is best to have one standard name for virtual machines running a specific .config file. For example, you could have the virtual machines nfs-pnfs-Trond and nfs-pnfs-Bruce for testing a pnfs .config. When running this script, enter nfs-pnfs as the virtual machine name.
Adding a new test
New tests are added through the web interface. From the home page, click "New Job" in the upper left side of the page. Enter a job name and choose the appropriate type of job. Fill out the next page with instructions for how the job should execute. It may be helpful to set a custom workspace (under "Advanced") to $NFS_WORKSPACE or some subdirectory.
When the job has been created, click on "Tests - All" and choose "Configure" on the left. Scroll down to the Post-build actions and add a new parameterized build. Enter the name of the project and choose to pass the current build parameters down to your new test.
Ideas for future work
- Rename .config file when added
- Check for conflicting names
- Prompt for environment variable values during setup?
- Create config file to source during startup / bashrc loading?
- Give VM time to shut down by itself before forcing it off
