From Linux NFS
Before doing anything, make sure the required environment variables are set to correct values (see the section "Required Environment Variables" below).
Begin by cloning the jenkins-nfs git repository (git clone <LOCATION>). Once you have the latest version of the scripts, cd into jenkins-nfs and run ./setup to download Jenkins, required plugins, and configure the jenkins environment. In addition, a launcher script called start_jenkins will be placed in $NFS_WORKSPACE directory. Jenkins is then started by running ./start_jenkins. If you ever move the jenkins.war file, be sure to update start_jenkins to reflect the new location.
Before you can use Jenkins to test kernels, you must configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this.
Once everything is configured, you can begin a new run by starting the job "Compile and Install".
Required Environment Variables
These need to be set before starting Jenkins
- The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory. This directory should be accessable through NFS.
- The username of the virtual machine user that will install the compiled kernel and run the tests. SSH should be configured to avoid prompting for a password.
- This is the ip address of the server that makes NFS_WORKSPACE available on the network. The server is mounted to install compiled kernels and run the various tests.
- The path on the NFS_SERVER to mount.
- The path on the virtual machine that NFS_SERVER is mounted on.
The final mount command run will look similar to this:
mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT
Adding a new architecture
Jenkins is configured for i386 and x86_64 by default. Other architectures can be added, but manual configuration is required. You will have to modify both "Compile and Install" and "Run Tests". Select each job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.
Adding a new .config
Copy the config file to $NFS_WORKSPACE/config using a unique name. Then modify "Compile and Install" and "Run Tests" to add the new file to the CONFIG variable in the "Test Matrix".
Adding a new virtual machine
Create the new VM in virt-manager. Once it has been created, start the "Initialize VM" job to configure passwordless SSH to the VM. See the section "Virtual machine setup" below for help creating new VMs.
Adding a new NFS version
Edit the "Run Tests" job and add the new version to the list of possible values for the NFS variable. NFS v2, v3, and v4 are configured by default. Adding v4.1 will require modification of the mount command to add in the "minorversion=1" mount option.
Adding a new test
First design your test and edit $NFS_WORKSPACE/tests.map to add a mapping from name -> command. The general format of this file is:
name:where:command name: The name of the test where: Where the command will be run (either "remote" or "local") command: The command to run the test
Once the new test has been added to the tests.map file, add the new test to the "Run Tests" job.
Blacklisting a combination
Both "Compile and Install" and "Run Tests" are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values is tested. Some combination of variables may not make sense, such as running 32bit tests on a 64bit virtual machine. You can modify the "Combination Filter" field in the "Configuration Matrix" section to either white-list or black-list a specific combination using a Groovy expression. If the expression evaluates to "True", the combination will be tested.
For example, say you have the architectures "i386" and "x86_64". You also have the virtual machines "jenkins32" (a 32bit VM) and "jenkins64" (a 64bit VM). Only the combinations (x86_64, jenkins64) and (i386, jenkins32) should be run, since (i386, jenkins64) and (x86_64, jenkins32) doesn't make sense. The following line accomplishes just that:
(ARCH=="x86_64" && MACHINE=="jenkins64") || (ARCH=="i386" && MACHINE=="jenkins32")
Connectathon and Architectures
Connectathon tests are compiled for i386 and x86_64 architectures. If you plan any other architecture, you will need to manually compile connectathon yourself. Once compiled, place the resulting directory in $NFS_WORKSPACE/cthon/ARCH (where ARCH is the name of the new architecture). Be sure to add the new architecture to both "Compile and Install" and "Run Tests".
This file provides many convenient functions for controlling virtual machines. You can boot, halt, and reboot machines. You can also do more complicated actions like sending files, mounting the nfs server and running remote commands. Before using any of these functions, you have to tell your script what machine everything should be run on. This is done through the set_machine function. If you want to ls the $NFS_SSH_USER's home directory on the machine jenkins-tests, you could do the following:
#!/bin/bash . functions set_machine jenkins-tests boot run_cmd "ls" halt
Virtual machine setup
There are a few requirements for a virtual machine to work correctly for the NFS Jenkins setup.
- Must have NFS client running
- Must be able to run connectathon tests
- Must be able to run xfs tests
- Passwordless sudo for NFS_SSH_USER user
- Passwordless ssh configured
The sample virtual machines come preconfigured with a valid user. Log in with:
username: jenkins password: jenkins
If you find you need root access:
username: root password: jenkins
Be sure to change the root password before hosting virtual machines publicly! You can download sample machines here:
Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm or by manually starting the job "Admin - Initialize VM". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from.
If you intend to use NFS v3, you will have to an entry to /etc/hosts on both the client and the server so that each machine is aware of the other.
If you intend to use NFS v2, you will need to set the environment variable MNTOPTIONS so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server.
Archlinux and Fedora virtual machines are known to work properly.
Ideas for future work
- Give VM time to shut down by itself before forcing it off
- Test against multiple servers
- Add server variables to multi-config project
- Compile connectathon / xfstests on each VM?
- Catch exit codes from remote commands for error checking
- Don't run with -e so that I can still do cleanup
- Put cleanup code in bash function
- Make new function for running command, checking exit value, and exiting script if necessarily