Jenkins CI

From Linux NFS

Revision as of 20:33, 31 January 2012

Jenkins Setup

Begin by cloning the jenkins-nfs git repository:

$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run:

$ ./setup

to download the Jenkins executable, required plugins, and to configure the Jenkins environment. In addition, a launcher script named start_jenkins will be placed in $NFS_WORKSPACE. Before starting Jenkins, edit the config/jenkins_nfs.sh file to match your desired setup (see the config/jenkins_nfs.sh section below for more information about this file). Once everything has been configured, start Jenkins by running:

$ $NFS_WORKSPACE/start_jenkins

This will start the Jenkins web server as a background process using nohup. The web server runs on port 8080 by default so you can access it by pointing your web browser to http://localhost:8080. Output from the Jenkins process will be appended to $NFS_WORKSPACE/jenkins.log.

Finally, you must also configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this. Once everything is configured, you can begin a new run by starting the job "Compile and Install" from the web interface.

The jenkins-nfs scripts do not have an official version, and instead are considered "rolling release". This means that at any point in time, you are one `git pull` away from the latest version. However, doing a plain `git pull` may cause merge conflicts with important configuraion files (like your config/jenkins_nfs.sh). The provided $NFS_WORKSPACE/update script should help with this problem. It will stash your modifications and then use `git rebase` to apply your changes on toy of the most recent patches. Additionally, the update script will check if your installed Jenkins jobs (~/.jenkins/jobs/) have been updated in git. You can either (a) accept the changes and copy over the new config file, (m) open vimdiff to manually merge the changes or (anything else) do nothing and go to the next file.

config/jenkins_nfs.sh

This script is used to set configuration variables that can affect how Jenkins behaves. This file is sourced by scripts/functions.sh, so any changes you make will not require you to restart Jenkins. Keep in mind that your changes may not be reflected in any currently running jobs. The following variables should be set:

HOST_SRV

This should be set to the ip address (or domain name) of the machine hosting Jenkins. This machine should also be hosting the Linux source code you wish to compile and test.

HOST_EXPORT

This should be set to the path on $HOST_SRV that is being exported by $HOST_SRV. For example, if you are exporting the directory /nfs/jenkins then you should set this variable to "/nfs/jenkins".

HOST_MOUNT

This should be set to the mountpath used by the client to mount $HOST_EXPORT. The client will mount $HOST_SRV through the command:

$ mount -o vers=4 $HOST_SRV:$HOST_MOUNT

This is needed due to public filehandles in NFS v4. If the server is configured to export the directory "/nfs" but $HOST_EXPORT is set to "/nfs/jenkins", then HOST_MOUNT should be set to "/jenkins", resulting in the mount command:

$ mount -o vers=4 $HOST_SRV:/jenkins

OBJ_SRV (optional)

This should be set to the ip address (or domain name) of the machine that will store the compiled kernel objects. This machine should already be mounted on the $HOST_SRV before compiling. Leave this blank if you don't want to store compiled objects on a different machine.

OBJ_HOST_PATH (optional)

This should be set to the path on $HOST_MOUNT where $OBJ_SRV has already been mounted. The compiled object files will be placed in $OBJ_HOST_PATH/jenkins/. You can leave this blank if $OBJ_SRV has not been configured.

OBJ_MOUNT (optional)

This should be set to the mountpath used by the client to mount $OBJ_SRV. The client will mount $OBJ_SRV through the command:

$ mount -o vers=4 $OBJ_SRV:$OBJ_MOUNT

VERBOSE_SCRIPTS

Set this to "true" if you want more output to be prined to the Jenkins console when jobs are run. This can be used to help debug any problems during each job.

USE_DISTCC

Set this to "true" if you want to use distcc during the Compile job. Set it to "pump" if you also want to use the pump mode of distcc 3 (see distcc.org for more information). If you set this variable to "true" or "pump" then you must also configure the DISTCC_SERVERS_64 and DISTCC_SERVERS_32 variables.

DISTCC_SERVERS_64

If you are using distcc for compiling, set this to the list of servers that support compiling a 64-bit Linux kernel. It is a good idea to at least set this to "localhost".

DISTCC_SERVERS_32

If you are using distcc for compiling, set this to the list of servers that support compiling a 32-bit Linux kernel. It is a good idea to at least set this to "localhost".

DELETE_COMPILED_OUTPUT

Set this to "true" if you want to delete the compiled output during the cleanup phase. This will free up space on your disk, but you will have to recompile kernels during the next run

REMOVE_INSTALLED_KERNELS

Set this to "true" if you want to uninstall kernels from the client machine at the end of testing. This will free up disk space on the clients, but kernels will have to be reinstalled before you can run tests.

SHOW_FIELDS

Set this to a space separated list of fields you want printed out at the beginning of each test. Note that setting VERBOSE_SCRIPTS to "true" will automatically enable all fields. Possible values are:

uname

Show the client's output of `uname -a`

path

Show the client's $PATH

libc

Show the client's current version of glibc

gcc

Show the client's current version of gcc

make

Show the client's current version of make

showmount

Show the client's current version of showmount

mounts

mounts: Show the client's mounted filesystems

COLLECT_TRACES

This variable is used to configure if packet traces should be collected during while running tests. Setting this variable to "yes" will always collect traces, "no" will never collect traces. A value of "onfail" will start a packet trace, but it will only be saved if the test fails.

Packet traces are collected through tshark. The sample VMs come preconfigured with passwordless tshark enabled for the "jenkins" user.

MAKE_THREADS

This variable is used to set the number of threads each make process will use. Setting MAKE_THREADS to "2" is the same as running:

$ make -j2

BOOT_TIMEOUT

This variable is used to determine the number of seconds to wait for a machine to boot and activate SSH. If this timeout is exceeded the current job will fail, allowing the remaining queued jobs to run.

COMPILE_TEST_SRC_DIR

This variable is used to set the Linux source directory that should be compiled as part of the compile test. This should be set relative to HOST_EXPORT.

config/environment

SSH uses a different environment when running in non-interactive mode. This can cause problems if a command cannot be located. To fix this problem, users have the ability to configure their own environment. The config/environment file is read and parsed before each ssh command is run. You can enter any valid bash command here, similar to the way you would write your ~/.bashrc.

Changing the Jenkins port

By default, Jenkins will run on port 8080 of the host machine. This works well for machines only running a single instance of Jenkins, but this may not work for all setups. For example, you may want to run multiple instances of Jenkins or you may already have a program running on port 8080. To change the Jenkins port, edit the file $NFS_WORKSPACE/start_jenkins and change the "PORT=" variable to a different value. You will need to restart Jenkins for this change to take effect.

Adding a new architecture

Jenkins is configured to compile kernels for i386 and x86_64 by default. Other architectures can be added, but manual configuration of the "Compile" job is required. Select this job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.

Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then modify the "Compile", "Install" and "Run Tests" jobs to add the new file to the CONFIG variable in the "Test Matrix".

Adding a pre-installed kernel

Jenkins can be configured to boot and test kernels that are already installed on the test machine. If you wish to use this, first install the new kernel on your test machine. Reboot into your new kernel and check the output from the command

$ uname -r

to find the kernel release name (such as "3.0-ARCH" for an Archlinux machine running Linux 3.0). Create a new entry in your /boot/grub/menu.lst file for this kernel, using "3.0-ARCH" as the "title" parameter (replace "3.0-ARCH" with your `uname -r` output). Finish configuring grub the way you normally would for that kernel. Switch back to the Jenkins host and create a new config file for this kernel in the $NFS_WORKSPACE/config/kernel directory. This file should contain a single line: `export UNAME="3.0-ARCH"` (once again replacing "3.0-ARCH" with your `uname -r` output. Add this file to the "Run Tests" job, but do NOT add it to the "Compile" or "Install" jobs since this kernel has already been compiled and installed. See the file $NFS_WORKSPACE/config/kernel/arch.default for an example config file.

Note, you have just created the file $NFS_WORKSPACE/config/kernel/$CONFIG. Jenkins will check for the existence of the directory $NFS_WORKSPACE/obj/$ARCH/$CONFIG when determining how to behave. If this directory does not exist, Jenkins will assume this is a precompiled kernel and read the kernel release name from $NFS_WORKSPACE/config/kernel/$CONFIG. If the directory does exist, the include/config/kernel.release file will be used for the kernel release name.

Adding a new virtual machine client

A virtual machine client will be booted before use and halted when it is no longer being used. Create the VM in virt-manager and start the "Admin - Add Client" job to configure passwordless SSH to the VM and install various tests (such as connectathon). See the section "Machine setup" below for help creating new VMs. Input values for the following fields:

NAME

This should be set to the name of the virtual machine as configured in virt-manager.

MOUNTPOINT

This is the full path on the client that the NFS server should be mounted on.

SSH_USER

This is the username of a user that has SSH access to the client.

IS_VM

Make sure this is set to "true" to properly configure this client as a VM.

Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."

Adding a new real machine client

A real machine client should already be running before use and will never be halted by the Jenkins. Be careful when using this option if you do not have physical access to the machine in case something goes wrong. When the machine is running, start the "Admin - Add Client" job to configure passwordless SSH to the machine and install various tests (such as connectathon). See the section "Machine setup" below for help configuring a new machine. Input values for the following fields:

NAME

This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.

MOUNTPOINT

This is the full path on the client that the NFS server should be mounted on.

SSH_USER

This is the username of a user that has SSH access to the client.

IS_VM

Make sure this is set to "false" to properly configure this client as a real machine.

IP

This should be set to the IP address or domain name of the machine so that it can be accessed through SSH.

Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."

Adding a virtual machine server

Set up a virtual machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:

NAME

This should be set to the name of the virtual machine as configured in virt-manager.

V2_MOUNT

This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run.

V3_MOUNT

This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run.

V4_MOUNT

This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run.

V41_MOUNT

This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run.

TEST_DIR

This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem.

SSH_USER

This should be set to the username of a user that has SSH access to the server. This is a mandatory field needed to halt the virtual machine when it is no longer being used.

FAULT_INJECTION

This should be set to "none" if the server does not support fault injection, otherwise choose the appropriate server type from the drop-down list.

IS_VM

This should be set to "true" to properly configure this server as a virtual machine.

Once configured, edit the "Run Tests" job and add the new server to the SERVER variable of the "Test Matrix."

Adding a real machine server

Set up a real machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:

NAME

This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.

V2_MOUNT

This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run.

V3_MOUNT

This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run.

V4_MOUNT

This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run.

V41_MOUNT

This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run.

TEST_DIR

This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem.

SSH_USER

This should be set to the username of a user that has SSH access to the server. This field is optional and will only be used if the server is configured to run fault injection tests.

FAULT_INJECTION

This should be set to "none" if the server does not support fault injection, otherwise choose the appropriate server type from the drop-down list.

IS_VM

This should be set to "false" to properly configure this server as a real machine.

IP

This should be set to the IP address or domain name of the machine so it can be mounted by clients and accessed through SSH.

Once configured, edit the "Run Tests" job and add the new server to the SERVER ariable of the "Test Matrix."

Adding a new test

First determine if your test can be executed as a simple command over ssh. If not, you should write a new script to run your test. Next run the "Admin - Add Test" job to create a new test file. Input values for the following fields:

NAME

This is simply an identifier to help keep track of what the test will do and to make it easier to reference in the Run Tests job.

ENABLED

When a test is disabled, it will not run when triggered by the Run Tests job. Set this to "No" if you want to create the test, but do not want it to run right away.

FAULT_INJECTION

This should be set to "Yes" if the test makes use of fault injection features. This requires a specially configured server and a fault injection driver for the server type.

SSH_COMMAND

Set this to "Yes" if your test is a simple command that can be run over ssh. Set this to "No" to run a command or script on the machine hosting Jenkins.

COMMAND

Set this to the command that should be run as the main part of the test.

The following variables are available for use when writing your COMMAND:

$CLIENT

Virtual machine to run test on

$SERVER

Server being tested against

$CONFIG

Config file used to compile the kernel to be tested

$NFS

The NFS version to use

$MNT_POINT

Location on the client that the server has been mounted to

$TEST_DIR

Directory test output should be written to

$WS

Full path to the Jenkins workspace once the server has been mounted

$TEST

The name of the test to run

Additionally, the bash function `test_scripts_dir` can be used as a shortcut to $NFS_WORKSPACE/scripts if you do not want the $NFS_WORKSPACE variable to be expanded. Some useful helper functions have been defined in the files in $NFS_WORKSPACE/scripts/functions/. You can source these files and use the helper functions in your commands.

After submitting your test, the generated file will be displayed in the Jenkins console output. If there are any errors, see the section below for editing tests.

Before you can use the test, you must edit the "Run Tests" job and add the new test into the "TEST" field.

Editing a test

Tests are configured through files placed in $NFS_WORKSPACE/config/tests/. You can modify a test by editing the appropriate file in that directory. The following fields can be modified:

ENABLED

This variable determines if the test should be run or not. Change from "true" to "false" to disable a test.

FAULT_INJECTION

This variable is used during automatic filtering of fault injection tests. Change to "true" if your test makes use of the NFSD fault injection framework

do_test()

This function is called to run the test. The command in this function is set based on the COMMAND parameter passed to the "Admin - Add Test" job.

Test Groups

If you have a lot of tests to run, you may want to consider breaking them into multiple projects featuring a smaller number of tests. This will create more jobs to manage, but you will more easily be able to tell if different "types" of tests pass or fail.

To begin with, clone the default "Run Tests" job that has already been configured. From the main Jenkins page:

1. Follow the "New Job" link at the top left-hand corner of the page
2. Fill out the "Job name" field on the new job configuration page
3. Select "Copy existing Job" at the end of the list
4. Choose to copy from "Run Tests"
5. Edit this job so it does what you want (see the previous sections beginning with "Adding a new architecture")

At this point, the new job should be created, but it will not run automatically after the "Install" job has completed. If you want the new tests scheduled after the "Install" job:

1. Open up the configuration page for the "Install" job
2. Scroll to the bottom of the page
3. Find the "Trigger parameterized build on other projects" line and edit the "Projects to build" field and add your new test job to the comma separated list of projects.

Fault Injection Tests

Jenkins-nfs comes with pre-configured fault injection test scripts, but they must be added to the "Run Tests" job before you can use them. The following tests are available:

forget_clients

This will test client recovery after the server has forgotten all associated client state.

forget_delegations

This will test client recovery after the server has forgotten all associated delegations.

recall_delegations

This will test how a client handles the DELEGRETURN call

reboot_delegations

This will test how a client recovers after the server reboots while the client holds a delegation.

forget_locks

This will test client recovery after the server has forgotten all associated locks.

reboot_locks

This will test how a client recovers after the server reboots while the client holds a lock.

forget_openowners

This will test client recovery after the server has forgotten all associated open file state.

reboot_openowners

This will test how a client recovers after the server reboots while the client holds an open file.

Filtering Out Combinations

The "Install" and "Run Tests" jobs are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values will be run unless they are filtered out. The files in $NFS_WORKSPACE/config/filters/ can be used to blacklist any combination that doesn't make sense for your setup. These files are simple shell scripts that hould exit 0 if the combination can be tested.

The following invalid combinations will automatically be detected and filtered out by Jenkins:

1. Requested client doesn't exist
2. Requested server doesn't exist
3. The server has not been configured to run the requested version of NFS.
4. The server is not a fault injection server
5. The test does not exist
6. Fault injection test over NFS v2 or v3

See the files in $NFS_WORKSPACE/config/filters/ for more information when writing your own filters. Jenkins will re-run filtering whenever a job is saved, so you can force a refilter by opening the configuration dialog and choosing "save" at the bottom of the page. Logs will be created in $NFS_WORKSPACE/filters/ to explain why a job has been filtered out.

Machine setup

There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.

1. Must have NFS client running
2. Must have prerequisites for running the connectathon tests
3. Passwordless sudo for SSH_USER user
4. Passwordless ssh enabled in the sshd config
5. (optional) Have prerequisites for running the xfs tests
6. (optional) Have tshark installed and configured for the SSH_USER user

Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.

Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).

Passwordless ssh can be configured by running

$ $NFS_WORKSPACE/scripts/init_client

or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server. Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.

If you intend to collect packet traces, you will need to install the tshark program (it may be packaged with wireshark). Once installed, add your ssh user to the wireshark group so packets can be safely captured

Sample Machines

Preconfigured Archlinux virtual machines can be downloaded from here:

• 32-bit sample machine
• 64-bit sample machine

These machines come preconfigured with a valid user. Log in with:

   username: jenkins
   password: jenkins

If you find you need root access:

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly! See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.

Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.

The Job Pool

Jenkins-NFS works best when configured with a large job pool, since a larger pool will increase the chances of a test running on an already-booted kernel. Most of the testing during Jenkins-NFS development was performed with a job pool of size 8. To increase the job pool size:

1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
2. Follow the "Configure System" link
3. Find the "# of executors" field and change to your desired value

Updating Jenkins

The Jenkins development team realeases a new jenkins.war file every week or two. The updated war can be downloaded and installed automatically from your currenly running version. To update Jenkins:

1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
2. At the top of the page, there may be a message saying that a new version is available. If you do not see this message, then you are already running the latest version of Jenkins
3. If you do see the new version message, click the "Or Upgrade Automatically" button to begin the upgrade process
4. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the new version has been installed. If you don't do this, you must manually stop and start Jenkins to switch to the new version

Installing and Upgrading Plugins

Plugins can add new features that are not included with the base jenkins.war program. To locate and install a new plugin:

1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
2. Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
3. Click the "Available" tab
4. Browse the list of available plugins, selecting the ones you wish to install
5. Click the "Install" button at the bottom of the page
6. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been installed. If you don't do this, you must manually stop and start Jenkins to enable your new plugins. Note that recent Jenkins releases support installing plugins without requiring Jenkins to restart.

Developers will occasionally release a new version of their plugin. To update your installed plugins:

1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
2. Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
3. Select the plugins you wish to update
4. Click the "Install" button at the bottom of the page
5. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been updated. If you don't do this, you must manually stop and start Jenkins to use the updated plugins. Note that recent Jenkins releases support updating plugins without requiring Jenkins to restart.