Jenkins CI

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
 +
== Jenkins Setup ==
== Jenkins Setup ==
Before doing anything, make sure the required environment variables are set
Before doing anything, make sure the required environment variables are set
to correct values (see the section "Required Environment Variables" below).
to correct values (see the section "Required Environment Variables" below).
-
Begin by cloning the jenkins-nfs git repository (git clone <LOCATION>)Once
+
Begin by cloning the jenkins-nfs git repository:
-
you have the latest version of the scripts, cd into jenkins-nfs and run ./setup
+
<pre>
-
to download Jenkins, required plugins, and configure the jenkins environment.
+
git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git
-
In addition, a launcher script called start_jenkins will be placed in  
+
</pre>
-
$NFS_WORKSPACE directory.  Jenkins is then started by running ./start_jenkins.
+
 
-
If you ever move the jenkins.war file, be sure to update start_jenkins to
+
The cloned directory should be placed in a location exported by the NFS server
-
reflect the new location.
+
running on the host machineThe full path up to and including this directory
 +
is referred to as $NFS_WORKSPACE throughout this document and various scripts.
 +
 
 +
Once you have the latest version of the scripts, cd into
 +
$NFS_WORKSPACE/jenkins-nfs and run ./setup to download the Jenkins executable,
 +
required plugins, and to configure the jenkins environment. In addition, a
 +
launcher script called start_jenkins will be placed in $NFS_WORKSPACE.  Jenkins
 +
is then started by running $NFS_WORKSPACE/start_jenkins.
-
Before you can use Jenkins to test kernels, you must configure what tests to run
+
Before you can use Jenkins to test kernels, you must first configure a "default"
-
and virtual machines to run them on.  See the sections below for instructions
+
server.  This is the server hosting the Jenkins installation, and will be used
-
on how to do this.
+
to install kernels on the virtual machine clientsTo add the default server,
 +
run the "Admin - Add Server" job and enter "default" for the "NAME" parameter.
-
Once everything is configured, you can begin a new run by starting the job
+
You must also configure what tests to run and virtual machines to run them on.
-
"Compile and Install".
+
See the sections below for instructions on how to do this.  Once everything is
 +
configured, you can begin a new run by starting the job "Compile and Install".
Line 22: Line 32:
These need to be set before starting Jenkins
These need to be set before starting Jenkins
-
* NFS_WORKSPACE
 
-
** The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory.  This directory should be accessable through NFS.
 
* NFS_SSH_USER
* NFS_SSH_USER
** The username of the virtual machine user that will install the compiled kernel and run the tests.  SSH should be configured to avoid prompting for a password.
** The username of the virtual machine user that will install the compiled kernel and run the tests.  SSH should be configured to avoid prompting for a password.
-
* NFS_SERVER
 
-
** This is the ip address of the server that makes NFS_WORKSPACE available on the network.  The server is mounted to install compiled kernels and run the various tests.
 
-
* NFS_MOUNT
 
-
** The path on the NFS_SERVER to mount.
 
* NFS_MOUNT_POINT
* NFS_MOUNT_POINT
** The path on the virtual machine that NFS_SERVER is mounted on.
** The path on the virtual machine that NFS_SERVER is mounted on.
-
 
-
The final mount command run will look similar to this:
 
-
  mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT
 
Line 47: Line 48:
== Adding a new .config ==
== Adding a new .config ==
-
Copy the config file to $NFS_WORKSPACE/config using a unique name.  Then modify
+
Copy the config file to $NFS_WORKSPACE/config/ using a unique name.  Then modify
"Compile and Install" and "Run Tests" to add the new file to the CONFIG variable
"Compile and Install" and "Run Tests" to add the new file to the CONFIG variable
in the "Test Matrix".
in the "Test Matrix".
-
== Adding a new virtual machine ==
+
== Adding a new virtual machine client ==
Create the new VM in virt-manager.  Once it has been created, start the
Create the new VM in virt-manager.  Once it has been created, start the
-
"Initialize VM" job to configure passwordless SSH to the VM.  See the section
+
"Initialize VM" job to configure passwordless SSH to the VM and install various
-
"Virtual machine setup" below for help creating new VMs.
+
tests (such as connectathon).  See the section "Virtual machine setup" below for
 +
help creating new VMs.
 +
 
 +
 
 +
== Adding a new server ==
 +
Set up the server (either as a VM or a real machine), and then run the "Add
 +
Server" job to create a server config file.  When the job has finished, edit
 +
"Run Tests" and add the server name to the SERVER variable of the "Test Matrix"
Line 84: Line 92:
multi-configuration projects.  This means that every combination of possible
multi-configuration projects.  This means that every combination of possible
variable values is tested.  Some combination of variables may not make sense,
variable values is tested.  Some combination of variables may not make sense,
-
such as running 32bit tests on a 64bit virtual machine.  You can modify the
+
such as running 32bit tests on a 64bit virtual machine.  The files in
-
"Combination Filter" field in the "Configuration Matrix" section to either
+
$NFS_WORKSPACE/filters/ can be used to blacklist (or whitelist) various
-
white-list or black-list a specific combination using a Groovy expressionIf
+
combinationsThese files are simple shell scripts that should exit 0 if the
-
the expression evaluates to "True", the combination will be tested.
+
combination can be tested.  See the files in $NFS_WORKSPACE/filters/ for more
 +
information.
-
For example, say you have the architectures "i386" and "x86_64".  You also have
 
-
the virtual machines "jenkins32" (a 32bit VM) and "jenkins64" (a 64bit VM).
 
-
Only the combinations (x86_64, jenkins64) and (i386, jenkins32) should be run,
 
-
since (i386, jenkins64) and (x86_64, jenkins32) doesn't make sense.  The
 
-
following line accomplishes just that:
 
-
<pre>
 
-
        (ARCH=="x86_64" && MACHINE=="jenkins64") ||
 
-
            (ARCH=="i386" && MACHINE=="jenkins32")
 
-
</pre>
 
 +
== Testing NFS v4.1 ==
 +
NFS v4.1 is still considered to be experimental and requires manual intervention
 +
to enable on both the client and server.  As a result, running tests over NFS
 +
v4.1 has been disabled by default.
-
== Connectathon and Architectures ==
+
Once you have your client and server configured correctly, edit the file
-
Connectathon tests are compiled for i386 and x86_64 architectures.  If you plan
+
$NFS_WORKSPACE/filters/runtests and remove or comment out the block of code
-
any other architecture, you will need to manually compile connectathon yourself.
+
under "#### Disable NFS v4.1 tests ####"
-
Once compiled, place the resulting directory in $NFS_WORKSPACE/cthon/ARCH
+
-
(where ARCH is the name of the new architecture).  Be sure to add the new
+
-
architecture to both "Compile and Install" and "Run Tests".
+
Line 164: Line 165:
shell that Jenkins was initially started from.
shell that Jenkins was initially started from.
-
If you intend to use NFS v3, you will have to an entry to /etc/hosts on both
+
You will also need to set the environment variable MNTOPTIONS on each virtual
-
the client and the server so that each machine is aware of the other.
+
machine so that the connectathon tests will work properly.  To do this, SSH
-
 
+
-
If you intend to use NFS v2, you will need to set the environment variable
+
-
MNTOPTIONS so that the connectathon tests will work properly.  To do this, SSH
+
needs to allow user environments to be set up. This can be done by editing
needs to allow user environments to be set up. This can be done by editing
/etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting
/etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting
Line 178: Line 176:
Archlinux and Fedora virtual machines are known to work properly.
Archlinux and Fedora virtual machines are known to work properly.
-
 
-
 
-
== Ideas for future work ==
 
-
* Give VM time to shut down by itself before forcing it off
 
-
* Test against multiple servers
 
-
** Add server variables to multi-config project
 
-
* Compile connectathon / xfstests on each VM?
 
-
* Catch exit codes from remote commands for error checking
 
-
** Don't run with -e so that I can still do cleanup
 
-
** Put cleanup code in bash function
 
-
** Make new function for running command, checking exit value, and exiting script if necessarily
 

Revision as of 20:31, 3 June 2011

Contents

Jenkins Setup

Before doing anything, make sure the required environment variables are set to correct values (see the section "Required Environment Variables" below).

Begin by cloning the jenkins-nfs git repository:

	git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE/jenkins-nfs and run ./setup to download the Jenkins executable, required plugins, and to configure the jenkins environment. In addition, a launcher script called start_jenkins will be placed in $NFS_WORKSPACE. Jenkins is then started by running $NFS_WORKSPACE/start_jenkins.

Before you can use Jenkins to test kernels, you must first configure a "default" server. This is the server hosting the Jenkins installation, and will be used to install kernels on the virtual machine clients. To add the default server, run the "Admin - Add Server" job and enter "default" for the "NAME" parameter.

You must also configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this. Once everything is configured, you can begin a new run by starting the job "Compile and Install".


Required Environment Variables

These need to be set before starting Jenkins

  • NFS_SSH_USER
    • The username of the virtual machine user that will install the compiled kernel and run the tests. SSH should be configured to avoid prompting for a password.
  • NFS_MOUNT_POINT
    • The path on the virtual machine that NFS_SERVER is mounted on.


Adding a new architecture

Jenkins is configured for i386 and x86_64 by default. Other architectures can be added, but manual configuration is required. You will have to modify both "Compile and Install" and "Run Tests". Select each job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.


Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/ using a unique name. Then modify "Compile and Install" and "Run Tests" to add the new file to the CONFIG variable in the "Test Matrix".


Adding a new virtual machine client

Create the new VM in virt-manager. Once it has been created, start the "Initialize VM" job to configure passwordless SSH to the VM and install various tests (such as connectathon). See the section "Virtual machine setup" below for help creating new VMs.


Adding a new server

Set up the server (either as a VM or a real machine), and then run the "Add Server" job to create a server config file. When the job has finished, edit "Run Tests" and add the server name to the SERVER variable of the "Test Matrix"


Adding a new NFS version

Edit the "Run Tests" job and add the new version to the list of possible values for the NFS variable. NFS v2, v3, and v4 are configured by default. Adding v4.1 will require modification of the mount command to add in the "minorversion=1" mount option.


Adding a new test

First design your test and edit $NFS_WORKSPACE/tests.map to add a mapping from name -> command. The general format of this file is:

        name:where:command

     name: The name of the test
    where: Where the command will be run (either "remote" or "local")
  command: The command to run the test

Once the new test has been added to the tests.map file, add the new test to the "Run Tests" job.


Blacklisting a combination

Both "Compile and Install" and "Run Tests" are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values is tested. Some combination of variables may not make sense, such as running 32bit tests on a 64bit virtual machine. The files in $NFS_WORKSPACE/filters/ can be used to blacklist (or whitelist) various combinations. These files are simple shell scripts that should exit 0 if the combination can be tested. See the files in $NFS_WORKSPACE/filters/ for more information.


Testing NFS v4.1

NFS v4.1 is still considered to be experimental and requires manual intervention to enable on both the client and server. As a result, running tests over NFS v4.1 has been disabled by default.

Once you have your client and server configured correctly, edit the file $NFS_WORKSPACE/filters/runtests and remove or comment out the block of code under "#### Disable NFS v4.1 tests ####"


$NFS_WORKSPACE/scripts/functions

This file provides many convenient functions for controlling virtual machines. You can boot, halt, and reboot machines. You can also do more complicated actions like sending files, mounting the nfs server and running remote commands. Before using any of these functions, you have to tell your script what machine everything should be run on. This is done through the set_machine function. If you want to ls the $NFS_SSH_USER's home directory on the machine jenkins-tests, you could do the following:

	#!/bin/bash
	. functions
	set_machine jenkins-tests
	boot
	run_cmd "ls"
	halt


Virtual machine setup

There are a few requirements for a virtual machine to work correctly for the NFS Jenkins setup.

  1. Must have NFS client running
  2. Must be able to run connectathon tests
  3. Must be able to run xfs tests
  4. Passwordless sudo for NFS_SSH_USER user
  5. Passwordless ssh configured

The sample virtual machines come preconfigured with a valid user. Log in with:

   username: jenkins
   password: jenkins

If you find you need root access:

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly! You can download sample machines here:

Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm or by manually starting the job "Admin - Initialize VM". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server.

Archlinux and Fedora virtual machines are known to work properly.

Personal tools