Jenkins CI

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
Line 46: Line 46:
* COMPILE_TEST_SRC_DIR
* COMPILE_TEST_SRC_DIR
** This variable is used to set the Linux source directory that should be compiled as part of the compile test.  This should be set relative to HOST_EXPORT.
** This variable is used to set the Linux source directory that should be compiled as part of the compile test.  This should be set relative to HOST_EXPORT.
 +
 +
== config/environment ==
 +
SSH uses a different environment when running in non-interactive mode.  This can cause problems if a command cannot be located.  To fix this problem, users have the ability to configure their own environment.  The config/environment file is read and parsed before each ssh command is run.  You can enter any valid bash command here, similar to the way you would write your ~/.bashrc.
== Changing the Jenkins port ==
== Changing the Jenkins port ==

Revision as of 14:36, 23 September 2011

Contents

Jenkins Setup

Begin by cloning the jenkins-nfs git repository:

$ git clone git:///git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run ./setup to download the Jenkins executable, required plugins, and to configure the Jenkins environment. In addition, a launcher script named start_jenkins will be placed in $NFS_WORKSPACE. Before starting Jenkins, edit the config/jenkins_nfs.sh file to match your desired setup (see the config/jenkins_nfs.sh section below for more information about this file). Once everything has been configured, start Jenkins by running $NFS_WORKSPACE/start_jenkins.

You must also configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this. Once everything is configured, you can begin a new run by starting the job "Compile and Install".

config/jenkins_nfs.sh

This script is used to set configuration variables that can affect how Jenkins behaves. This file is sourced by scripts/functions.sh, so any changes you make will not require you to restart Jenkins. Keep in mind that your changes may not be reflected in any currently running jobs. The following variables should be set:

  • HOST_SRV
    • This should be set to the ip address (or domain name) of the machine hosting Jenkins. This machine should also be hosting the Linux source code you wish to compile and test.
  • HOST_EXPORT
    • This should be set to the path on $HOST_SRV that is being exported by $HOST_SRV. For example, if you are exporting the directory /nfs/jenkins then you should set this variable to "/nfs/jenkins".
  • HOST_MOUNT
    • This should be set to the mountpath used by the client to mount $HOST_EXPORT. The client will mount $HOST_SRV through the command `mount -o vers=4 $HOST_SRV:$HOST_MOUNT`. This is needed due to public filehandles in NFS v4. If the server is configured to export the directory "/nfs" but $HOST_EXPORT is set to "/nfs/jenkins", then HOST_MOUNT should be set to "/jenkins", resulting in the mount command `mount -o vers=4 $HOST_SRV:/jenkins`.
  • OBJ_SRV
    • (optional) This should be set to the ip address (or domain name) of the machine that will store the compiled kernel objects. This machine should already be mounted on the $HOST_SRV before compiling. Leave this blank if you don't want to store compiled objects on a different machine.
  • OBJ_HOST_PATH
    • (optional) This should be set to the path on $HOST_MOUNT where $OBJ_SRV has already been mounted. The compiled object files will be placed in $OBJ_HOST_PATH/jenkins/. You can leave this blank if $OBJ_SRV has not been configured.
  • OBJ_MOUNT
    • (optional) This should be set to the mountpath used by the client to mount $OBJ_SRV. The client will mount $OBJ_SRV through the command `mount -o vers=4 $OBJ_SRV:$OBJ_MOUNT`.
  • VERBOSE_SCRIPTS
    • Set this to "true" if you want more output to be prined to the Jenkins console when jobs are run. This can be used to help debug any problems during each job.
  • COLLECT_TRACES
    • This variable is used to configure if packet traces should be collected during while running tests. Setting this variable to "yes" will always collect traces, "no" will never collect traces. A value of "onfail" will start a packet trace, but it will only be saved if the test fails.
    • Packet traces are collected through tshark. The sample VMs come preconfigured with passwordless tshark enabled for the "jenkins" user.
  • MAKE_THREADS
    • This variable is used to set the number of threads each make process will use. Setting MAKE_THREADS to "2" is the same as running `make -j2`.
  • BOOT_TIMEOUT
    • This variable is used to determine the number of seconds to wait for a machine to boot and activate SSH. If this timeout is exceeded the current job will fail, allowing the remaining queued jobs to run.
  • COMPILE_TEST_SRC_DIR
    • This variable is used to set the Linux source directory that should be compiled as part of the compile test. This should be set relative to HOST_EXPORT.

config/environment

SSH uses a different environment when running in non-interactive mode. This can cause problems if a command cannot be located. To fix this problem, users have the ability to configure their own environment. The config/environment file is read and parsed before each ssh command is run. You can enter any valid bash command here, similar to the way you would write your ~/.bashrc.

Changing the Jenkins port

By default, Jenkins will run on port 8080 of the host machine. This works well for machines only running a single instance of Jenkins, but this may not work for all setups. For example, you may want to run multiple instances of Jenkins or you may already have a program running on port 8080. To change the Jenkins port, edit the file $NFS_WORKSPACE/start_jenkins and change the "PORT=" variable to a different value. You will need to restart Jenkins for this change to take effect.

Adding a new architecture

Jenkins is configured for i386 and x86_64 by default. Other architectures can be added, but manual configuration is required. You will have to modify the "Compile", "Install" and "Run Tests" jobs. Select each job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.

Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then modify the "Compile", "Install" and "Run Tests" jobs to add the new file to the CONFIG variable in the "Test Matrix".

Adding a pre-installed kernel

Jenkins can be configured to boot and test kernels that are already installed on the test machine. If you wish to use this, first install the new kernel on your test machine. Reboot into your new kernel and check the output from the command `uname -r` to find the kernel release name (such as "3.0-ARCH" for an Archlinux machine running Linux 3.0). Create a new entry in your /boot/grub/menu.lst file for this kernel, using "3.0-ARCH" as the "title" parameter (replace "3.0-ARCH" with your `uname -r` output). Finish configuring grub the way you normally would for that kernel. Switch back to the Jenkins host and create a new config file for this kernel in the $NFS_WORKSPACE/config/kernel directory. This file should contain a single line: `export UNAME="3.0-ARCH"` (once again replacing "3.0-ARCH" with your `uname -r` output. Add this file to the "Run Tests" job, but do NOT add it to the "Compile" or "Install" jobs since this kernel has already been compiled and installed. See the file $NFS_WORKSPACE/config/kernel/arch.default for an example config file.

Note, you have just created the file $NFS_WORKSPACE/config/kernel/$CONFIG. Jenkins will check for the existence of the directory $NFS_WORKSPACE/obj/$ARCH/$CONFIG when determining how to behave. If this directory does not exist, Jenkins will assume this is a precompiled kernel and read the kernel release name from $NFS_WORKSPACE/config/kernel/$CONFIG. If the directory does exist, the include/config/kernel.release file will be used for the kernel release name.

Adding a new virtual machine client

A virtual machine client will be booted before use and halted when it is no longer being used. Create the VM in virt-manager and start the "Admin - Add Client" job to configure passwordless SSH to the VM and install various tests (such as connectathon). See the section "Machine setup" below for help creating new VMs. Input values for the following fields:

  • NAME
    • This should be set to the name of the virtual machine as configured in virt-manager.
  • ARCH
    • This should be set to the architecture of the installed kernel (usually i386 or x86_64).
  • MOUNTPOINT
    • This is the full path on the client that the NFS server should be mounted on.
  • SSH_USER
    • This is the username of a user that has SSH access to the client.
  • IS_VM
    • Make sure this is set to "Yes" to properly configure this client as a VM.

Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."

Adding a new real machine client

A real machine client should already be running before use and will never be halted by the Jenkins. Be careful when using this option if you do not have physical access to the machine in case something goes wrong. When the machine is running, start the "Admin - Add Client" job to configure passwordless SSH to the machine and install various tests (such as connectathon). See the section "Machine setup" below for help configuring a new machine. Input values for the following fields:

  • NAME
    • This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.
  • ARCH
    • This should be set to the architecture of the installed kernel (usually i386 or x86_64). This will be used to to prevent installing a kernel with a different architecture.
  • MOUNTPOINT
    • This is the full path on the client that the NFS server should be mounted on.
  • SSH_USER
    • This is the username of a user that has SSH access to the client.
  • IS_VM
    • Make sure this is set to "No" to properly configure this client as a real machine.
  • IP
    • This should be set to the IP address of the machine so that it can be accessed through SSH.

Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."

Adding a virtual machine server

Set up a virtual machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:

  • NAME
    • This should be set to the name of the virtual machine as configured in virt-manager.
  • V2_MOUNT
    • This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run.
  • V3_MOUNT
    • This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run.
  • V4_MOUNT
    • This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run.
  • V41_MOUNT
    • This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run.
  • TEST_DIR
    • This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem.
  • SSH_USER
    • This should be set to the username of a user that has SSH access to the server. This is a mandatory field needed to halt the virtual machine when it is no longer being used.
  • FAULT_INJECTION
    • This should be set to "Yes" if the server has been configured as an NFSD fault injection server.
  • IS_VM
    • This should be set to "Yes" to properly configure this server as a virtual machine.

Once configured, edit the "Run Tests" job and add the new server to the SERVER variable of the "Test Matrix."

Adding a real machine server

Set up a real machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:

  • NAME
    • This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.
  • V2_MOUNT
    • This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run.
  • V3_MOUNT
    • This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run.
  • V4_MOUNT
    • This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run.
  • V41_MOUNT
    • This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run.
  • TEST_DIR
    • This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem.
  • SSH_USER
    • This should be set to the username of a user that has SSH access to the server. This field is optional and will only be used if the server is configured to run fault injection tests.
  • FAULT_INJECTION
    • This should be set to "Yes" if the server has been configured as an NFSD fault injection server.
  • IS_VM
    • This should be set to "No" to properly configure this server as a real machine.
  • IP
    • This should be set to the IP address of the machine so it can be mounted by clients and accessed through SSH.

Once configured, edit the "Run Tests" job and add the new server to the SERVER ariable of the "Test Matrix."

Adding a new NFS version

Edit the "Run Tests" job and add the new version to the list of possible values for the NFS variable. NFS v2, v3, and v4 are configured by default. Adding v4.1 will require modification of the mount command to add in the "minorversion=1" mount option.

Adding a new test

First determine if your test can be executed as a simple command over ssh. If not, you should write a new script to run your test. Next run the "Admin - Add Test" job to create a new test file. Input values for the following fields:

  • NAME
    • This is simply an identifier to help keep track of what the test will do and to make it easier to reference in the Run Tests job.
  • ENABLED
    • When a test is disabled, it will not run when triggered by the Run Tests job. Set this to "No" if you want to create the test, but do not want it to run right away.
  • FAULT_INJECTION
    • This should be set to "Yes" if the test makes use of NFSD fault injection. This does require a special Linux server that has been configured with the NFSD fault injection patches.
  • SSH_COMMAND
    • Set this to "Yes" if your test is a simple command that can be run over ssh. Set this to "No" to run a command or script on the machine hosting Jenkins.
  • COMMAND
    • Set this to the command that should be run as the main part of the test.

The following variables are available for use when writing your COMMAND:

  • $CLIENT
    • Virtual machine to run test on
  • $SERVER
    • Server being tested against
  • $ARCH
    • Architecture of virtual machine
  • $CONFIG
    • Config file used to compile the kernel to be tested
  • $NFS
    • The NFS version to use
  • $MNT_POINT
    • Location on the client that the server has been mounted to
  • $TEST_DIR
    • Directory test output should be written to
  • $WS
    • Full path to the Jenkins workspace once the server has been mounted
  • $TEST
    • The name of the test to run

Additionally, the bash function `test_scripts_dir` can be used as a shortcut to $NFS_WORKSPACE/scripts if you do not want the $NFS_WORKSPACE variable to be expanded.

Editing a test

Tests are configured through files placed in $NFS_WORKSPACE/config/tests. You can modify a test by editing the appropriate file in that directory. The following fields can be modified:

  • ENABLED
    • This variable determines if the test should be run or not. Change from "true" to "false" to disable a test.
  • do_test()
    • This function is called to run the test. The command in this function is set based on the COMMAND parameter passed to the "Admin - Add Test" job.

Test Groups

If you have a lot of tests to run, you may want to consider breaking them into multiple projects featuring a smaller number of tests. This will create more jobs to manage, but you will more easily be able to tell if different "types" of tests pass or fail.

To begin with, clone the default "Run Tests" job that has already been configured. From the main Jenkins page:

  1. Follow the "New Job" link at the top left-hand corner of the page
  2. Fill out the "Job name" field on the new job configuration page
  3. Select "Copy existing Job" at the end of the list
  4. Choose to copy from "Run Tests"
  5. Edit this job so it does what you want (see the previous sections beginning with "Adding a new architecture")

At this point, the new job should be created, but it will not run automatically after the "Install" job has completed. If you want the new tests scheduled after the "Install" job:

  1. Open up the configuration page for the "Install" job
  2. Scroll to the bottom of the page
  3. Find the "Trigger parameterized build on other projects" line and edit the "Projects to build" field and add your new test job to the comma separated list of projects.

Blacklisting a combination

"Compile," "Install" and "Run Tests" are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values will be run. Some combination of variables may not make sense, such as running 32-bit tests on a 64-bit client. The files in $NFS_WORKSPACE/config/filters/ can be used to blacklist any combination that doesn't make sense for your setup. These files are simple shell scripts that hould exit 0 if the combination can be tested.

The following invalid combinations will automatically be detected and filtered out by Jenkins:

  1. The architecture variable does not match the architecture of the machine.
  2. The server has not been configured to run the requested version of NFS.

See the files in $NFS_WORKSPACE/config/filters/ for more information when writing your own filters.

Machine setup

There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.

  1. Must have NFS client running
  2. Must have prerequisites for running the connectathon tests
  3. Passwordless sudo for SSH_USER user
  4. Passwordless ssh enabled in the sshd config
  5. (optional) Have prerequisites for running the xfs tests
  6. (optional) Have tshark installed and configured for the SSH_USER user

Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.

Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).

Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_client or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server. Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.

If you intend to collect packet traces, you will need to install the tshark program (it may be packaged with wireshark). Once installed, add your ssh user to the wireshark group: sudo gpasswd -a <username> wireshark.

Sample Machines

Preconfigured Archlinux virtual machines can be downloaded from here:

These machines come preconfigured with a valid user. Log in with:

   username: jenkins
   password: jenkins

If you find you need root access:

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly!

Archlinux and Fedora based machines are known to work properly

The Job Pool

Jenkins-NFS works best when configured with a large job pool, since a larger pool will increase the chances of a test running on an already-booted kernel. Most of the testing during Jenkins-NFS development was performed with a job pool of size 8. To increase the job pool size:

  1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
  2. Follow the "Configure System" link
  3. Find the "# of executors" field and change to your desired value

Updating Jenkins

The Jenkins development team realeases a new jenkins.war file every week or two. The updated war can be downloaded and installed automatically from your currenly running version. To update Jenkins:

  1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
  2. At the top of the page, there may be a message saying that a new version is available. If you do not see this message, then you are already running the latest version of Jenkins
  3. If you do see the new version message, click the "Or Upgrade Automatically" button to begin the upgrade process
  4. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the new version has been installed. If you don't do this, you must manually stop and start Jenkins to switch to the new version

Installing and Upgrading Plugins

Plugins can add new features that are not included with the base jenkins.war program. To locate and install a new plugin:

  1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
  2. Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
  3. Click the "Available" tab
  4. Browse the list of available plugins, selecting the ones you wish to install
  5. Click the "Install" button at the bottom of the page
  6. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been installed. If you don't do this, you must manually stop and start Jenkins to enable your new plugins

Developers will occasionally release a new version of their plugin. To update your installed plugins:

  1. From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
  2. Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
  3. Select the plugins you wish to update
  4. Click the "Install" button at the bottom of the page
  5. (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been updated. If you don't do this, you must manually stop and start Jenkins to use the updated plugins
Personal tools