Jenkins CI

From Linux NFS

You do not have permission to edit this page, for the following reason:

You can view and copy the source of this page:

== Jenkins Setup == Before doing anything, make sure the required environment variables are set to correct values (see the section "Required Environment Variables" below). Begin by cloning the jenkins-nfs git repository (git clone <LOCATION>). Once you have the latest version of the scripts, cd into jenkins-nfs and run ./setup to download Jenkins, required plugins, and configure the jenkins environment. In addition, a launcher script called start_jenkins will be placed in $NFS_WORKSPACE directory. Jenkins is then started by running ./start_jenkins. If you ever move the jenkins.war file, be sure to update start_jenkins to reflect the new location. Before you can use Jenkins to test kernels, you must configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this. Once everything is configured, you can begin a new run by starting the job "Compile and Install". == Required Environment Variables == These need to be set before starting Jenkins * NFS_WORKSPACE ** The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory. This directory should be accessable through NFS. * NFS_SSH_USER ** The username of the virtual machine user that will install the compiled kernel and run the tests. SSH should be configured to avoid prompting for a password. * NFS_SERVER ** This is the ip address of the server that makes NFS_WORKSPACE available on the network. The server is mounted to install compiled kernels and run the various tests. * NFS_MOUNT ** The path on the NFS_SERVER to mount. * NFS_MOUNT_POINT ** The path on the virtual machine that NFS_SERVER is mounted on. The final mount command run will look similar to this: mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT == Adding a new architecture == Jenkins is configured for i386 and x86_64 by default. Other architectures can be added, but manual configuration is required. You will have to modify both "Compile and Install" and "Run Tests". Select each job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable. == Adding a new .config == Copy the config file to $NFS_WORKSPACE/config using a unique name. Then modify "Compile and Install" and "Run Tests" to add the new file to the CONFIG variable in the "Test Matrix". == Adding a new virtual machine == Create the new VM in virt-manager. Once it has been created, start the "Initialize VM" job to configure passwordless SSH to the VM. See the section "Virtual machine setup" below for help creating new VMs. == Adding a new NFS version == Edit the "Run Tests" job and add the new version to the list of possible values for the NFS variable. NFS v2, v3, and v4 are configured by default. Adding v4.1 will require modification of the mount command to add in the "minorversion=1" mount option. == Adding a new test == First design your test and edit $NFS_WORKSPACE/tests.map to add a mapping from name -> command. The general format of this file is: <pre> name:where:command name: The name of the test where: Where the command will be run (either "remote" or "local") command: The command to run the test </pre> Once the new test has been added to the tests.map file, add the new test to the "Run Tests" job. == Blacklisting a combination == Both "Compile and Install" and "Run Tests" are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values is tested. Some combination of variables may not make sense, such as running 32bit tests on a 64bit virtual machine. You can modify the "Combination Filter" field in the "Configuration Matrix" section to either white-list or black-list a specific combination using a Groovy expression. If the expression evaluates to "True", the combination will be tested. For example, say you have the architectures "i386" and "x86_64". You also have the virtual machines "jenkins32" (a 32bit VM) and "jenkins64" (a 64bit VM). Only the combinations (x86_64, jenkins64) and (i386, jenkins32) should be run, since (i386, jenkins64) and (x86_64, jenkins32) doesn't make sense. The following line accomplishes just that: <pre> (ARCH=="x86_64" && MACHINE=="jenkins64") || (ARCH=="i386" && MACHINE=="jenkins32") </pre> == Connectathon and Architectures == Connectathon tests are compiled for i386 and x86_64 architectures. If you plan any other architecture, you will need to manually compile connectathon yourself. Once compiled, place the resulting directory in $NFS_WORKSPACE/cthon/ARCH (where ARCH is the name of the new architecture). Be sure to add the new architecture to both "Compile and Install" and "Run Tests". == $NFS_WORKSPACE/scripts/functions == This file provides many convenient functions for controlling virtual machines. You can boot, halt, and reboot machines. You can also do more complicated actions like sending files, mounting the nfs server and running remote commands. Before using any of these functions, you have to tell your script what machine everything should be run on. This is done through the set_machine function. If you want to ls the $NFS_SSH_USER's home directory on the machine jenkins-tests, you could do the following: <pre> #!/bin/bash . functions set_machine jenkins-tests boot run_cmd "ls" halt </pre> == Virtual machine setup == There are a few requirements for a virtual machine to work correctly for the NFS Jenkins setup. # Must have NFS client running # Must be able to run connectathon tests # Must be able to run xfs tests # Passwordless sudo for NFS_SSH_USER user # Passwordless ssh configured The sample virtual machines come preconfigured with a valid user. Log in with: <pre> username: jenkins password: jenkins </pre> If you find you need root access: <pre> username: root password: jenkins </pre> Be sure to change the root password before hosting virtual machines publicly! You can download sample machines here: * [http://www.linux-nfs.org/~bjschuma/jenkins_master.32.img 32-bit sample machine] * [http://www.linux-nfs.org/~bjschuma/jenkins_master.64.img 64-bit sample machine] Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm or by manually starting the job "Admin - Initialize VM". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file. This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from. If you intend to use NFS v3, you will have to an entry to /etc/hosts on both the client and the server so that each machine is aware of the other. If you intend to use NFS v2, you will need to set the environment variable MNTOPTIONS so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins. The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server. Archlinux and Fedora virtual machines are known to work properly. == Ideas for future work == * Give VM time to shut down by itself before forcing it off * Test against multiple servers * Add server variables to multi-config project * Compile connectathon / xfstests on each VM? * Catch exit codes from remote commands for error checking * Don't run with -e so that I can still do cleanup * Put cleanup code in bash function * Make new function for running command, checking exit value, and exiting script if necessarily == Required Environment Variables == These need to be set before starting Jenkins * NFS_WORKSPACE ** The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory. This directory should be accessable through NFS. * NFS_SSH_USER ** The username of the virtual machine user that will install the compiled kernel and run the tests. SSH should be configured to avoid prompting for a password. * NFS_SERVER ** This is the ip address of the server that makes NFS_WORKSPACE available on the network. The server is mounted to install compiled kernels and run the various tests. * NFS_MOUNT ** The path on the NFS_SERVER to mount. * NFS_MOUNT_POINT ** The path on the virtual machine that NFS_SERVER is mounted on. * NFS_PYTHON_PATH ** The path to a python interpreter to use when executing python scripts. (Scripts have been written for Python 2.7) The final mount command run will look similar to this: mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT == Connectathon and Architectures == When adding a virtual machine, one required argument is the architecture of the processor. This is required for properly selecting the correct compiled connectathon tests. The tests are compiled with -m32 and -m64 as compiler flags and placed in the $NFS_WORKSPACE/cthon/$ARCHITECTURE directory, where $ARCHITECTURE is either "i386" or "x86_64". You can add a virtual machine with a different architecture, but you will be responsible for compiling the connectathon tests and placing them in an appropriate directory. == $NFS_WORKSPACE/scripts/functions == This file provides many convenient functions for controlling virtual machines. You can boot, halt, and reboot machines. You can also do more complicated actions like sending files, mounting the nfs server and running remote commands. Before using any of these functions, you have to tell your script what machine everything should be run on. This is done through the set_machine function. If you want to ls the $NFS_SSH_USER's home directory on the machine jenkins-tests, you could do the following: <pre> #!/bin/bash . functions set_machine jenkins-tests boot run_cmd "ls" halt </pre> == Virtual machine requirements and initialization == There are a few requirements for a virtual machine to work correctly for the NFS Jenkins setup. # Must have NFS client running # Must be able to run connectathon tests # Must be able to run xfs tests # Passwordless sudo for NFS_SSH_USER user # Passwordless ssh configured The sample virtual machines come preconfigured with a valid user. Log in with: username: jenkins password: jenkins If you find you need root access: username: root password: jenkins Be sure to change the root password before hosting virtual machines publicly! You can download sample machines here: * [http://www.linux-nfs.org/~bjschuma/jenkins_master.32.img 32-bit sample machine] * [http://www.linux-nfs.org/~bjschuma/jenkins_master.64.img 64-bit sample machine] Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm or by manually starting the job "Admin - Initialize VM". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file. This process will require you to enter your password twice, once for the file transfer and once for appending to the authorized_keys file. Due to a limitation in the Jenkins software, the password must be entered in the same shell that Jenkins was initially started from. Virtual machines should be named based on the job it will be used to test (See "Adding a git repo" below). If you intend to use NFS v3, you will have to an entry to /etc/hosts on both the client and the server so that each machine is aware of the other. If you intend to use NFS v2, you will need to set the environment variable MNTOPTIONS so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins. The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server. Archlinux and Fedora virtual machines are known to work properly. == Adding a git repo == Jenkins needs to be told what git repo contains the Linux source you want to compile. This is done by running $NFS_WORKSPACE/scripts/admin/add_git or by starting the job "Admin - Add Git" from the Jenkins web interface. ./add_git name git_url $name is the name used to identify this build job. In the Jenkins web interface, this will be expanded to "Linux - $name". $git_url is the url where the git repo can be found. For example, git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git When adding virtual machines for testing, be sure to add a "-$name" to the name of the VM. This will prevent conflicts later on, when installing or testing a kernel. If a VM is running tests, you don't want it to reboot after installing a different kernel! == Adding a Linux configuration file == Before Linux can be compiled, Jenkins needs to be told what .config files to use. This is done by running the script $NFS_WORKSPACE/scripts/admin/add_config or by starting the "Admin - Add Config" job from the Jenkins web interface. ./add_config name config_file architecture virtual_machine_name $name is the name of this particular configuration. In the Jenkins web interface, this will be expanded to "Linux - Config - $name" $config_file is the configuration file used when compiling. This will be copied to $NFS_WORKSPACE/config. Try not to add two configuration files with the same name. $architecture is the architecture of the virtual machine that will run the kernel and various tests. Use "i386" for a 32-bit machine and "x86_64" for a 64-bit machine. (See "Connectathon and Architectures" for more information) $virtual_machine_name is the name of the virtual machine that will boot the resulting kernel and run the tests. The git repo name will be appended automatically, so it is best to have one standard name for virtual machines running a specific .config file. For example, you could have the virtual machines nfs-pnfs-Trond and nfs-pnfs-Bruce for testing a pnfs .config. When running this script, enter nfs-pnfs as the virtual machine name. == Adding a new test == New tests are added through the web interface. From the home page, click "New Job" in the upper left side of the page. Enter a job name and choose the appropriate type of job. Fill out the next page with instructions for how the job should execute. It may be helpful to set a custom workspace (under "Advanced") to $NFS_WORKSPACE or some subdirectory. When the job has been created, click on "Tests - All" and choose "Configure" on the left. Scroll down to the Post-build actions and add a new parameterized build. Enter the name of the project and choose to pass the current build parameters down to your new test.

Return to Jenkins CI.