Jenkins CI

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
 
(5 intermediate revisions not shown)
Line 1: Line 1:
-
== Jenkins Setup ==
+
I'm going to skip over basic Jenkins usage to keep this document short.  You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.
 +
 
 +
 
 +
== Basic Setup ==
Begin by cloning the jenkins-nfs git repository:
Begin by cloning the jenkins-nfs git repository:
-
<pre>$ git clone git:///git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git</pre>
+
<pre>$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git</pre>
The cloned directory should be placed in a location exported by the NFS server running on the host machine.  The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.
The cloned directory should be placed in a location exported by the NFS server running on the host machine.  The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.
-
Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run ./setup to download the Jenkins executable, required plugins, and to configure the Jenkins environment.  In addition, a launcher script named start_jenkins will be placed in $NFS_WORKSPACE.  Before starting Jenkins, edit the config/jenkins_nfs.sh file to match your desired setup (see the config/jenkins_nfs.sh section below for more information about this file).  Once everything has been configured, start Jenkins by running:
+
Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run:
-
<pre>$ $NFS_WORKSPACE/start_jenkins</pre>
+
<pre>$ ./setup.sh</pre>
-
You must also configure what tests to run and virtual machines to run them on.  See the sections below for instructions on how to do this.  Once everything is configured, you can begin a new run by starting the job "Compile and Install".
+
to download the Jenkins executable, required plugins, and to configure the Jenkins environment.  This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins.  When you are ready, run
-
== config/jenkins_nfs.sh ==
+
<pre>$ ./start_jenkins</pre>
-
This script is used to set configuration variables that can affect how Jenkins behaves.  This file is sourced by scripts/functions.sh, so any changes you make will not require you to restart Jenkins.  Keep in mind that your changes may not be reflected in any currently running jobs.  The following variables should be set:
+
-
;HOST_SRV
+
to start Jenkins and point your web browser to localhost:8080 to see the web gui.
-
:This should be set to the ip address (or domain name) of the machine hosting Jenkins.  This machine should also be hosting the Linux source code you wish to compile and test.
+
-
;HOST_EXPORT
+
Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing.  Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field.  Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.
-
:This should be set to the path on $HOST_SRV that is being exported by $HOST_SRV.  For example, if you are exporting the directory /nfs/jenkins then you should set this variable to "/nfs/jenkins".
+
-
;HOST_MOUNT
+
If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code.  You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).
-
:This should be set to the mountpath used by the client to mount $HOST_EXPORT.  The client will mount $HOST_SRV through the command:
+
-
:<pre>$ mount -o vers=4 $HOST_SRV:$HOST_MOUNT</pre>
+
You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.
-
:This is needed due to public filehandles in NFS v4.  If the server is configured to export the directory "/nfs" but $HOST_EXPORT is set to "/nfs/jenkins", then HOST_MOUNT should be set to "/jenkins", resulting in the mount command:
 
-
:<pre>$ mount -o vers=4 $HOST_SRV:/jenkins</pre>
+
== config/nfsrc ==
 +
This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment.  Any changes here will not require you to restart Jenkins and should take effect almost immediately.
-
;OBJ_SRV (optional)
+
;NFS_DEFAULT_SERVER
-
:This should be set to the ip address (or domain name) of the machine that will store the compiled kernel objects.  This machine should already be mounted on the $HOST_SRV before compiling.  Leave this blank if you don't want to store compiled objects on a different machine.
+
:This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified.  In most cases you probably want to set this to the machine hosting Jenkins.
-
;OBJ_HOST_PATH (optional)
+
;NFS_DEFAULT_VERSION
-
:This should be set to the path on $HOST_MOUNT where $OBJ_SRV has already been mounted.  The compiled object files will be placed in $OBJ_HOST_PATH/jenkins/.  You can leave this blank if $OBJ_SRV has not been configured.
+
:This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER.
-
;OBJ_MOUNT (optional)
+
;NFS_SOURCE_EXPORT
-
:This should be set to the mountpath used by the client to mount $OBJ_SRV. The client will mount $OBJ_SRV through the command:
+
:This should be set to the directory exported by the NFS server hosting the compiled Linux code.
-
:<pre>$ mount -o vers=4 $OBJ_SRV:$OBJ_MOUNT</pre>
+
;NFS_SOURCE_SERVER
 +
:This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER).
-
;VERBOSE_SCRIPTS
+
;NFS_SOURCE_VERSION
-
:Set this to "true" if you want more output to be prined to the Jenkins console when jobs are run.  This can be used to help debug any problems during each job.
+
:This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION).
-
;SHOW_FIELDS
+
;NFS_CLIENT_SCRIPTS
-
:Set this to a space separated list of fields you want printed out at the beginning of each test.  Note that setting VERBOSE_SCRIPTS to "true" will automatically enable all fields.  Possible values are:
+
:This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located.  If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/"
-
:;uname
+
;NFS_TEST_DEFAULT_TIMEOUT
-
::Show the client's output of `uname -a`
+
:The default number of seconds to let a test run before killing the SSH job that launched it.  This may be overridden by tests that need a different value
-
:;path
+
-
::Show the client's $PATH
+
-
:;libc
+
-
::Show the client's current version of glibc
+
-
:;gcc
+
-
::Show the client's current version of gcc
+
-
:;make
+
-
::Show the client's current version of make
+
-
:;showmount
+
-
::Show the client's current version of showmount
+
-
:;mounts
+
-
::mounts: Show the client's mounted filesystems
+
-
;COLLECT_TRACES
 
-
:This variable is used to configure if packet traces should be collected during while running tests.  Setting this variable to "yes" will always collect traces, "no" will never collect traces.  A value of "onfail" will start a packet trace, but it will only be saved if the test fails.
 
-
:Packet traces are collected through tshark.  The sample VMs come preconfigured with passwordless tshark enabled for the "jenkins" user.
 
-
;MAKE_THREADS
+
== Compiling a different architecture ==
-
:This variable is used to set the number of threads each make process will use.  Setting MAKE_THREADS to "2" is the same as running:
+
Jenkins is configured to compile kernels x86_64 by default.  To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.
-
:<pre>$ make -j2</pre>
 
-
 
-
;BOOT_TIMEOUT
 
-
:This variable is used to determine the number of seconds to wait for a machine to boot and activate SSH.  If this timeout is exceeded the current job will fail, allowing the remaining queued jobs to run.
 
-
 
-
;COMPILE_TEST_SRC_DIR
 
-
:This variable is used to set the Linux source directory that should be compiled as part of the compile test.  This should be set relative to HOST_EXPORT.
 
-
 
-
== config/environment ==
 
-
SSH uses a different environment when running in non-interactive mode.  This can cause problems if a command cannot be located.  To fix this problem, users have the ability to configure their own environment.  The config/environment file is read and parsed before each ssh command is run.  You can enter any valid bash command here, similar to the way you would write your ~/.bashrc.
 
-
 
-
== Changing the Jenkins port ==
 
-
By default, Jenkins will run on port 8080 of the host machine.  This works well for machines only running a single instance of Jenkins, but this may not work for all setups.  For example, you may want to run multiple instances of Jenkins or you may already have a program running on port 8080.  To change the Jenkins port, edit the file $NFS_WORKSPACE/start_jenkins and change the "PORT=" variable to a different value.  You will need to restart Jenkins for this change to take effect.
 
-
 
-
== Adding a new architecture ==
 
-
Jenkins is configured for i386 and x86_64 by default.  Other architectures can be added, but manual configuration is required.  You will have to modify the "Compile", "Install" and "Run Tests" jobs.  Select each job from the dashboard and then click "Configure" on the left.  Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.
 
== Adding a new .config ==
== Adding a new .config ==
-
Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name.  Then modify the "Compile", "Install" and "Run Tests" jobs to add the new file to the CONFIG variable in the "Test Matrix".
+
Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name.  Then add the file to the NFS_CONFIG values list in the "Test Controller" job.
-
== Adding a pre-installed kernel ==
 
-
Jenkins can be configured to boot and test kernels that are already installed on the test machine.  If you wish to use this, first install the new kernel on your test machine.  Reboot into your new kernel and check the output from the command
 
-
<pre>$ uname -r</pre>
+
== Machine setup ==
 +
There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.
-
to find the kernel release name (such as "3.0-ARCH" for an Archlinux machine running Linux 3.0).  Create a new entry in your /boot/grub/menu.lst file for this kernel, using "3.0-ARCH" as the "title" parameter (replace "3.0-ARCH" with your `uname -r` output).  Finish configuring grub the way you normally would for that kernel.  Switch back to the Jenkins host and create a new config file for this kernel in the $NFS_WORKSPACE/config/kernel directory.  This file should contain a single line: `export UNAME="3.0-ARCH"` (once again replacing "3.0-ARCH" with your `uname -r` output.  Add this file to the "Run Tests" job, but do NOT add it to the "Compile" or "Install" jobs since this kernel has already been compiled and installed.  See the file $NFS_WORKSPACE/config/kernel/arch.default for an example config file.
+
# Must have NFS client running
 +
# Must be able to run connectathon ad xfs tests
 +
# Passwordless sudo for SSH_USER user
 +
# Passwordless ssh enabled in the sshd config
-
Note, you have just created the file $NFS_WORKSPACE/config/kernel/$CONFIG.  Jenkins will check for the existence of the directory $NFS_WORKSPACE/obj/$ARCH/$CONFIG when determining how to behave. If this directory does not exist, Jenkins will assume this is a precompiled kernel and read the kernel release name from $NFS_WORKSPACE/config/kernel/$CONFIG.  If the directory does exist, the include/config/kernel.release file will be used for the kernel release name.
+
Grub must be configured to allow changing the default kernels.  This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved".  You may also need to create the file /boot/grub/default to store the new default value.
-
== Adding a new virtual machine client ==
+
Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels.  If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).
-
A virtual machine client will be booted before use and halted when it is no longer being used.  Create the VM in virt-manager and start the "Admin - Add Client" job to configure passwordless SSH to the VM and install various tests (such as connectathon). See the section "Machine setup" below for help creating new VMs.  Input values for the following fields:
+
-
;NAME
+
SSH keys are generated by running
-
:This should be set to the name of the virtual machine as configured in virt-manager.
+
-
;ARCH
+
<pre>$ $NFS_WORKSPACE/nfs/nfs-client-init</pre>
-
:This should be set to the architecture of the installed kernel (usually i386 or x86_64).
+
-
;MOUNTPOINT
+
or by manually starting the job "Admin - Add Client".  This script will check if ~/.ssh/id_rsa.pub (your public key) exists.  If it doesn't, then ssh-keygen will be used to generate a key.  The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
-
:This is the full path on the client that the NFS server should be mounted on.
+
-
;SSH_USER
+
This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.
-
:This is the username of a user that has SSH access to the client.
+
-
;IS_VM
+
You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly.  To do this, SSH needs to allow user environments to be set up.  This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
-
:Make sure this is set to "true" to properly configure this client as a VM.
+
-
Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."
+
Note that the default userid on Fedora is 500, while other distros use 1000.  This may have an effect on your ability to access files on the NFS server.
-
== Adding a new real machine client ==
 
-
A real machine client should already be running before use and will never be halted by the Jenkins.  Be careful when using this option if you do not have physical access to the machine in case something goes wrong.  When the machine is running, start the "Admin - Add Client" job to configure passwordless SSH to the machine and install various tests (such as connectathon).  See the section "Machine setup" below for help configuring a new machine.  Input values for the following fields:
 
-
;NAME
+
== Sample Machines ==
-
:This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.
+
Preconfigured Archlinux virtual machines can be downloaded from here:
-
;ARCH
+
* [http://www.linux-nfs.org/~bjschuma/Jenkins32.img 32-bit sample machine]
-
:This should be set to the architecture of the installed kernel (usually i386 or x86_64). This will be used to to prevent installing a kernel with a different architecture.
+
* [http://www.linux-nfs.org/~bjschuma/Jenkins64.img 64-bit sample machine]
-
;MOUNTPOINT
+
These machines come preconfigured with a valid user. Log in with:
-
:This is the full path on the client that the NFS server should be mounted on.
+
-
;SSH_USER
+
<pre>
-
:This is the username of a user that has SSH access to the client.
+
  username: jenkins
 +
  password: jenkins
 +
</pre>
-
;IS_VM
+
If you find you need root access:
-
:Make sure this is set to "false" to properly configure this client as a real machine.
+
-
;IP
+
<pre>
-
:This should be set to the IP address of the machine so that it can be accessed through SSH.
+
  username: root
 +
  password: jenkins
 +
</pre>
-
Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."
+
Be sure to change the root password before hosting virtual machines publicly!
 +
See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.
-
== Adding a virtual machine server ==
+
Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.
-
Set up a virtual machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:
+
-
;NAME
 
-
:This should be set to the name of the virtual machine as configured in virt-manager.
 
-
;V2_MOUNT
+
== Bonus scripts ==
-
:This should be set to the path exported by the NFS v2 server.  Leave this blank if tests over NFS v2 should not be run.
+
All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework.  Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion).  Here is a partial list of commands:
-
;V3_MOUNT
+
;nfs-backup
-
:This should be set to the path exported by the NFS v3 server.  Leave this blank if tests over NFS v3 should not be run.
+
:Creates a copy of important files from the current directory.  The backup will be in ~/.nfs/backups/[directory name hash].  "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup.  The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory.
-
;V4_MOUNT
+
:Backups don't stack.  Only the most recent backup can be restored.
-
:This should be set to the path exported by the NFS v4 server.  Leave this blank if tests over NFS v4 should not be run.
+
-
;V41_MOUNT
+
;nfs-boot
-
:This should be set to the path exported by the NFS v4.1 server.  Leave this blank if tests over NFS v4.1 should not be run.
+
:Boots a machine.
-
;TEST_DIR
+
;nfs-client-init
-
:This should be set to a directory where test output should be placed.  If left blank, test output directories will be created in the root of the exported filesystem.
+
:Configures a client for use in other scripts.
-
;SSH_USER
+
;nfs-client-show
-
:This should be set to the username of a user that has SSH access to the server.  This is a mandatory field needed to halt the virtual machine when it is no longer being used.
+
:Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels.
-
;FAULT_INJECTION
+
;nfs-config
-
:This should be set to "true" if the server has been configured as an NFSD fault injection server, "false" otherwise.
+
:Get or set values in the current .config file.
-
;IS_VM
+
;nfs-halt
-
:This should be set to "true" to properly configure this server as a virtual machine.
+
:Halt a machine.
-
Once configured, edit the "Run Tests" job and add the new server to the SERVER variable of the "Test Matrix."
+
;nfs-install
 +
:Install a kernel on a machine.
-
== Adding a real machine server ==
+
;nfs-make
-
Set up a real machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:
+
:Build a kernel.
-
;NAME
+
;nfs-mount
-
:This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.
+
:SSH into a client and mount a server over a given NFS version.
-
;V2_MOUNT
+
;nfs-reboot
-
:This should be set to the path exported by the NFS v2 server.  Leave this blank if tests over NFS v2 should not be run.
+
:Reboot a machine.
-
;V3_MOUNT
+
;nfs-restore
-
:This should be set to the path exported by the NFS v3 server.  Leave this blank if tests over NFS v3 should not be run.
+
:The opposite of nfs-backup.  Restores files from the backup directory to the current directory.
-
;V4_MOUNT
+
;nfs-run
-
:This should be set to the path exported by the NFS v4 server.  Leave this blank if tests over NFS v4 should not be run.
+
:This script is like a "mini-jenkins" for running basic tests during development.  Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here].  `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests.
-
;V41_MOUNT
+
:;MACHINES
-
:This should be set to the path exported by the NFS v4.1 server.  Leave this blank if tests over NFS v4.1 should not be run.
+
::Space separated list of machine names to use.
-
;TEST_DIR
+
:;NFS_VERSION
-
:This should be set to a directory where test output should be placed.  If left blank, test output directories will be created in the root of the exported filesystem.
+
::NFS version to test over.
-
;SSH_USER
+
:;NFS_SERVER
-
:This should be set to the username of a user that has SSH access to the server.  This field is optional and will only be used if the server is configured to run fault injection tests.
+
::Name of the server to test against.
-
;FAULT_INJECTION
+
:;TEST_ITERATIONS
-
:This should be set to "true" if the server has been configured as an NFSD fault injection server, "false" otherwise.
+
::Number of times to mount, nfs_do_work, unmount.
-
;IS_VM
+
:;TEST_SLEEP
-
:This should be set to "false" to properly configure this server as a real machine.
+
::Seconds to sleep between tests.
-
;IP
+
:;TEST_GRACE_WAIT
-
:This should be set to the IP address of the machine so it can be mounted by clients and accessed through SSH.
+
::Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted.
-
Once configured, edit the "Run Tests" job and add the new server to the SERVER ariable of the "Test Matrix."
+
:;REBOOT_AFTER_TEST
 +
::Set to "true" (default "true") if machines should be rebooted after testing.
-
== Adding a new NFS version ==
+
:;UNINSTALL_AFTER_TEST
-
Edit the "Run Tests" job and add the new version to the list of possible values for the NFS variable.  NFS v2, v3, and v4 are configured by default.  Adding v4.1 will require modification of the mount command to add in the "minorversion=1" mount option.
+
::Set to "true" (default "true") if kernels should be uninstalled from machines after testihng.
-
== Adding a new test ==
+
:;NFS_POST_MOUNT
-
First determine if your test can be executed as a simple command over ssh.  If not, you should write a new script to run your test. Next run the "Admin - Add Test" job to create a new test file.  Input values for the following fields:
+
::Command to run after all mounts have finished.
-
;NAME
+
:;NFS_POST_UMOUNT
-
:This is simply an identifier to help keep track of what the test will do and to make it easier to reference in the Run Tests job.
+
::Command to run after all unmounts have finished.
-
;ENABLED
+
:;NFS_EXIT
-
:When a test is disabled, it will not run when triggered by the Run Tests job.  Set this to "No" if you want to create the test, but do not want it to run right away.
+
::Command to run when `nfs run` exits.
-
;FAULT_INJECTION
+
;nfs-server-init
-
:This should be set to "Yes" if the test makes use of NFSD fault injection.  This does require a special Linux server that has been configured with the NFSD fault injection patches.
+
:Initializes a server.  Similar to `nfs-client-init`
-
 
+
-
;SSH_COMMAND
+
-
:Set this to "Yes" if your test is a simple command that can be run over ssh.  Set this to "No" to run a command or script on the machine hosting Jenkins.
+
-
 
+
-
;COMMAND
+
-
:Set this to the command that should be run as the main part of the test.
+
-
 
+
-
The following variables are available for use when writing your COMMAND:
+
-
 
+
-
:;$CLIENT
+
-
::Virtual machine to run test on
+
-
:;$SERVER
+
-
::Server being tested against
+
-
:;$ARCH
+
-
::Architecture of virtual machine
+
-
:;$CONFIG
+
-
::Config file used to compile the kernel to be tested
+
-
:;$NFS
+
-
::The NFS version to use
+
-
:;$MNT_POINT
+
-
::Location on the client that the server has been mounted to
+
-
:;$TEST_DIR
+
-
::Directory test output should be written to
+
-
:;$WS
+
-
::Full path to the Jenkins workspace once the server has been mounted
+
-
:;$TEST
+
-
::The name of the test to run
+
-
 
+
-
Additionally, the bash function `test_scripts_dir` can be used as a shortcut to $NFS_WORKSPACE/scripts if you do not want the $NFS_WORKSPACE variable to be expanded.
+
-
 
+
-
== Editing a test ==
+
-
Tests are configured through files placed in $NFS_WORKSPACE/config/tests.  You can modify a test by editing the appropriate file in that directory.  The following fields can be modified:
+
-
 
+
-
;ENABLED
+
-
:This variable determines if the test should be run or not.  Change from "true" to "false" to disable a test.
+
-
 
+
-
;FAULT_INJECTION
+
-
:This variable is used during automatic filtering of fault injection tests.  Change to "true" if your test makes use of the NFSD fault injection framework
+
-
 
+
-
;do_test()
+
-
:This function is called to run the test.  The command in this function is set based on the COMMAND parameter passed to the "Admin - Add Test" job.
+
-
 
+
-
== Test Groups ==
+
-
If you have a lot of tests to run, you may want to consider breaking them into multiple projects featuring a smaller number of tests.  This will create more jobs to manage, but you will more easily be able to tell if different "types" of tests pass or fail.
+
-
 
+
-
To begin with, clone the default "Run Tests" job that has already been configured.  From the main Jenkins page:
+
-
 
+
-
# Follow the "New Job" link at the top left-hand corner of the page
+
-
# Fill out the "Job name" field on the new job configuration page
+
-
# Select "Copy existing Job" at the end of the list
+
-
# Choose to copy from "Run Tests"
+
-
# Edit this job so it does what you want (see the previous sections beginning with "Adding a new architecture")
+
-
 
+
-
At this point, the new job should be created, but it will not run automatically after the "Install" job has completed.  If you want the new tests scheduled after the "Install" job:
+
-
 
+
-
# Open up the configuration page for the "Install" job
+
-
# Scroll to the bottom of the page
+
-
# Find the "Trigger parameterized build on other projects" line and edit the "Projects to build" field and add your new test job to the comma separated list of projects.
+
-
 
+
-
== Blacklisting a combination ==
+
-
"Compile," "Install" and "Run Tests" are configured as Jenkins multi-configuration projects.  This means that every combination of possible variable values will be run.  Some combination of variables may not make sense, such as running 32-bit tests on a 64-bit client.  The files in $NFS_WORKSPACE/config/filters/ can be used to blacklist any combination that doesn't make sense for your setup.  These files are simple shell scripts that hould exit 0 if the combination can be tested.
+
-
 
+
-
The following invalid combinations will automatically be detected and filtered out by Jenkins:
+
-
 
+
-
# The architecture variable does not match the architecture of the machine.
+
-
# The server has not been configured to run the requested version of NFS.
+
-
 
+
-
See the files in $NFS_WORKSPACE/config/filters/ for more information when writing your own filters.
+
-
 
+
-
== Machine setup ==
+
-
There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.
+
-
 
+
-
# Must have NFS client running
+
-
# Must have prerequisites for running the connectathon tests
+
-
# Passwordless sudo for SSH_USER user
+
-
# Passwordless ssh enabled in the sshd config
+
-
# (optional) Have prerequisites for running the xfs tests
+
-
# (optional) Have tshark installed and configured for the SSH_USER user
+
-
 
+
-
Grub must be configured to allow changing the default kernels.  This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved".  You may also need to create the file /boot/grub/default to store the new default value.
+
-
 
+
-
Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels.  If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).
+
-
 
+
-
Passwordless ssh can be configured by running
+
-
 
+
-
<pre>$ $NFS_WORKSPACE/scripts/init_client</pre>
+
-
 
+
-
or by manually starting the job "Admin - Add Client".  This script will check if ~/.ssh/id_rsa.pub (your public key) exists.  If it doesn't, then ssh-keygen will be used to generate a key.  The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
+
-
 
+
-
This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.
+
-
 
+
-
You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly.  To do this, SSH needs to allow user environments to be set up.  This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
+
-
 
+
-
The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server.  Note that the default userid on Fedora is 500, while other distros use 1000.  This may have an effect on your ability to access files on the NFS server.
+
-
 
+
-
If you intend to collect packet traces, you will need to install the tshark program (it may be packaged with wireshark).  Once installed, add your ssh user to the wireshark group so packets can be safely captured
+
-
 
+
-
== Sample Machines ==
+
-
Preconfigured Archlinux virtual machines can be downloaded from here:
+
-
 
+
-
* [http://www.linux-nfs.org/~bjschuma/Jenkins32.img 32-bit sample machine]
+
-
* [http://www.linux-nfs.org/~bjschuma/Jenkins64.img 64-bit sample machine]
+
-
 
+
-
These machines come preconfigured with a valid user.  Log in with:
+
-
 
+
-
<pre>
+
-
  username: jenkins
+
-
  password: jenkins
+
-
</pre>
+
-
 
+
-
If you find you need root access:
+
-
 
+
-
<pre>
+
-
  username: root
+
-
  password: jenkins
+
-
</pre>
+
-
 
+
-
Be sure to change the root password before hosting virtual machines publicly!
+
-
Archlinux and Fedora based machines are known to work properly
+
-
== The Job Pool ==
+
;nfs-server-show
-
Jenkins-NFS works best when configured with a large job pool, since a larger pool will increase the chances of a test running on an already-booted kernel.  Most of the testing during Jenkins-NFS development was performed with a job pool of size 8.  To increase the job pool size:
+
:Shows useful information about a server.  Similar to `nfs-client-show`
-
# From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
+
;nfs-show
-
# Follow the "Configure System" link
+
:Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file.
-
# Find the "# of executors" field and change to your desired value
+
-
== Updating Jenkins ==
+
;nfs-ssh
-
The Jenkins development team realeases a new jenkins.war file every week or two.  The updated war can be downloaded and installed automatically from your currenly running version.  To update Jenkins:
+
:SSH into a machine
-
# From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
+
;nfs-test
-
# At the top of the page, there may be a message saying that a new version is available.  If you do not see this message, then you are already running the latest version of Jenkins
+
:Run a test on a client against a server over a specific NFS version
-
# If you do see the new version message, click the "Or Upgrade Automatically" button to begin the upgrade process
+
-
# (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the new version has been installed.  If you don't do this, you must manually stop and start Jenkins to switch to the new version
+
-
== Installing and Upgrading Plugins ==
+
;nfs-umount
-
Plugins can add new features that are not included with the base jenkins.war program.  To locate and install a new plugin:
+
:Unmounts a client's NFS mountpoint
-
# From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
+
;nfs-uninstall
-
# Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
+
:Removes a kernel from a client
-
# Click the "Available" tab
+
-
# Browse the list of available plugins, selecting the ones you wish to install
+
-
# Click the "Install" button at the bottom of the page
+
-
# (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been installed.  If you don't do this, you must manually stop and start Jenkins to enable your new plugins
+
-
Developers will occasionally release a new version of their plugin.  To update your installed plugins:
+
These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory.  The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).
-
# From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
+
Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it.  Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented.  Commands must be run with correct positional arguments.  This will hopefully change in the future.
-
# Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
+
-
# Select the plugins you wish to update
+
-
# Click the "Install" button at the bottom of the page
+
-
# (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been updated.  If you don't do this, you must manually stop and start Jenkins to use the updated plugins
+

Latest revision as of 20:32, 13 November 2012

I'm going to skip over basic Jenkins usage to keep this document short. You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.


Contents

Basic Setup

Begin by cloning the jenkins-nfs git repository: 

$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run: 

$ ./setup.sh

to download the Jenkins executable, required plugins, and to configure the Jenkins environment. This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins. When you are ready, run 

$ ./start_jenkins

to start Jenkins and point your web browser to localhost:8080 to see the web gui.

Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing. Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field. Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.

If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code. You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).

You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.


config/nfsrc

This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment. Any changes here will not require you to restart Jenkins and should take effect almost immediately.

NFS_DEFAULT_SERVER
This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified. In most cases you probably want to set this to the machine hosting Jenkins.
NFS_DEFAULT_VERSION
This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER.
NFS_SOURCE_EXPORT
This should be set to the directory exported by the NFS server hosting the compiled Linux code.
NFS_SOURCE_SERVER
This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER).
NFS_SOURCE_VERSION
This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION).
NFS_CLIENT_SCRIPTS
This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located. If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/"
NFS_TEST_DEFAULT_TIMEOUT
The default number of seconds to let a test run before killing the SSH job that launched it. This may be overridden by tests that need a different value


Compiling a different architecture

Jenkins is configured to compile kernels x86_64 by default. To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.


Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then add the file to the NFS_CONFIG values list in the "Test Controller" job.


Machine setup

There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.

  1. Must have NFS client running
  2. Must be able to run connectathon ad xfs tests
  3. Passwordless sudo for SSH_USER user
  4. Passwordless ssh enabled in the sshd config

Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.

Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).

SSH keys are generated by running 

$ $NFS_WORKSPACE/nfs/nfs-client-init

or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.


Sample Machines

Preconfigured Archlinux virtual machines can be downloaded from here:

These machines come preconfigured with a valid user. Log in with: 

   username: jenkins
   password: jenkins

If you find you need root access: 

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly! See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.

Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.


Bonus scripts

All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework. Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion). Here is a partial list of commands:

nfs-backup
Creates a copy of important files from the current directory. The backup will be in ~/.nfs/backups/[directory name hash]. "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup. The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory.
Backups don't stack. Only the most recent backup can be restored.
nfs-boot
Boots a machine.
nfs-client-init
Configures a client for use in other scripts.
nfs-client-show
Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels.
nfs-config
Get or set values in the current .config file.
nfs-halt
Halt a machine.
nfs-install
Install a kernel on a machine.
nfs-make
Build a kernel.
nfs-mount
SSH into a client and mount a server over a given NFS version.
nfs-reboot
Reboot a machine.
nfs-restore
The opposite of nfs-backup. Restores files from the backup directory to the current directory.
nfs-run
This script is like a "mini-jenkins" for running basic tests during development. Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here]. `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests.
MACHINES
Space separated list of machine names to use.
NFS_VERSION
NFS version to test over.
NFS_SERVER
Name of the server to test against.
TEST_ITERATIONS
Number of times to mount, nfs_do_work, unmount.
TEST_SLEEP
Seconds to sleep between tests.
TEST_GRACE_WAIT
Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted.
REBOOT_AFTER_TEST
Set to "true" (default "true") if machines should be rebooted after testing.
UNINSTALL_AFTER_TEST
Set to "true" (default "true") if kernels should be uninstalled from machines after testihng.
NFS_POST_MOUNT
Command to run after all mounts have finished.
NFS_POST_UMOUNT
Command to run after all unmounts have finished.
NFS_EXIT
Command to run when `nfs run` exits.
nfs-server-init
Initializes a server. Similar to `nfs-client-init`
nfs-server-show
Shows useful information about a server. Similar to `nfs-client-show`
nfs-show
Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file.
nfs-ssh
SSH into a machine
nfs-test
Run a test on a client against a server over a specific NFS version
nfs-umount
Unmounts a client's NFS mountpoint
nfs-uninstall
Removes a kernel from a client

These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory. The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).

Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it. Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented. Commands must be run with correct positional arguments. This will hopefully change in the future.

Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/Jenkins_CI"
Personal tools