Jenkins CI

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
 
(13 intermediate revisions not shown)
Line 1: Line 1:
-
== Jenkins Setup ==
+
I'm going to skip over basic Jenkins usage to keep this document short.  You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.
-
Before doing anything, make sure the required environment variables are set
+
-
to correct values (see the section "Required Environment Variables" below).
+
-
Begin by cloning the jenkins-nfs git repository (git clone <LOCATION>).  Once
 
-
you have the latest version of the scripts, cd into jenkins-nfs and run ./setup
 
-
to download Jenkins, required plugins, and configure the jenkins environment.
 
-
In addition, a launcher script called start_jenkins will be placed in
 
-
$NFS_WORKSPACE directory.  Jenkins is then started by running ./start_jenkins.
 
-
If you ever move the jenkins.war file, be sure to update start_jenkins to
 
-
reflect the new location.
 
-
Before you can use Jenkins to test kernels, you must configure what tests to run
+
== Basic Setup ==
-
and virtual machines to run them on.  See the sections below for instructions
+
Begin by cloning the jenkins-nfs git repository:
-
on how to do this.
+
-
Once everything is configured, you can begin a new run by starting the job
+
<pre>$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git</pre>
-
"Compile and Install".
+
 +
The cloned directory should be placed in a location exported by the NFS server running on the host machine.  The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.
-
== Required Environment Variables ==
+
Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run:
-
These need to be set before starting Jenkins
+
-
* NFS_WORKSPACE
+
<pre>$ ./setup.sh</pre>
-
** The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory.  This directory should be accessable through NFS.
+
-
* NFS_SSH_USER
+
-
** The username of the virtual machine user that will install the compiled kernel and run the tests.  SSH should be configured to avoid prompting for a password.
+
-
* NFS_SERVER
+
-
** This is the ip address of the server that makes NFS_WORKSPACE available on the network.  The server is mounted to install compiled kernels and run the various tests.
+
-
* NFS_MOUNT
+
-
** The path on the NFS_SERVER to mount.
+
-
* NFS_MOUNT_POINT
+
-
** The path on the virtual machine that NFS_SERVER is mounted on.
+
-
The final mount command run will look similar to this:
+
to download the Jenkins executable, required plugins, and to configure the Jenkins environment.  This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins.  When you are ready, run
-
  mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT
+
 +
<pre>$ ./start_jenkins</pre>
-
== Adding a new architecture ==
+
to start Jenkins and point your web browser to localhost:8080 to see the web gui.
-
Jenkins is configured for i386 and x86_64 by default.  Other architectures can
+
-
be added, but manual configuration is required.  You will have to modify both
+
-
"Compile and Install" and "Run Tests".  Select each job from the dashboard and
+
-
then click "Configure" on the left.  Scroll down to the "Configuration Matrix"
+
-
and add the new architecture to the list of possible values for the ARCH
+
-
variable.
+
 +
Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing.  Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field.  Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.
-
== Adding a new .config ==
+
If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code.  You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).
-
Copy the config file to $NFS_WORKSPACE/config using a unique name.  Then modify
+
-
"Compile and Install" and "Run Tests" to add the new file to the CONFIG variable
+
-
in the "Test Matrix".
+
 +
You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.
-
== Adding a new virtual machine ==
 
-
Create the new VM in virt-manager.  Once it has been created, start the
 
-
"Initialize VM" job to configure passwordless SSH to the VM.  See the section
 
-
"Virtual machine setup" below for help creating new VMs.
 
 +
== config/nfsrc ==
 +
This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment.  Any changes here will not require you to restart Jenkins and should take effect almost immediately.
-
== Adding a new NFS version ==
+
;NFS_DEFAULT_SERVER
-
Edit the "Run Tests" job and add the new version to the list of possible
+
:This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified.  In most cases you probably want to set this to the machine hosting Jenkins.
-
values for the NFS variable.  NFS v2, v3, and v4 are configured by default.
+
-
Adding v4.1 will require modification of the mount command to add in the
+
-
"minorversion=1" mount option.
+
 +
;NFS_DEFAULT_VERSION
 +
:This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER.
-
== Adding a new test ==
+
;NFS_SOURCE_EXPORT
-
First design your test and edit $NFS_WORKSPACE/tests.map to add a mapping from
+
:This should be set to the directory exported by the NFS server hosting the compiled Linux code.
-
name -> command. The general format of this file is:
+
-
<pre>
+
-
        name:where:command
+
-
    name: The name of the test
+
;NFS_SOURCE_SERVER
-
    where: Where the command will be run (either "remote" or "local")
+
:This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER).
-
  command: The command to run the test
+
-
</pre>
+
-
Once the new test has been added to the tests.map file, add the new test to
+
;NFS_SOURCE_VERSION
-
the "Run Tests" job.
+
:This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION).
 +
;NFS_CLIENT_SCRIPTS
 +
:This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located.  If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/"
-
== Blacklisting a combination ==
+
;NFS_TEST_DEFAULT_TIMEOUT
-
Both "Compile and Install" and "Run Tests" are configured as Jenkins
+
:The default number of seconds to let a test run before killing the SSH job that launched it.  This may be overridden by tests that need a different value
-
multi-configuration projects.  This means that every combination of possible
+
-
variable values is tested.  Some combination of variables may not make sense,
+
-
such as running 32bit tests on a 64bit virtual machine.  You can modify the
+
-
"Combination Filter" field in the "Configuration Matrix" section to either
+
-
white-list or black-list a specific combination using a Groovy expression.  If
+
-
the expression evaluates to "True", the combination will be tested.
+
-
For example, say you have the architectures "i386" and "x86_64".  You also have
 
-
the virtual machines "jenkins32" (a 32bit VM) and "jenkins64" (a 64bit VM).
 
-
Only the combinations (x86_64, jenkins64) and (i386, jenkins32) should be run,
 
-
since (i386, jenkins64) and (x86_64, jenkins32) doesn't make sense.  The
 
-
following line accomplishes just that:
 
-
<pre>
 
-
        (ARCH=="x86_64" && MACHINE=="jenkins64") ||
 
-
            (ARCH=="i386" && MACHINE=="jenkins32")
 
-
</pre>
 
 +
== Compiling a different architecture ==
 +
Jenkins is configured to compile kernels x86_64 by default.  To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.
-
== Connectathon and Architectures ==
 
-
Connectathon tests are compiled for i386 and x86_64 architectures.  If you plan
 
-
any other architecture, you will need to manually compile connectathon yourself.
 
-
Once compiled, place the resulting directory in $NFS_WORKSPACE/cthon/ARCH
 
-
(where ARCH is the name of the new architecture).  Be sure to add the new
 
-
architecture to both "Compile and Install" and "Run Tests".
 
 +
== Adding a new .config ==
 +
Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name.  Then add the file to the NFS_CONFIG values list in the "Test Controller" job.
-
== $NFS_WORKSPACE/scripts/functions ==
 
-
This file provides many convenient functions for controlling virtual machines.
 
-
You can boot, halt, and reboot machines.  You can also do more complicated
 
-
actions like sending files, mounting the nfs server and running remote commands.
 
-
Before using any of these functions, you have to tell your script what machine
 
-
everything should be run on.  This is done through the set_machine function.
 
-
If you want to ls the $NFS_SSH_USER's home directory on the machine
 
-
jenkins-tests, you could do the following:
 
-
<pre>
 
-
#!/bin/bash
 
-
. functions
 
-
set_machine jenkins-tests
 
-
boot
 
-
run_cmd "ls"
 
-
halt
 
-
</pre>
 
-
 
+
== Machine setup ==
-
== Virtual machine setup ==
+
There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.
-
There are a few requirements for a virtual machine to work correctly for the
+
-
NFS Jenkins setup.
+
# Must have NFS client running
# Must have NFS client running
-
# Must be able to run connectathon tests
+
# Must be able to run connectathon ad xfs tests
-
# Must be able to run xfs tests
+
# Passwordless sudo for SSH_USER user
-
# Passwordless sudo for NFS_SSH_USER user
+
# Passwordless ssh enabled in the sshd config
-
# Passwordless ssh configured
+
-
The sample virtual machines come preconfigured with a valid user.  Log in with:
+
Grub must be configured to allow changing the default kernels.  This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved".  You may also need to create the file /boot/grub/default to store the new default value.
-
<pre>
+
-
  username: jenkins
+
-
  password: jenkins
+
-
</pre>
+
-
If you find you need root access:
+
Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels.  If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).
-
<pre>
+
-
  username: root
+
-
  password: jenkins
+
-
</pre>
+
-
Be sure to change the root password before hosting virtual machines publicly!
+
SSH keys are generated by running
-
You can download sample machines here:
+
-
* [http://www.linux-nfs.org/~bjschuma/jenkins_master.32.img 32-bit sample machine]
+
-
* [http://www.linux-nfs.org/~bjschuma/jenkins_master.64.img 64-bit sample machine]
+
-
Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm
+
<pre>$ $NFS_WORKSPACE/nfs/nfs-client-init</pre>
-
or by manually starting the job "Admin - Initialize VM".  This script will check
+
-
if ~/.ssh/id_rsa.pub (your public key) exists.  If it doesn't, then ssh-keygen
+
-
will be used to generate a key.  The public key will then be copied over to the
+
-
virtual machine and appended to the ~/.ssh/authorized_keys file.
+
-
This process will require you to enter your password twice, once for the file
+
or by manually starting the job "Admin - Add Client".  This script will check if ~/.ssh/id_rsa.pub (your public key) exists.  If it doesn't, then ssh-keygen will be used to generate a key.  The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
-
transfer and once for appending to the authorized_keys file.  Due to a
+
-
limitation in the Jenkins software, the password must be entered in the same
+
-
shell that Jenkins was initially started from.
+
-
If you intend to use NFS v3, you will have to an entry to /etc/hosts on both
+
This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.
-
the client and the server so that each machine is aware of the other.
+
-
If you intend to use NFS v2, you will need to set the environment variable
+
You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly.  To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
-
MNTOPTIONS so that the connectathon tests will work properly.  To do this, SSH
+
-
needs to allow user environments to be set up. This can be done by editing
+
-
/etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting
+
-
through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in
+
-
/home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
+
-
The install step will mount the server through NFS v4, so at the very least
+
Note that the default userid on Fedora is 500, while other distros use 1000.  This may have an effect on your ability to access files on the NFS server.
-
NFS v4 needs to be working on the client and server.
+
-
Archlinux and Fedora virtual machines are known to work properly.
 
 +
== Sample Machines ==
 +
Preconfigured Archlinux virtual machines can be downloaded from here:
-
== Ideas for future work ==
+
* [http://www.linux-nfs.org/~bjschuma/Jenkins32.img 32-bit sample machine]
-
* Give VM time to shut down by itself before forcing it off
+
* [http://www.linux-nfs.org/~bjschuma/Jenkins64.img 64-bit sample machine]
-
* Test against multiple servers
+
-
** Add server variables to multi-config project
+
-
* Compile connectathon / xfstests on each VM?
+
-
* Catch exit codes from remote commands for error checking
+
-
** Don't run with -e so that I can still do cleanup
+
-
** Put cleanup code in bash function
+
-
** Make new function for running command, checking exit value, and exiting script if necessarily
+
 +
These machines come preconfigured with a valid user.  Log in with:
 +
<pre>
 +
  username: jenkins
 +
  password: jenkins
 +
</pre>
 +
If you find you need root access:
 +
<pre>
 +
  username: root
 +
  password: jenkins
 +
</pre>
 +
Be sure to change the root password before hosting virtual machines publicly!
 +
See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.
 +
Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.
 +
== Bonus scripts ==
 +
All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework.  Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion).  Here is a partial list of commands:
 +
;nfs-backup
 +
:Creates a copy of important files from the current directory.  The backup will be in ~/.nfs/backups/[directory name hash].  "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup.  The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory.
 +
:Backups don't stack.  Only the most recent backup can be restored.
 +
;nfs-boot
 +
:Boots a machine.
 +
;nfs-client-init
 +
:Configures a client for use in other scripts.
 +
;nfs-client-show
 +
:Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels.
-
== Required Environment Variables ==
+
;nfs-config
-
These need to be set before starting Jenkins
+
:Get or set values in the current .config file.
-
* NFS_WORKSPACE
+
;nfs-halt
-
** The setup scripts will place .config files, helpful scripts and a fresh checkout of the Linux kernel in this directory.  This directory should be accessable through NFS.
+
:Halt a machine.
-
* NFS_SSH_USER
+
-
** The username of the virtual machine user that will install the compiled kernel and run the tests.  SSH should be configured to avoid prompting for a password.
+
-
* NFS_SERVER
+
-
** This is the ip address of the server that makes NFS_WORKSPACE available on the network.  The server is mounted to install compiled kernels and run the various tests.
+
-
* NFS_MOUNT
+
-
** The path on the NFS_SERVER to mount.
+
-
* NFS_MOUNT_POINT
+
-
** The path on the virtual machine that NFS_SERVER is mounted on.
+
-
* NFS_PYTHON_PATH
+
-
** The path to a python interpreter to use when executing python scripts. (Scripts have been written for Python 2.7)
+
-
The final mount command run will look similar to this:
+
;nfs-install
-
  mount $NFS_SERVER:$NFS_MOUNT $NFS_MOUNT_POINT
+
:Install a kernel on a machine.
 +
;nfs-make
 +
:Build a kernel.
-
== Connectathon and Architectures ==
+
;nfs-mount
-
When adding a virtual machine, one required argument is the architecture of the
+
:SSH into a client and mount a server over a given NFS version.
-
processor.  This is required for properly selecting the correct compiled
+
-
connectathon tests.  The tests are compiled with -m32 and -m64 as compiler
+
-
flags and placed in the $NFS_WORKSPACE/cthon/$ARCHITECTURE directory, where
+
-
$ARCHITECTURE is either "i386" or "x86_64".  You can add a virtual machine
+
-
with a different architecture, but you will be responsible for compiling the
+
-
connectathon tests and placing them in an appropriate directory.
+
-
 
+
-
 
+
-
== $NFS_WORKSPACE/scripts/functions ==
+
-
This file provides many convenient functions for controlling virtual machines.
+
-
You can boot, halt, and reboot machines.  You can also do more complicated
+
-
actions like sending files, mounting the nfs server and running remote commands.
+
-
Before using any of these functions, you have to tell your script what machine
+
-
everything should be run on.  This is done through the set_machine function.
+
-
If you want to ls the $NFS_SSH_USER's home directory on the machine
+
-
jenkins-tests, you could do the following:
+
-
<pre>
+
-
#!/bin/bash
+
-
. functions
+
-
set_machine jenkins-tests
+
-
boot
+
-
run_cmd "ls"
+
-
halt
+
-
</pre>
+
-
 
+
-
== Virtual machine requirements and initialization ==
+
-
There are a few requirements for a virtual machine to work correctly for the
+
-
NFS Jenkins setup.
+
-
 
+
-
# Must have NFS client running
+
-
# Must be able to run connectathon tests
+
-
# Must be able to run xfs tests
+
-
# Passwordless sudo for NFS_SSH_USER user
+
-
# Passwordless ssh configured
+
-
 
+
-
The sample virtual machines come preconfigured with a valid user.  Log in with:
+
-
  username: jenkins
+
-
  password: jenkins
+
-
 
+
-
If you find you need root access:
+
-
  username: root
+
-
  password: jenkins
+
-
 
+
-
Be sure to change the root password before hosting virtual machines publicly!
+
-
You can download sample machines here:
+
-
* [http://www.linux-nfs.org/~bjschuma/jenkins_master.32.img 32-bit sample machine]
+
-
* [http://www.linux-nfs.org/~bjschuma/jenkins_master.64.img 64-bit sample machine]
+
-
Passwordless ssh can be configured by running $NFS_WORKSPACE/scripts/init_vm
+
;nfs-reboot
-
or by manually starting the job "Admin - Initialize VM".  This script will check
+
:Reboot a machine.
-
if ~/.ssh/id_rsa.pub (your public key) exists.  If it doesn't, then ssh-keygen
+
-
will be used to generate a key.  The public key will then be copied over to the
+
-
virtual machine and appended to the ~/.ssh/authorized_keys file.
+
-
This process will require you to enter your password twice, once for the file
+
;nfs-restore
-
transfer and once for appending to the authorized_keys file.  Due to a
+
:The opposite of nfs-backup.  Restores files from the backup directory to the current directory.
-
limitation in the Jenkins software, the password must be entered in the same
+
-
shell that Jenkins was initially started from.
+
-
Virtual machines should be named based on the job it will be used to test (See
+
;nfs-run
-
"Adding a git repo" below).
+
:This script is like a "mini-jenkins" for running basic tests during development.  Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here].  `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests.
-
If you intend to use NFS v3, you will have to an entry to /etc/hosts on both
+
:;MACHINES
-
the client and the server so that each machine is aware of the other.
+
::Space separated list of machine names to use.
-
If you intend to use NFS v2, you will need to set the environment variable
+
:;NFS_VERSION
-
MNTOPTIONS so that the connectathon tests will work properly.  To do this, SSH
+
::NFS version to test over.
-
needs to allow user environments to be set up. This can be done by editing
+
-
/etc/ssh/sshd_config and setting PermitUserEnvironment to "yes".  When mounting
+
-
through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in
+
-
/home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
+
-
The install step will mount the server through NFS v4, so at the very least
+
:;NFS_SERVER
-
NFS v4 needs to be working on the client and server.
+
::Name of the server to test against.
-
Archlinux and Fedora virtual machines are known to work properly.
+
:;TEST_ITERATIONS
 +
::Number of times to mount, nfs_do_work, unmount.
 +
:;TEST_SLEEP
 +
::Seconds to sleep between tests.
-
== Adding a git repo ==
+
:;TEST_GRACE_WAIT
-
Jenkins needs to be told what git repo contains the Linux source you want to
+
::Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted.
-
compile.  This is done by running $NFS_WORKSPACE/scripts/admin/add_git or by
+
-
starting the job "Admin - Add Git" from the Jenkins web interface.
+
-
  ./add_git  name  git_url
+
:;REBOOT_AFTER_TEST
 +
::Set to "true" (default "true") if machines should be rebooted after testing.
-
$name is the name used to identify this build job.  In the Jenkins web interface,
+
:;UNINSTALL_AFTER_TEST
-
this will be expanded to "Linux - $name".
+
::Set to "true" (default "true") if kernels should be uninstalled from machines after testihng.
-
$git_url is the url where the git repo can be found.  For example,
+
:;NFS_POST_MOUNT
-
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
+
::Command to run after all mounts have finished.
-
When adding virtual machines for testing, be sure to add a "-$name" to the
+
:;NFS_POST_UMOUNT
-
name of the VM.  This will prevent conflicts later on, when installing or
+
::Command to run after all unmounts have finished.
-
testing a kernel.  If a VM is running tests, you don't want it to reboot after
+
-
installing a different kernel!
+
 +
:;NFS_EXIT
 +
::Command to run when `nfs run` exits.
-
== Adding a Linux configuration file ==
+
;nfs-server-init
-
Before Linux can be compiled, Jenkins needs to be told what .config files to use.
+
:Initializes a server. Similar to `nfs-client-init`
-
This is done by running the script $NFS_WORKSPACE/scripts/admin/add_config or by
+
-
starting the "Admin - Add Config" job from the Jenkins web interface.
+
-
  ./add_config name  config_file  architecture  virtual_machine_name
+
;nfs-server-show
 +
:Shows useful information about a server.  Similar to `nfs-client-show`
-
$name is the name of this particular configuration. In the Jenkins web
+
;nfs-show
-
interface, this will be expanded to "Linux - Config - $name"
+
:Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file.
-
$config_file is the configuration file used when compiling.  This will be copied
+
;nfs-ssh
-
to $NFS_WORKSPACE/config.  Try not to add two configuration files with
+
:SSH into a machine
-
the same name.
+
-
$architecture is the architecture of the virtual machine that will run the
+
;nfs-test
-
kernel and various tests.  Use "i386" for a 32-bit machine and "x86_64" for
+
:Run a test on a client against a server over a specific NFS version
-
a 64-bit machine.  (See "Connectathon and Architectures" for more information)
+
-
$virtual_machine_name is the name of the virtual machine that will boot the
+
;nfs-umount
-
resulting kernel and run the tests.  The git repo name will be appended
+
:Unmounts a client's NFS mountpoint
-
automatically, so it is best to have one standard name for virtual machines
+
-
running a specific .config file.  For example, you could have the virtual
+
-
machines nfs-pnfs-Trond and nfs-pnfs-Bruce for testing a pnfs .config.  When
+
-
running this script, enter nfs-pnfs as the virtual machine name.
+
 +
;nfs-uninstall
 +
:Removes a kernel from a client
-
== Adding a new test ==
+
These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory.  The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).
-
New tests are added through the web interface.  From the home page, click
+
-
"New Job" in the upper left side of the page. Enter a job name and choose the
+
-
appropriate type of job.  Fill out the next page with instructions for how
+
-
the job should execute.  It may be helpful to set a custom workspace (under
+
-
"Advanced") to $NFS_WORKSPACE or some subdirectory.
+
-
When the job has been created, click on "Tests - All" and choose "Configure"
+
Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it.  Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented.  Commands must be run with correct positional arguments.  This will hopefully change in the future.
-
on the left.  Scroll down to the Post-build actions and add a new parameterized
+
-
build.  Enter the name of the project and choose to pass the current build
+
-
parameters down to your new test.
+

Latest revision as of 20:32, 13 November 2012

I'm going to skip over basic Jenkins usage to keep this document short. You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.


Contents

Basic Setup

Begin by cloning the jenkins-nfs git repository: 

$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run: 

$ ./setup.sh

to download the Jenkins executable, required plugins, and to configure the Jenkins environment. This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins. When you are ready, run 

$ ./start_jenkins

to start Jenkins and point your web browser to localhost:8080 to see the web gui.

Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing. Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field. Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.

If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code. You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).

You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.


config/nfsrc

This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment. Any changes here will not require you to restart Jenkins and should take effect almost immediately.

NFS_DEFAULT_SERVER
This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified. In most cases you probably want to set this to the machine hosting Jenkins.
NFS_DEFAULT_VERSION
This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER.
NFS_SOURCE_EXPORT
This should be set to the directory exported by the NFS server hosting the compiled Linux code.
NFS_SOURCE_SERVER
This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER).
NFS_SOURCE_VERSION
This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION).
NFS_CLIENT_SCRIPTS
This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located. If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/"
NFS_TEST_DEFAULT_TIMEOUT
The default number of seconds to let a test run before killing the SSH job that launched it. This may be overridden by tests that need a different value


Compiling a different architecture

Jenkins is configured to compile kernels x86_64 by default. To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.


Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then add the file to the NFS_CONFIG values list in the "Test Controller" job.


Machine setup

There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.

  1. Must have NFS client running
  2. Must be able to run connectathon ad xfs tests
  3. Passwordless sudo for SSH_USER user
  4. Passwordless ssh enabled in the sshd config

Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.

Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).

SSH keys are generated by running 

$ $NFS_WORKSPACE/nfs/nfs-client-init

or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.


Sample Machines

Preconfigured Archlinux virtual machines can be downloaded from here:

These machines come preconfigured with a valid user. Log in with: 

   username: jenkins
   password: jenkins

If you find you need root access: 

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly! See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.

Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.


Bonus scripts

All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework. Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion). Here is a partial list of commands:

nfs-backup
Creates a copy of important files from the current directory. The backup will be in ~/.nfs/backups/[directory name hash]. "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup. The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory.
Backups don't stack. Only the most recent backup can be restored.
nfs-boot
Boots a machine.
nfs-client-init
Configures a client for use in other scripts.
nfs-client-show
Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels.
nfs-config
Get or set values in the current .config file.
nfs-halt
Halt a machine.
nfs-install
Install a kernel on a machine.
nfs-make
Build a kernel.
nfs-mount
SSH into a client and mount a server over a given NFS version.
nfs-reboot
Reboot a machine.
nfs-restore
The opposite of nfs-backup. Restores files from the backup directory to the current directory.
nfs-run
This script is like a "mini-jenkins" for running basic tests during development. Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here]. `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests.
MACHINES
Space separated list of machine names to use.
NFS_VERSION
NFS version to test over.
NFS_SERVER
Name of the server to test against.
TEST_ITERATIONS
Number of times to mount, nfs_do_work, unmount.
TEST_SLEEP
Seconds to sleep between tests.
TEST_GRACE_WAIT
Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted.
REBOOT_AFTER_TEST
Set to "true" (default "true") if machines should be rebooted after testing.
UNINSTALL_AFTER_TEST
Set to "true" (default "true") if kernels should be uninstalled from machines after testihng.
NFS_POST_MOUNT
Command to run after all mounts have finished.
NFS_POST_UMOUNT
Command to run after all unmounts have finished.
NFS_EXIT
Command to run when `nfs run` exits.
nfs-server-init
Initializes a server. Similar to `nfs-client-init`
nfs-server-show
Shows useful information about a server. Similar to `nfs-client-show`
nfs-show
Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file.
nfs-ssh
SSH into a machine
nfs-test
Run a test on a client against a server over a specific NFS version
nfs-umount
Unmounts a client's NFS mountpoint
nfs-uninstall
Removes a kernel from a client

These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory. The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).

Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it. Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented. Commands must be run with correct positional arguments. This will hopefully change in the future.

Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/Jenkins_CI"
Personal tools