Jenkins CI

From Linux NFS

I'm going to skip over basic Jenkins usage to keep this document short. You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.

Contents 1 Basic Setup 2 config/nfsrc 3 Compiling a different architecture 4 Adding a new .config 5 Machine setup 6 Sample Machines 7 Bonus scripts

Basic Setup

Begin by cloning the jenkins-nfs git repository:

$ git clone git://git.linux-nfs.org/projects/bjschuma/jenkins-nfs.git

The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.

Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run:

$ ./setup.sh

to download the Jenkins executable, required plugins, and to configure the Jenkins environment. This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins. When you are ready, run

$ ./start_jenkins

to start Jenkins and point your web browser to localhost:8080 to see the web gui.

Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing. Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field. Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.

If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code. You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).

You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.

config/nfsrc

This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment. Any changes here will not require you to restart Jenkins and should take effect almost immediately.

NFS_DEFAULT_SERVER

This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified. In most cases you probably want to set this to the machine hosting Jenkins.

NFS_DEFAULT_VERSION

This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER.

NFS_SOURCE_EXPORT

This should be set to the directory exported by the NFS server hosting the compiled Linux code.

NFS_SOURCE_SERVER

This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER).

NFS_SOURCE_VERSION

This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION).

NFS_CLIENT_SCRIPTS

This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located. If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/"

NFS_TEST_DEFAULT_TIMEOUT

The default number of seconds to let a test run before killing the SSH job that launched it. This may be overridden by tests that need a different value

Compiling a different architecture

Jenkins is configured to compile kernels x86_64 by default. To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.

Adding a new .config

Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then add the file to the NFS_CONFIG values list in the "Test Controller" job.

Machine setup

There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.

1. Must have NFS client running
2. Must be able to run connectathon ad xfs tests
3. Passwordless sudo for SSH_USER user
4. Passwordless ssh enabled in the sshd config

Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.

Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).

SSH keys are generated by running

$ $NFS_WORKSPACE/nfs/nfs-client-init

or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.

This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.

You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.

Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.

Sample Machines

Preconfigured Archlinux virtual machines can be downloaded from here:

• 32-bit sample machine
• 64-bit sample machine

These machines come preconfigured with a valid user. Log in with:

   username: jenkins
   password: jenkins

If you find you need root access:

   username: root
   password: jenkins

Be sure to change the root password before hosting virtual machines publicly! See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.

Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.

Bonus scripts

All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework. Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion). Here is a partial list of commands:

nfs-backup

Creates a copy of important files from the current directory. The backup will be in ~/.nfs/backups/[directory name hash]. "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup. The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory.

Backups don't stack. Only the most recent backup can be restored.

nfs-boot

Boots a machine.

nfs-client-init

Configures a client for use in other scripts.

nfs-client-show

Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels.

nfs-config

Get or set values in the current .config file.

nfs-halt

Halt a machine.

nfs-install

Install a kernel on a machine.

nfs-make

Build a kernel.

nfs-mount

SSH into a client and mount a server over a given NFS version.

nfs-reboot

Reboot a machine.

nfs-restore

The opposite of nfs-backup. Restores files from the backup directory to the current directory.

nfs-run

This script is like a "mini-jenkins" for running basic tests during development. Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here]. `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests.

MACHINES

Space separated list of machine names to use.

NFS_VERSION

NFS version to test over.

NFS_SERVER

Name of the server to test against.

TEST_ITERATIONS

Number of times to mount, nfs_do_work, unmount.

TEST_SLEEP

Seconds to sleep between tests.

TEST_GRACE_WAIT

Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted.

REBOOT_AFTER_TEST

Set to "true" (default "true") if machines should be rebooted after testing.

UNINSTALL_AFTER_TEST

Set to "true" (default "true") if kernels should be uninstalled from machines after testihng.

NFS_POST_MOUNT

Command to run after all mounts have finished.

NFS_POST_UMOUNT

Command to run after all unmounts have finished.

NFS_EXIT

Command to run when `nfs run` exits.

nfs-server-init

Initializes a server. Similar to `nfs-client-init`

nfs-server-show

Shows useful information about a server. Similar to `nfs-client-show`

nfs-show

Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file.

nfs-ssh

SSH into a machine

nfs-test

Run a test on a client against a server over a specific NFS version

nfs-umount

Unmounts a client's NFS mountpoint

nfs-uninstall

Removes a kernel from a client

These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory. The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).

Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it. Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented. Commands must be run with correct positional arguments. This will hopefully change in the future.