|
|
| Line 1: |
Line 1: |
| - | == Jenkins Setup == | + | I'm going to skip over basic Jenkins usage to keep this document short. You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program. |
| | + | |
| | + | |
| | + | == Basic Setup == |
| | Begin by cloning the jenkins-nfs git repository: | | Begin by cloning the jenkins-nfs git repository: |
| | | | |
| Line 8: |
Line 11: |
| | Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run: | | Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run: |
| | | | |
| - | <pre>$ ./setup</pre> | + | <pre>$ ./setup.sh</pre> |
| | | | |
| - | to download the Jenkins executable, required plugins, and to configure the Jenkins environment. In addition, a launcher script named start_jenkins will be placed in $NFS_WORKSPACE. Before starting Jenkins, edit the config/jenkins_nfs.sh file to match your desired setup (see the config/jenkins_nfs.sh section below for more information about this file). Once everything has been configured, start Jenkins by running: | + | to download the Jenkins executable, required plugins, and to configure the Jenkins environment. This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins. When you are ready, run |
| | | | |
| - | <pre>$ $NFS_WORKSPACE/start_jenkins</pre> | + | <pre>$ ./start_jenkins</pre> |
| | | | |
| - | This will start the Jenkins web server as a background process using nohup. The web server runs on port 8080 by default so you can access it by pointing your web browser to http://localhost:8080. Output from the Jenkins process will be appended to $NFS_WORKSPACE/jenkins.log.
| + | to start Jenkins and point your web browser to localhost:8080 to see the web gui. |
| | | | |
| - | Finally, you must also configure what tests to run and virtual machines to run them on. See the sections below for instructions on how to do this. Once everything is configured, you can begin a new run by starting the job "Compile and Install" from the web interface.
| + | Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing. Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field. Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field. |
| | | | |
| - | The jenkins-nfs scripts do not have an official version, and instead are considered "rolling release". This means that at any point in time, you are one `git pull` away from the latest version. However, doing a plain `git pull` may cause merge conflicts with important configuraion files (like your config/jenkins_nfs.sh). The provided $NFS_WORKSPACE/update script should help with this problem. It will stash your modifications and then use `git rebase` to apply your changes on toy of the most recent patches. Additionally, the update script will check if your installed Jenkins jobs (~/.jenkins/jobs/) have been updated in git. You can either (a) accept the changes and copy over the new config file, (m) open vimdiff to manually merge the changes or (anything else) do nothing and go to the next file.
| + | If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code. You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...). |
| | | | |
| - | == config/jenkins_nfs.sh ==
| + | You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first. |
| - | This script is used to set configuration variables that can affect how Jenkins behaves. This file is sourced by scripts/functions.sh, so any changes you make will not require you to restart Jenkins. Keep in mind that your changes may not be reflected in any currently running jobs. The following variables should be set:
| + | |
| | | | |
| - | ;HOST_SRV
| |
| - | :This should be set to the ip address (or domain name) of the machine hosting Jenkins. This machine should also be hosting the Linux source code you wish to compile and test.
| |
| | | | |
| - | ;HOST_EXPORT
| + | == config/nfsrc == |
| - | :This should be set to the path on $HOST_SRV that is being exported by $HOST_SRV. For example, if you are exporting the directory /nfs/jenkins then you should set this variable to "/nfs/jenkins".
| + | This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment. Any changes here will not require you to restart Jenkins and should take effect almost immediately. |
| | | | |
| - | ;HOST_MOUNT | + | ;NFS_DEFAULT_SERVER |
| - | :This should be set to the mountpath used by the client to mount $HOST_EXPORT. The client will mount $HOST_SRV through the command: | + | :This should be set to the name (see `nfs server init`) of the server used by the nfs* commands when no server is specified. In most cases you probably want to set this to the machine hosting Jenkins. |
| | | | |
| - | :<pre>$ mount -o vers=4 $HOST_SRV:$HOST_MOUNT</pre> | + | ;NFS_DEFAULT_VERSION |
| | + | :This should be set to the NFS version used by clients when mounting $NFS_DEFAULT_SERVER. |
| | | | |
| - | :This is needed due to public filehandles in NFS v4. If the server is configured to export the directory "/nfs" but $HOST_EXPORT is set to "/nfs/jenkins", then HOST_MOUNT should be set to "/jenkins", resulting in the mount command: | + | ;NFS_SOURCE_EXPORT |
| | + | :This should be set to the directory exported by the NFS server hosting the compiled Linux code. |
| | | | |
| - | :<pre>$ mount -o vers=4 $HOST_SRV:/jenkins</pre> | + | ;NFS_SOURCE_SERVER |
| | + | :This should be set to the machine hosting the compiled Linux code (most likely $NFS_DEFAULT_SERVER). |
| | | | |
| - | ;OBJ_SRV (optional) | + | ;NFS_SOURCE_VERSION |
| - | :This should be set to the ip address (or domain name) of the machine that will store the compiled kernel objects. This machine should already be mounted on the $HOST_SRV before compiling. Leave this blank if you don't want to store compiled objects on a different machine. | + | :This should be set to the NFS version used to install the kernel from $NFS_SOURCE_SERVER (most likely $NFS_DEFAULT_VERSION). |
| | | | |
| - | ;OBJ_HOST_PATH (optional) | + | ;NFS_CLIENT_SCRIPTS |
| - | :This should be set to the path on $HOST_MOUNT where $OBJ_SRV has already been mounted. The compiled object files will be placed in $OBJ_HOST_PATH/jenkins/. You can leave this blank if $OBJ_SRV has not been configured. | + | :This is a path relative to $NFS_SOURCE_EXPORT/ where the client scripts are located. If $NFS_WORKSPACE is "/home/bjschuma/jenkins/jenkins-nfs" and $NFS_SOURCE_EXPORT is "/home/bjschuma" then you should set this to "jenkins/jenkins-nfs/client/" |
| | | | |
| - | ;OBJ_MOUNT (optional) | + | ;NFS_TEST_DEFAULT_TIMEOUT |
| - | :This should be set to the mountpath used by the client to mount $OBJ_SRV. The client will mount $OBJ_SRV through the command: | + | :The default number of seconds to let a test run before killing the SSH job that launched it. This may be overridden by tests that need a different value |
| | | | |
| - | :<pre>$ mount -o vers=4 $OBJ_SRV:$OBJ_MOUNT</pre>
| |
| | | | |
| - | ;VERBOSE_SCRIPTS
| + | == Compiling a different architecture == |
| - | :Set this to "true" if you want more output to be prined to the Jenkins console when jobs are run. This can be used to help debug any problems during each job.
| + | Jenkins is configured to compile kernels x86_64 by default. To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis. |
| | | | |
| - | ;USE_DISTCC
| |
| - | :Set this to "true" if you want to use distcc during the Compile job. Set it to "pump" if you also want to use the pump mode of distcc 3 (see distcc.org for more information). If you set this variable to "true" or "pump" then you must also configure the DISTCC_SERVERS_64 and DISTCC_SERVERS_32 variables.
| |
| - |
| |
| - | ;DISTCC_SERVERS_64
| |
| - | :If you are using distcc for compiling, set this to the list of servers that support compiling a 64-bit Linux kernel. It is a good idea to at least set this to "localhost".
| |
| - |
| |
| - | ;DISTCC_SERVERS_32
| |
| - | :If you are using distcc for compiling, set this to the list of servers that support compiling a 32-bit Linux kernel. It is a good idea to at least set this to "localhost".
| |
| - |
| |
| - | ;DELETE_COMPILED_OUTPUT
| |
| - | :Set this to "true" if you want to delete the compiled output during the cleanup phase. This will free up space on your disk, but you will have to recompile kernels during the next run
| |
| - |
| |
| - | ;REMOVE_INSTALLED_KERNELS
| |
| - | :Set this to "true" if you want to uninstall kernels from the client machine at the end of testing. This will free up disk space on the clients, but kernels will have to be reinstalled before you can run tests.
| |
| - |
| |
| - | ;SHOW_FIELDS
| |
| - | :Set this to a space separated list of fields you want printed out at the beginning of each test. Note that setting VERBOSE_SCRIPTS to "true" will automatically enable all fields. Possible values are:
| |
| - |
| |
| - | :;uname
| |
| - | ::Show the client's output of `uname -a`
| |
| - | :;path
| |
| - | ::Show the client's $PATH
| |
| - | :;libc
| |
| - | ::Show the client's current version of glibc
| |
| - | :;gcc
| |
| - | ::Show the client's current version of gcc
| |
| - | :;make
| |
| - | ::Show the client's current version of make
| |
| - | :;showmount
| |
| - | ::Show the client's current version of showmount
| |
| - | :;mounts
| |
| - | ::mounts: Show the client's mounted filesystems
| |
| - |
| |
| - | ;COLLECT_TRACES
| |
| - | :This variable is used to configure if packet traces should be collected during while running tests. Setting this variable to "yes" will always collect traces, "no" will never collect traces. A value of "onfail" will start a packet trace, but it will only be saved if the test fails.
| |
| - | :Packet traces are collected through tshark. The sample VMs come preconfigured with passwordless tshark enabled for the "jenkins" user.
| |
| - |
| |
| - | ;MAKE_THREADS
| |
| - | :This variable is used to set the number of threads each make process will use. Setting MAKE_THREADS to "2" is the same as running:
| |
| - |
| |
| - | :<pre>$ make -j2</pre>
| |
| - |
| |
| - | ;BOOT_TIMEOUT
| |
| - | :This variable is used to determine the number of seconds to wait for a machine to boot and activate SSH. If this timeout is exceeded the current job will fail, allowing the remaining queued jobs to run.
| |
| - |
| |
| - | ;COMPILE_TEST_SRC_DIR
| |
| - | :This variable is used to set the Linux source directory that should be compiled as part of the compile test. This should be set relative to HOST_EXPORT.
| |
| - |
| |
| - | == config/environment ==
| |
| - | SSH uses a different environment when running in non-interactive mode. This can cause problems if a command cannot be located. To fix this problem, users have the ability to configure their own environment. The config/environment file is read and parsed before each ssh command is run. You can enter any valid bash command here, similar to the way you would write your ~/.bashrc.
| |
| - |
| |
| - | == Changing the Jenkins port ==
| |
| - | By default, Jenkins will run on port 8080 of the host machine. This works well for machines only running a single instance of Jenkins, but this may not work for all setups. For example, you may want to run multiple instances of Jenkins or you may already have a program running on port 8080. To change the Jenkins port, edit the file $NFS_WORKSPACE/start_jenkins and change the "PORT=" variable to a different value. You will need to restart Jenkins for this change to take effect.
| |
| - |
| |
| - | == Adding a new architecture ==
| |
| - | Jenkins is configured to compile kernels for i386 and x86_64 by default. Other architectures can be added, but manual configuration of the "Compile" job is required. Select this job from the dashboard and then click "Configure" on the left. Scroll down to the "Configuration Matrix" and add the new architecture to the list of possible values for the ARCH variable.
| |
| | | | |
| | == Adding a new .config == | | == Adding a new .config == |
| - | Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then modify the "Compile", "Install" and "Run Tests" jobs to add the new file to the CONFIG variable in the "Test Matrix". | + | Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then add the file to the NFS_CONFIG values list in the "Test Controller" job. |
| | | | |
| - | == Adding a pre-installed kernel ==
| |
| - | Jenkins can be configured to boot and test kernels that are already installed on the test machine. If you wish to use this, first install the new kernel on your test machine. Reboot into your new kernel and check the output from the command
| |
| | | | |
| - | <pre>$ uname -r</pre>
| + | == Machine setup == |
| | + | There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup. |
| | | | |
| - | to find the kernel release name (such as "3.0-ARCH" for an Archlinux machine running Linux 3.0). Create a new entry in your /boot/grub/menu.lst file for this kernel, using "3.0-ARCH" as the "title" parameter (replace "3.0-ARCH" with your `uname -r` output). Finish configuring grub the way you normally would for that kernel. Switch back to the Jenkins host and create a new config file for this kernel in the $NFS_WORKSPACE/config/kernel directory. This file should contain a single line: `export UNAME="3.0-ARCH"` (once again replacing "3.0-ARCH" with your `uname -r` output. Add this file to the "Run Tests" job, but do NOT add it to the "Compile" or "Install" jobs since this kernel has already been compiled and installed. See the file $NFS_WORKSPACE/config/kernel/arch.default for an example config file.
| + | # Must have NFS client running |
| | + | # Must be able to run connectathon ad xfs tests |
| | + | # Passwordless sudo for SSH_USER user |
| | + | # Passwordless ssh enabled in the sshd config |
| | | | |
| - | Note, you have just created the file $NFS_WORKSPACE/config/kernel/$CONFIG. Jenkins will check for the existence of the directory $NFS_WORKSPACE/obj/$ARCH/$CONFIG when determining how to behave. If this directory does not exist, Jenkins will assume this is a precompiled kernel and read the kernel release name from $NFS_WORKSPACE/config/kernel/$CONFIG. If the directory does exist, the include/config/kernel.release file will be used for the kernel release name.
| + | Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value. |
| | | | |
| - | == Adding a new virtual machine client ==
| + | Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled). |
| - | A virtual machine client will be booted before use and halted when it is no longer being used. Create the VM in virt-manager and start the "Admin - Add Client" job to configure passwordless SSH to the VM and install various tests (such as connectathon). See the section "Machine setup" below for help creating new VMs. Input values for the following fields:
| + | |
| | | | |
| - | ;NAME
| + | SSH keys are generated by running |
| - | :This should be set to the name of the virtual machine as configured in virt-manager.
| + | |
| | | | |
| - | ;MOUNTPOINT
| + | <pre>$ $NFS_WORKSPACE/nfs/nfs-client-init</pre> |
| - | :This is the full path on the client that the NFS server should be mounted on.
| + | |
| | | | |
| - | ;SSH_USER
| + | or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file. |
| - | :This is the username of a user that has SSH access to the client.
| + | |
| | | | |
| - | ;IS_VM
| + | This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from. |
| - | :Make sure this is set to "true" to properly configure this client as a VM.
| + | |
| | | | |
| - | Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."
| + | You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins. |
| | | | |
| - | == Adding a new real machine client ==
| + | Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server. |
| - | A real machine client should already be running before use and will never be halted by the Jenkins. Be careful when using this option if you do not have physical access to the machine in case something goes wrong. When the machine is running, start the "Admin - Add Client" job to configure passwordless SSH to the machine and install various tests (such as connectathon). See the section "Machine setup" below for help configuring a new machine. Input values for the following fields:
| + | |
| | | | |
| - | ;NAME
| |
| - | :This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs.
| |
| | | | |
| - | ;MOUNTPOINT
| + | == Sample Machines == |
| - | :This is the full path on the client that the NFS server should be mounted on.
| + | Preconfigured Archlinux virtual machines can be downloaded from here: |
| | | | |
| - | ;SSH_USER
| + | * [http://www.linux-nfs.org/~bjschuma/Jenkins32.img 32-bit sample machine] |
| - | :This is the username of a user that has SSH access to the client. | + | * [http://www.linux-nfs.org/~bjschuma/Jenkins64.img 64-bit sample machine] |
| | | | |
| - | ;IS_VM
| + | These machines come preconfigured with a valid user. Log in with: |
| - | :Make sure this is set to "false" to properly configure this client as a real machine.
| + | |
| | | | |
| - | ;IP
| + | <pre> |
| - | :This should be set to the IP address or domain name of the machine so that it can be accessed through SSH. | + | username: jenkins |
| | + | password: jenkins |
| | + | </pre> |
| | | | |
| - | Once configured, edit the "Install" and "Run Tests" jobs and add the new client to the CLIENT variable of the "Test Matrix."
| + | If you find you need root access: |
| | | | |
| - | == Adding a virtual machine server ==
| + | <pre> |
| - | Set up a virtual machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:
| + | username: root |
| | + | password: jenkins |
| | + | </pre> |
| | | | |
| - | ;NAME
| + | Be sure to change the root password before hosting virtual machines publicly! |
| - | :This should be set to the name of the virtual machine as configured in virt-manager.
| + | See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide. |
| | | | |
| - | ;V2_MOUNT
| + | Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients. |
| - | :This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run.
| + | |
| | | | |
| - | ;V3_MOUNT
| |
| - | :This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run.
| |
| | | | |
| - | ;V4_MOUNT
| + | == Bonus scripts == |
| - | :This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run.
| + | All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework. Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion). Here is a partial list of commands: |
| | | | |
| - | ;V41_MOUNT | + | ;nfs-backup |
| - | :This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run. | + | :Creates a copy of important files from the current directory. The backup will be in ~/.nfs/backups/[directory name hash]. "Important files" can be configured globally by editing ~/.nfs/backup or locally by editing ./.backup. The default ~/.nfs/backup file will create copies of .nfsrc, .config, and *.patch from the current directory. |
| | | | |
| - | ;TEST_DIR
| + | :Backups don't stack. Only the most recent backup can be restored. |
| - | :This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem. | + | |
| | | | |
| - | ;SSH_USER | + | ;nfs-boot |
| - | :This should be set to the username of a user that has SSH access to the server. This is a mandatory field needed to halt the virtual machine when it is no longer being used. | + | :Boots a machine. |
| | | | |
| - | ;FAULT_INJECTION | + | ;nfs-client-init |
| - | :This should be set to "none" if the server does not support fault injection, otherwise choose the appropriate server type from the drop-down list. | + | :Configures a client for use in other scripts. |
| | | | |
| - | ;IS_VM | + | ;nfs-client-show |
| - | :This should be set to "true" to properly configure this server as a virtual machine. | + | :Displays useful information about a client, including config file location, mountpoint, ip address, supported NFS versions and possible kernels. |
| | | | |
| - | Once configured, edit the "Run Tests" job and add the new server to the SERVER variable of the "Test Matrix."
| + | ;nfs-config |
| | + | :Get or set values in the current .config file. |
| | | | |
| - | == Adding a real machine server ==
| + | ;nfs-halt |
| - | Set up a real machine as a NFS server and then run the "Admin - Add Server" job to create a server config file. Input values for the following fields:
| + | :Halt a machine. |
| | | | |
| - | ;NAME | + | ;nfs-install |
| - | :This is simply an identifier to help keep track of what the machine is and to make it easier to reference in various jobs. | + | :Install a kernel on a machine. |
| | | | |
| - | ;V2_MOUNT | + | ;nfs-make |
| - | :This should be set to the path exported by the NFS v2 server. Leave this blank if tests over NFS v2 should not be run. | + | :Build a kernel. |
| | | | |
| - | ;V3_MOUNT | + | ;nfs-mount |
| - | :This should be set to the path exported by the NFS v3 server. Leave this blank if tests over NFS v3 should not be run. | + | :SSH into a client and mount a server over a given NFS version. |
| | | | |
| - | ;V4_MOUNT | + | ;nfs-reboot |
| - | :This should be set to the path exported by the NFS v4 server. Leave this blank if tests over NFS v4 should not be run. | + | :Reboot a machine. |
| | | | |
| - | ;V41_MOUNT | + | ;nfs-restore |
| - | :This should be set to the path exported by the NFS v4.1 server. Leave this blank if tests over NFS v4.1 should not be run. | + | :The opposite of nfs-backup. Restores files from the backup directory to the current directory. |
| | | | |
| - | ;TEST_DIR | + | ;nfs-run |
| - | :This should be set to a directory where test output should be placed. If left blank, test output directories will be created in the root of the exported filesystem. | + | :This script is like a "mini-jenkins" for running basic tests during development. Create a script with your tests called by the function "nfs_do_work" and update your .nfsrc file to set NFS_RUN_SCRIPT=[your script here]. `nfs run` will then compile and install a kernel onto your list of configured machines, reboot into these kernels, and then run your tests. |
| | | | |
| - | ;SSH_USER | + | :;MACHINES |
| - | :This should be set to the username of a user that has SSH access to the server. This field is optional and will only be used if the server is configured to run fault injection tests. | + | ::Space separated list of machine names to use. |
| | | | |
| - | ;FAULT_INJECTION | + | :;NFS_VERSION |
| - | :This should be set to "none" if the server does not support fault injection, otherwise choose the appropriate server type from the drop-down list. | + | ::NFS version to test over. |
| | | | |
| - | ;IS_VM | + | :;NFS_SERVER |
| - | :This should be set to "false" to properly configure this server as a real machine. | + | ::Name of the server to test against. |
| | | | |
| - | ;IP | + | :;TEST_ITERATIONS |
| - | :This should be set to the IP address or domain name of the machine so it can be mounted by clients and accessed through SSH. | + | ::Number of times to mount, nfs_do_work, unmount. |
| | | | |
| - | Once configured, edit the "Run Tests" job and add the new server to the SERVER ariable of the "Test Matrix."
| + | :;TEST_SLEEP |
| | + | ::Seconds to sleep between tests. |
| | | | |
| - | == Adding a new test ==
| + | :;TEST_GRACE_WAIT |
| - | First determine if your test can be executed as a simple command over ssh. If not, you should write a new script to run your test. Next run the "Admin - Add Test" job to create a new test file. Input values for the following fields:
| + | ::Set to "true" (defaults to "true") if you want to wait for the grace period to end once the server has been mounted. |
| | | | |
| - | ;NAME | + | :;REBOOT_AFTER_TEST |
| - | :This is simply an identifier to help keep track of what the test will do and to make it easier to reference in the Run Tests job. | + | ::Set to "true" (default "true") if machines should be rebooted after testing. |
| | | | |
| - | ;ENABLED | + | :;UNINSTALL_AFTER_TEST |
| - | :When a test is disabled, it will not run when triggered by the Run Tests job. Set this to "No" if you want to create the test, but do not want it to run right away. | + | ::Set to "true" (default "true") if kernels should be uninstalled from machines after testihng. |
| | | | |
| - | ;FAULT_INJECTION | + | :;NFS_POST_MOUNT |
| - | :This should be set to "Yes" if the test makes use of fault injection features. This requires a specially configured server and a fault injection driver for the server type. | + | ::Command to run after all mounts have finished. |
| | | | |
| - | ;SSH_COMMAND | + | :;NFS_POST_UMOUNT |
| - | :Set this to "Yes" if your test is a simple command that can be run over ssh. Set this to "No" to run a command or script on the machine hosting Jenkins. | + | ::Command to run after all unmounts have finished. |
| | | | |
| - | ;COMMAND | + | :;NFS_EXIT |
| - | :Set this to the command that should be run as the main part of the test. | + | ::Command to run when `nfs run` exits. |
| | | | |
| - | The following variables are available for use when writing your COMMAND:
| + | ;nfs-server-init |
| - | | + | :Initializes a server. Similar to `nfs-client-init` |
| - | :;$CLIENT
| + | |
| - | ::Virtual machine to run test on
| + | |
| - | :;$SERVER
| + | |
| - | ::Server being tested against
| + | |
| - | :;$CONFIG
| + | |
| - | ::Config file used to compile the kernel to be tested
| + | |
| - | :;$NFS
| + | |
| - | ::The NFS version to use
| + | |
| - | :;$MNT_POINT
| + | |
| - | ::Location on the client that the server has been mounted to
| + | |
| - | :;$TEST_DIR
| + | |
| - | ::Directory test output should be written to
| + | |
| - | :;$WS
| + | |
| - | ::Full path to the Jenkins workspace once the server has been mounted
| + | |
| - | :;$TEST
| + | |
| - | ::The name of the test to run
| + | |
| - | | + | |
| - | Additionally, the bash function `test_scripts_dir` can be used as a shortcut to $NFS_WORKSPACE/scripts if you do not want the $NFS_WORKSPACE variable to be expanded. Some useful helper functions have been defined in the files in $NFS_WORKSPACE/scripts/functions/. You can source these files and use the helper functions in your commands.
| + | |
| - | | + | |
| - | After submitting your test, the generated file will be displayed in the Jenkins console output. If there are any errors, see the section below for editing tests.
| + | |
| - | | + | |
| - | Before you can use the test, you must edit the "Run Tests" job and add the new test into the "TEST" field.
| + | |
| - | | + | |
| - | == Editing a test ==
| + | |
| - | Tests are configured through files placed in $NFS_WORKSPACE/config/tests/. You can modify a test by editing the appropriate file in that directory. The following fields can be modified:
| + | |
| - | | + | |
| - | ;ENABLED
| + | |
| - | :This variable determines if the test should be run or not. Change from "true" to "false" to disable a test.
| + | |
| - | | + | |
| - | ;FAULT_INJECTION
| + | |
| - | :This variable is used during automatic filtering of fault injection tests. Change to "true" if your test makes use of the NFSD fault injection framework
| + | |
| - | | + | |
| - | ;do_test()
| + | |
| - | :This function is called to run the test. The command in this function is set based on the COMMAND parameter passed to the "Admin - Add Test" job.
| + | |
| - | | + | |
| - | == Test Groups ==
| + | |
| - | If you have a lot of tests to run, you may want to consider breaking them into multiple projects featuring a smaller number of tests. This will create more jobs to manage, but you will more easily be able to tell if different "types" of tests pass or fail.
| + | |
| - | | + | |
| - | To begin with, clone the default "Run Tests" job that has already been configured. From the main Jenkins page:
| + | |
| - | | + | |
| - | # Follow the "New Job" link at the top left-hand corner of the page
| + | |
| - | # Fill out the "Job name" field on the new job configuration page
| + | |
| - | # Select "Copy existing Job" at the end of the list
| + | |
| - | # Choose to copy from "Run Tests"
| + | |
| - | # Edit this job so it does what you want (see the previous sections beginning with "Adding a new architecture")
| + | |
| - | | + | |
| - | At this point, the new job should be created, but it will not run automatically after the "Install" job has completed. If you want the new tests scheduled after the "Install" job:
| + | |
| - | | + | |
| - | # Open up the configuration page for the "Install" job
| + | |
| - | # Scroll to the bottom of the page
| + | |
| - | # Find the "Trigger parameterized build on other projects" line and edit the "Projects to build" field and add your new test job to the comma separated list of projects.
| + | |
| - | | + | |
| - | == Fault Injection Tests ==
| + | |
| - | Jenkins-nfs comes with pre-configured fault injection test scripts, but they must be added to the "Run Tests" job before you can use them. The following tests are available:
| + | |
| - | | + | |
| - | ;forget_clients
| + | |
| - | :This will test client recovery after the server has forgotten all associated client state.
| + | |
| - | | + | |
| - | ;forget_delegations
| + | |
| - | :This will test client recovery after the server has forgotten all associated delegations.
| + | |
| - | | + | |
| - | ;recall_delegations
| + | |
| - | :This will test how a client handles the DELEGRETURN call
| + | |
| - | | + | |
| - | ;reboot_delegations
| + | |
| - | :This will test how a client recovers after the server reboots while the client holds a delegation.
| + | |
| - | | + | |
| - | ;forget_locks
| + | |
| - | :This will test client recovery after the server has forgotten all associated locks.
| + | |
| - | | + | |
| - | ;reboot_locks
| + | |
| - | :This will test how a client recovers after the server reboots while the client holds a lock.
| + | |
| - | | + | |
| - | ;forget_openowners
| + | |
| - | :This will test client recovery after the server has forgotten all associated open file state.
| + | |
| - | | + | |
| - | ;reboot_openowners
| + | |
| - | :This will test how a client recovers after the server reboots while the client holds an open file.
| + | |
| - | | + | |
| - | == Filtering Out Combinations ==
| + | |
| - | The "Install" and "Run Tests" jobs are configured as Jenkins multi-configuration projects. This means that every combination of possible variable values will be run unless they are filtered out. The files in $NFS_WORKSPACE/config/filters/ can be used to blacklist any combination that doesn't make sense for your setup. These files are simple shell scripts that hould exit 0 if the combination can be tested.
| + | |
| - | | + | |
| - | The following invalid combinations will automatically be detected and filtered out by Jenkins:
| + | |
| - | | + | |
| - | # Requested client doesn't exist
| + | |
| - | # Requested server doesn't exist
| + | |
| - | # The server has not been configured to run the requested version of NFS.
| + | |
| - | # The server is not a fault injection server
| + | |
| - | # The test does not exist
| + | |
| - | # Fault injection test over NFS v2 or v3
| + | |
| - | | + | |
| - | See the files in $NFS_WORKSPACE/config/filters/ for more information when writing your own filters. Jenkins will re-run filtering whenever a job is saved, so you can force a refilter by opening the configuration dialog and choosing "save" at the bottom of the page. Logs will be created in $NFS_WORKSPACE/filters/ to explain why a job has been filtered out.
| + | |
| - | | + | |
| - | == Machine setup ==
| + | |
| - | There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.
| + | |
| - | | + | |
| - | # Must have NFS client running
| + | |
| - | # Must have prerequisites for running the connectathon tests
| + | |
| - | # Passwordless sudo for SSH_USER user
| + | |
| - | # Passwordless ssh enabled in the sshd config
| + | |
| - | # (optional) Have prerequisites for running the xfs tests
| + | |
| - | # (optional) Have tshark installed and configured for the SSH_USER user
| + | |
| - | | + | |
| - | Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.
| + | |
| - | | + | |
| - | Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).
| + | |
| - | | + | |
| - | Passwordless ssh can be configured by running
| + | |
| - | | + | |
| - | <pre>$ $NFS_WORKSPACE/scripts/init_client</pre>
| + | |
| - | | + | |
| - | or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
| + | |
| - | | + | |
| - | This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.
| + | |
| - | | + | |
| - | You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
| + | |
| - | | + | |
| - | The install step will mount the server through NFS v4, so at the very least NFS v4 needs to be working on the client and server. Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.
| + | |
| - | | + | |
| - | If you intend to collect packet traces, you will need to install the tshark program (it may be packaged with wireshark). Once installed, add your ssh user to the wireshark group so packets can be safely captured
| + | |
| - | | + | |
| - | == Sample Machines ==
| + | |
| - | Preconfigured Archlinux virtual machines can be downloaded from here:
| + | |
| - | | + | |
| - | * [http://www.linux-nfs.org/~bjschuma/Jenkins32.img 32-bit sample machine]
| + | |
| - | * [http://www.linux-nfs.org/~bjschuma/Jenkins64.img 64-bit sample machine]
| + | |
| - | | + | |
| - | These machines come preconfigured with a valid user. Log in with:
| + | |
| - | | + | |
| - | <pre>
| + | |
| - | username: jenkins
| + | |
| - | password: jenkins
| + | |
| - | </pre>
| + | |
| - | | + | |
| - | If you find you need root access:
| + | |
| - | | + | |
| - | <pre>
| + | |
| - | username: root
| + | |
| - | password: jenkins
| + | |
| - | </pre>
| + | |
| - | | + | |
| - | Be sure to change the root password before hosting virtual machines publicly!
| + | |
| - | See the Archlinux Beginners' Guide for help admining the sample machines: https://wiki.archlinux.org/index.php/Beginners%27_Guide.
| + | |
| - | | + | |
| - | Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.
| + | |
| | | | |
| - | == The Job Pool ==
| + | ;nfs-server-show |
| - | Jenkins-NFS works best when configured with a large job pool, since a larger pool will increase the chances of a test running on an already-booted kernel. Most of the testing during Jenkins-NFS development was performed with a job pool of size 8. To increase the job pool size:
| + | :Shows useful information about a server. Similar to `nfs-client-show` |
| | | | |
| - | # From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
| + | ;nfs-show |
| - | # Follow the "Configure System" link
| + | :Shows the resulting NFS environment after first reading the global ~/.nfs/nfsrc file and then the local .nfsrc file. |
| - | # Find the "# of executors" field and change to your desired value
| + | |
| | | | |
| - | == Updating Jenkins ==
| + | ;nfs-ssh |
| - | The Jenkins development team realeases a new jenkins.war file every week or two. The updated war can be downloaded and installed automatically from your currenly running version. To update Jenkins:
| + | :SSH into a machine |
| | | | |
| - | # From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column
| + | ;nfs-test |
| - | # At the top of the page, there may be a message saying that a new version is available. If you do not see this message, then you are already running the latest version of Jenkins
| + | :Run a test on a client against a server over a specific NFS version |
| - | # If you do see the new version message, click the "Or Upgrade Automatically" button to begin the upgrade process
| + | |
| - | # (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the new version has been installed. If you don't do this, you must manually stop and start Jenkins to switch to the new version
| + | |
| | | | |
| - | == Installing and Upgrading Plugins ==
| + | ;nfs-umount |
| - | Plugins can add new features that are not included with the base jenkins.war program. To locate and install a new plugin:
| + | :Unmounts a client's NFS mountpoint |
| | | | |
| - | # From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
| + | ;nfs-uninstall |
| - | # Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
| + | :Removes a kernel from a client |
| - | # Click the "Available" tab
| + | |
| - | # Browse the list of available plugins, selecting the ones you wish to install
| + | |
| - | # Click the "Install" button at the bottom of the page
| + | |
| - | # (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been installed. If you don't do this, you must manually stop and start Jenkins to enable your new plugins. Note that recent Jenkins releases support installing plugins without requiring Jenkins to restart.
| + | |
| | | | |
| - | Developers will occasionally release a new version of their plugin. To update your installed plugins:
| + | These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory. The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers). |
| | | | |
| - | # From the main Jenkins page, follow the "Manage Jenkins" link on the left-hand column.
| + | Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it. Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented. Commands must be run with correct positional arguments. This will hopefully change in the future. |
| - | # Find and follow the "Manage Plugins" link on the "Manage Jenkins" page
| + | |
| - | # Select the plugins you wish to update
| + | |
| - | # Click the "Install" button at the bottom of the page
| + | |
| - | # (Optional) Click the "Restart Jenkins when installation is complete and no jobs are running" box to reload Jenkins once the plugins have been updated. If you don't do this, you must manually stop and start Jenkins to use the updated plugins. Note that recent Jenkins releases support updating plugins without requiring Jenkins to restart.
| + | |
I'm going to skip over basic Jenkins usage to keep this document short. You can look at http://www.jenkins-ci.org if you need help using the Jenkins java program.
The cloned directory should be placed in a location exported by the NFS server running on the host machine. The full path up to and including this directory is referred to as $NFS_WORKSPACE throughout this document and various scripts.
Once you have the latest version of the scripts, cd into $NFS_WORKSPACE and run:
to download the Jenkins executable, required plugins, and to configure the Jenkins environment. This includes creating a symlink in your home directory ~/.jenkins -> $NFS_WORKSPACE/jenkins. When you are ready, run
to start Jenkins and point your web browser to localhost:8080 to see the web gui.
Use the "Admin - Add Client" and "Admin - Add Server" jobs on the "Admin" tab to set up machine profiles for testing. Edit the "Run Clients" job to set your list of clients as possible values for the NFS_CLIENT field. Edit the "Run Tests" job to set your list of servers as possible values for the NFS_SERVER field.
If you want to test the latest linux-next code from Trond's tree, run the "Linux NFS" job to check out and run the code. You may want to change the default value for the SRC_DIR parameter so you don't have to enter the path all the time (and also because the job is currently configured for Bryan's setup...).
You can also run the "Test Controller" job on the "Common" tab to test a kernel without updating it from git first.
This is the basic configuration file used by the scripts in $NFS_WORKSPACE/nfs/ to set up a default environment. Any changes here will not require you to restart Jenkins and should take effect almost immediately.
Jenkins is configured to compile kernels x86_64 by default. To add other architectures, edit the "Compile" job and add a new value (such as i386) to the values list of the NFS_ARCH axis.
Copy the config file to $NFS_WORKSPACE/config/kernel using a unique name. Then add the file to the NFS_CONFIG values list in the "Test Controller" job.
There are a few requirements for a machine to work correctly as a client in the NFS Jenkins setup.
Grub must be configured to allow changing the default kernels. This can be done by editing /boot/grub/menu.lst and changing the value "default=<some number>" to "default=saved". You may also need to create the file /boot/grub/default to store the new default value.
Using SELinux can slow down the testing step because the harddrive may occasionally need to be relabeled when booting multiple kernels. If your tests do not depend on SELinux, you may consider disabling it by editing the file /etc/selinux/config and setting the variable SELINUX to the value "disabled" (SELINUX=disabled).
or by manually starting the job "Admin - Add Client". This script will check if ~/.ssh/id_rsa.pub (your public key) exists. If it doesn't, then ssh-keygen will be used to generate a key. The public key will then be copied over to the virtual machine and appended to the ~/.ssh/authorized_keys file.
This process uses ssh-copy-id to transfer your public key to the test machine, and you will be prompted for your password in the same shell that Jenkins was initially started from.
You will also need to set the environment variable MNTOPTIONS on each virtual machine so that the connectathon tests will work properly. To do this, SSH needs to allow user environments to be set up. This can be done by editing /etc/ssh/sshd_config and setting PermitUserEnvironment to "yes". When mounting through the scripts in $NFS_WORKSPACE/scripts/, $MNTOPTIONS will be stored in /home/$NFS_SSH_USER/.ssh/environment to be sourced on future logins.
Note that the default userid on Fedora is 500, while other distros use 1000. This may have an effect on your ability to access files on the NFS server.
These machines come preconfigured with a valid user. Log in with:
Archlinux, Fedora and Red Hat machines are known to work properly as NFS clients.
All of the scripts in $NFS_WORKSPACE/nfs can be run by hand for simple testing outside of the Jenkins framework. Make sure that $NFS_WORKSPACE/nfs is in your shell's $PATH (and possibly source $NFS_WORKSPACE/nfs/nfs-completion.bash for command completion). Here is a partial list of commands:
These scripts store their data in ~/.nfs/ when running by hand to avoid collisions with Jenkins config directory. The downside is that clients and servers must be reinitialized (or just copy the config file) from the Jenkins directory ($NFS_WORKSPACE/config/clients or $NFS_WORKSPACE/config/servers).
Similar to git, you can create any script prefixed with "nfs-" and the generic `nfs` command will be able to run (and autocomplete) it. Unfortunately, help text isn't the best and getopt-style arguments have yet to be implemented. Commands must be run with correct positional arguments. This will hopefully change in the future.
