Fedora pNFS Client Setup

From Linux NFS

Revision as of 13:51, 27 September 2010

Select Hardware

For best performance, select hardware that is capable of running 64-bit os and at a minimum of two GigE copper NIC ports. Connect all necessary network ports to VLAN. For iSCSI target access, it is recommended that you use separate subnets for the backbone and iSCSI. For testing purposes, pNFS will work with 32-bit linux and lesser hardware, but the you will not obtain the performance benefits of pNFS.

Installing Fedora

The Install Guide at http://docs.fedoraproject.org/ describes numerous ways in which to install Fedora. Choose the best method for you.

• Select local disk and take default disk partitions for /boot, /swap LVM etc
• If your block device is accessed through iSCSI then click on “Advanced disk configuration” tab and select iSCSI. If you are using FC you are recommended to unplug Fiber cable before installation and connect back before reboot. Do not initialize Block devices if you are using EMC Unified storage devices (Celerra NAS)
• In order to run the Connectathon test suite you will need the “software development” package.
  • Click on the "Software Development" button in the package install screen similar to this
  • Its also advisable to add in the 'Fedora' repository on the same page.
  • Finally it's also a good idea to do a 'yum -y update' first thing after the install completes. This will ensure you have the most up to date bits available.

Installing pNFS Enabled Fedora kernel

There are two way to install the pNFS enabled kernels. Either using a yum repository or directly downloading.

Yum repository

For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.34) kernels use the http://steved.fedorapeople.org/pnfs.repo

For Fedora development kernels (kernel-2.6.32) repository http://steved.fedorapeople.org/pnfs-rawhide.repo

Download the appropriate repository into /etc/yum.repos.d directory. Then use one of the following commands to install the kernel of choice (including the dependencies):

To install non-debug kernels:

yum --disablerepo='*' --enablerepo=pnfs install kernel\*

To install debug enabled kernels:

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

To install debuginfo rpms , which aid with debugging:

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'

Direct download

You can directly download the pNFS kernel rpms from http://steved.fedorapeople.org/repos/pnfs. With the direct downloads you will need periodicity check for updates as well as figure the dependencies. If you use this method, and do not add the pnfs repo to yum, then a subsequent "yum update" may erase your pnfs changes and revert your system to pre-pnfs. It may even render your system unusable.

Target and blkmapd setup for block layout client

If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.

You need the pnfs version of nfs-utils. If you have already added the pnfs yum repo, you can get this just by doing a "yum update". You can also install an rpm from http://steved.fedorapeople.org/repos/pnfs or build it from the git source tree at git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git.

You will need to set up iscsi targets on your server, and any logins or permissions required by local policy. The procedure depends on the server.

You should lower the hung task timeout to aid in debugging:

sysctl -w kernel.hung_task_timeout_secs=10

Now connect to your iscsi targets, similar to:

iscsiadm -m discovery -t sendtargets -p <iscsi-server> -l

And start the block layout service, which will load kernel modules and start blkmapd:

service blkmapd restart

If you get an error message "blkmapd: unrecognized service" you may be missing the init file. Install it from the nfs-utils git repo:

wget -O /etc/rc.d/init.d/blkmapd "http://git.linux-nfs.org/?p=bhalevy/pnfs-nfs-utils.git;a=blob_plain;f=utils/blkmapd/etc/initd/initd.redhat;hb=HEAD"
chmod +x /etc/rc.d/init.d/blkmapd

Mount Filesystem

Use the '-o minorversion=1' mount option when mounting the server, similar to:

mount -t nfs4 -o minorversion=1 <server>:/export  /mnt

Generate Traffic

Generate some I/O using “dd” or run “Connectathon ”. You may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.

To verify pNFS is up and working, grep for the word 'LAYOUT' in /proc/self/mountstat. You should see some non-zero values.

fedora# grep LAYOUT /proc/self/mountstats
        nfsv4:  bm0=0xfcff8fff,bm1=0x40f9bfff,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
        PNFS_LAYOUTGET: 2561 2561 0 655616 256284 34 1698 2575
        PNFS_LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
        PNFS_LAYOUTRETURN: 1 1 0 252 88 0 0 1

Unmount and disconnect

umount /mnt
iscsiadm -m node -U all