Fedora pNFS Client Setup

From Linux NFS

Latest revision as of 12:14, 15 May 2012

Installing Fedora

The Install Guide at http://docs.fedoraproject.org/ describes numerous ways in which to install Fedora. Choose the best method for you.

• Select local disk and take default disk partitions for /boot, /swap LVM etc
• If your block device is accessed through iSCSI then click on “Advanced disk configuration” tab and select iSCSI. If you are using FC you are recommended to unplug Fiber cable before installation and connect back before reboot. Do not initialize Block devices if you are using EMC Unified storage devices (Celerra NAS)
• In order to run the Connectathon test suite you will need the “software development” package.
  • Click on the "Software Development" button in the package install screen similar to this
  • Its also advisable to add in the 'Fedora' repository on the same page.
  • Finally it's also a good idea to do a 'yum -y update' first thing after the install completes. This will ensure you have the most up to date bits available.

Installing pNFS Enabled Fedora kernel

As of kernel 3.1 the block layout client is included in the standard kernel and you can skip this section.

There are two way to install the pNFS enabled kernels. Either using a yum repository or directly downloading.

Yum repository

For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.34) kernels use the http://steved.fedorapeople.org/pnfs.repo

For Fedora development kernels (kernel-2.6.32) repository http://steved.fedorapeople.org/pnfs-rawhide.repo

Download the appropriate repository into /etc/yum.repos.d directory:

cd /etc/yum.repos.d; wget http://steved.fedorapeople.org/pnfs.repo

Then use one of the following commands to install the kernel of choice (including the dependencies). To install non-debug kernels:

yum --disablerepo='*' --enablerepo=pnfs install kernel\*

To install debug enabled kernels:

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

To install debuginfo rpms , which aid with debugging:

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'

Direct download

You can directly download the pNFS kernel rpms from http://steved.fedorapeople.org/repos/pnfs. With the direct downloads you will need periodicity check for updates as well as figure the dependencies. If you use this method, and do not add the pnfs repo to yum, then a subsequent "yum update" may erase your pnfs changes and revert your system to pre-pnfs. It may even render your system unusable.

Kernel source

As of linux kernel 3.1, the block layout driver is included in the standard kernel. The latest pnfs source code, including bug fixes and experimental code that has not yet been merged upstream, is in git://linux-nfs.org/~bhalevy/linux-pnfs.git on the pnfs-all-latest branch.

You can find the latest block layout kernel source in git://citi.umich.edu/projects/linux-pnfs-blk.git. This is for developers only.

Target and blkmapd setup for block layout client

If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.

You need the pnfs version of nfs-utils. If you have already added the pnfs yum repo, you can get this just by doing a "yum update". You can also install an rpm from http://steved.fedorapeople.org/repos/pnfs or build it from the git source tree at git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git.

As of version 1.2.5, the standard nfs-utils package includes support for pNFS, including the block layout client, but you should apply this patch to prevent your logs from filling with garbage: 0001-remove-pretty_sig.patch

You will need to set up iscsi targets on your server, and any logins or permissions required by local policy. The procedure depends on the server.

You should lower the hung task timeout to aid in debugging:

sysctl -w kernel.hung_task_timeout_secs=10

Now connect to your iscsi targets, similar to:

iscsiadm -m discovery -t sendtargets -p <iscsi-server> -l

And start the block layout service, which will load kernel modules and start blkmapd:

service blkmapd restart

If you get an error message "blkmapd: unrecognized service" you may be missing the init file. Install it from the CITI pnfs web site:

wget -O /etc/rc.d/init.d/blkmapd http://www.citi.umich.edu/projects/nfsv4/pnfs/block/download/rh-init.txt
chmod +x /etc/rc.d/init.d/blkmapd

Mount Filesystem

Use the '-o minorversion=1' mount option when mounting the server, similar to:

mount -t nfs4 -o minorversion=1 <server>:/export  /mnt

Generate Traffic

Generate some I/O using “dd” or run “Connectathon ”. You may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.

To verify pNFS is up and working, grep for the word 'LAYOUT' in /proc/self/mountstat. You should see some non-zero values.

fedora# grep LAYOUT /proc/self/mountstats
        nfsv4:  bm0=0xfcff8fff,bm1=0x40f9bfff,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
        PNFS_LAYOUTGET: 2561 2561 0 655616 256284 34 1698 2575
        PNFS_LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
        PNFS_LAYOUTRETURN: 1 1 0 252 88 0 0 1

Unmount and disconnect

umount /mnt
iscsiadm -m node -U all

Troubleshooting

If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.

If this doesn't work for you, follow these steps to find the problem. First make sure your iscsi targets have been attached. For each target device you should see something like this in /var/log/messages:

Mar  7 09:46:34 rhcl1 kernel: scsi 7:0:0:15: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: Attached scsi generic sg32 type 0
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: [sdq] 1125628928 512-byte logical blocks: (576 GB/536 GiB)
Mar  7 09:46:36 rhcl1 kernel: sd 7:0:0:15: [sdq] Write Protect is off
Mar  7 09:46:37 rhcl1 kernel: sd 7:0:0:15: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar  7 09:46:37 rhcl1 kernel: sdq: unknown partition table
Mar  7 09:46:38 rhcl1 kernel: sd 7:0:0:15: [sdq] Attached SCSI disk

You should also see the block devices in the /sys/block directory:

pdsi7# ls /sys/block
loop0  loop6  ram11  ram3  ram9  sdae  sdak  sdaq  sdb  sdh  sdn  sdt  sdz
loop1  loop7  ram12  ram4  sda   sdaf  sdal  sdar  sdc  sdi  sdo  sdu
loop2  md127  ram13  ram5  sdaa  sdag  sdam  sdas  sdd  sdj  sdp  sdv
loop3  ram0   ram14  ram6  sdab  sdah  sdan  sdat  sde  sdk  sdq  sdw
loop4  ram1   ram15  ram7  sdac  sdai  sdao  sdau  sdf  sdl  sdr  sdx
loop5  ram10  ram2   ram8  sdad  sdaj  sdap  sdav  sdg  sdm  sds  sdy

Next load the kernel module:

modprobe blocklayoutdriver
pdsi7# modprobe blocklayoutdriver
pdsi7#

Now run the daemon in foreground mode:

pdsi7# /usr/sbin/blkmapd -f

Finally, run your mount command and verify that you have a pnfs mount (see "Mount Filesystem" and "Generate Traffic" above). The daemon should print some messages as it discovers your devices:

pdsi7# /usr/sbin/blkmapd -f
blkmapd: process_deviceinfo: 12 vols
blkmapd: decode_blk_signature: si_comps[0]: bs_length 4, bs_string 0x14
blkmapd: decode_blk_signature: si_comps[1]: bs_length 32, bs_string APM000644032240000
blkmapd: read_cmp_blk_sig: /dev/sdn sig 0x14 at -65536
blkmapd: read_cmp_blk_sig: /dev/sdn sig APM000644032240000 at -65436
blkmapd: decode_blk_volume: simple 0
...
blkmapd: decode_blk_volume: stripe 10 nvols=10 unit=512
blkmapd: decode_blk_volume: concat 11 1
blkmapd: dm_device_create: 10 pnfs_vol_0 253:0
blkmapd: dm_device_create: 11 pnfs_vol_1 253:1

These messages will vary depending on your device topology. If you still don't have a pnfs mount, the daemon may have printed something useful, or you may find something in /var/log/messages.