Fedora pNFS Client Setup
From Linux NFS
Contents |
Installing Fedora
The Install Guide at http://docs.fedoraproject.org/ describes numerous ways in which to install Fedora. Choose the best method for you.
- Select local disk and take default disk partitions for /boot, /swap LVM etc
- If your block device is accessed through iSCSI then click on “Advanced disk configuration” tab and select iSCSI. If you are using FC you are recommended to unplug Fiber cable before installation and connect back before reboot. Do not initialize Block devices if you are using EMC Unified storage devices (Celerra NAS)
- In order to run the Connectathon test suite you will need the “software development” package.
- Click on the "Software Development" button in the package install screen similar to this
- Its also advisable to add in the 'Fedora' repository on the same page.
- Finally it's also a good idea to do a 'yum -y update' first thing after the install completes. This will ensure you have the most up to date bits available.
Installing pNFS Enabled Fedora kernel
There are two way to install the pNFS enabled kernels. Either using a yum repository or directly downloading.
Yum repository
For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.34) kernels use the http://steved.fedorapeople.org/pnfs.repo
For Fedora development kernels (kernel-2.6.32) repository http://steved.fedorapeople.org/pnfs-rawhide.repo
Download the appropriate repository into /etc/yum.repos.d directory:
cd /etc/yum.repos.d; wget http://steved.fedorapeople.org/pnfs.repo
Then use one of the following commands to install the kernel of choice (including the dependencies). To install non-debug kernels:
yum --disablerepo='*' --enablerepo=pnfs install kernel\*
To install debug enabled kernels:
yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
To install debuginfo rpms , which aid with debugging:
yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'
Direct download
You can directly download the pNFS kernel rpms from http://steved.fedorapeople.org/repos/pnfs. With the direct downloads you will need periodicity check for updates as well as figure the dependencies. If you use this method, and do not add the pnfs repo to yum, then a subsequent "yum update" may erase your pnfs changes and revert your system to pre-pnfs. It may even render your system unusable.
Kernel source
As of linux kernel 3.1, the block layout driver is included in the standard kernel. The latest pnfs source code, including bug fixes and experimental code that has not yet been merged upstream, is in git://linux-nfs.org/~bhalevy/linux-pnfs.git on the pnfs-all-latest branch.
You can find the latest block layout kernel source in git://citi.umich.edu/projects/linux-pnfs-blk.git. This is for developers only.
Target and blkmapd setup for block layout client
If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.
You need the pnfs version of nfs-utils. If you have already added the pnfs yum repo, you can get this just by doing a "yum update". You can also install an rpm from http://steved.fedorapeople.org/repos/pnfs or build it from the git source tree at git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git.
For the very latest block layout client, see the rpms at http://www.citi.umich.edu/projects/nfsv4/pnfs/block/ .
You will need to set up iscsi targets on your server, and any logins or permissions required by local policy. The procedure depends on the server.
You should lower the hung task timeout to aid in debugging:
sysctl -w kernel.hung_task_timeout_secs=10
Now connect to your iscsi targets, similar to:
iscsiadm -m discovery -t sendtargets -p <iscsi-server> -l
And start the block layout service, which will load kernel modules and start blkmapd:
service blkmapd restart
If you get an error message "blkmapd: unrecognized service" you may be missing the init file. Install it from the CITI pnfs web site:
wget -O /etc/rc.d/init.d/blkmapd http://www.citi.umich.edu/projects/nfsv4/pnfs/block/download/rh-init.txt chmod +x /etc/rc.d/init.d/blkmapd
Mount Filesystem
Use the '-o minorversion=1' mount option when mounting the server, similar to:
mount -t nfs4 -o minorversion=1 <server>:/export /mnt
Generate Traffic
Generate some I/O using “dd” or run “Connectathon ”. You may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.
To verify pNFS is up and working, grep for the word 'LAYOUT' in /proc/self/mountstat. You should see some non-zero values.
fedora# grep LAYOUT /proc/self/mountstats
nfsv4: bm0=0xfcff8fff,bm1=0x40f9bfff,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
PNFS_LAYOUTGET: 2561 2561 0 655616 256284 34 1698 2575
PNFS_LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
PNFS_LAYOUTRETURN: 1 1 0 252 88 0 0 1
Unmount and disconnect
umount /mnt iscsiadm -m node -U all
Troubleshooting
If this doesn't work for you, follow these steps to find the problem. First make sure your iscsi targets have been attached. For each target device you should see something like this in /var/log/messages:
Mar 7 09:46:34 rhcl1 kernel: scsi 7:0:0:15: Direct-Access DGC RAID 5 0326 PQ: 0 ANSI: 4 Mar 7 09:46:34 rhcl1 kernel: sd 7:0:0:15: Attached scsi generic sg32 type 0 Mar 7 09:46:34 rhcl1 kernel: sd 7:0:0:15: [sdq] 1125628928 512-byte logical blocks: (576 GB/536 GiB) Mar 7 09:46:36 rhcl1 kernel: sd 7:0:0:15: [sdq] Write Protect is off Mar 7 09:46:37 rhcl1 kernel: sd 7:0:0:15: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Mar 7 09:46:37 rhcl1 kernel: sdq: unknown partition table Mar 7 09:46:38 rhcl1 kernel: sd 7:0:0:15: [sdq] Attached SCSI disk
You should also see the block devices in the /sys/block directory:
pdsi7# ls /sys/block loop0 loop6 ram11 ram3 ram9 sdae sdak sdaq sdb sdh sdn sdt sdz loop1 loop7 ram12 ram4 sda sdaf sdal sdar sdc sdi sdo sdu loop2 md127 ram13 ram5 sdaa sdag sdam sdas sdd sdj sdp sdv loop3 ram0 ram14 ram6 sdab sdah sdan sdat sde sdk sdq sdw loop4 ram1 ram15 ram7 sdac sdai sdao sdau sdf sdl sdr sdx loop5 ram10 ram2 ram8 sdad sdaj sdap sdav sdg sdm sds sdy
Next load the kernel module:
modprobe blocklayoutdriver pdsi7# modprobe blocklayoutdriver pdsi7#
Now run the daemon in foreground mode:
pdsi7# /usr/sbin/blkmapd -f
Finally, run your mount command and verify that you have a pnfs mount (see "Mount Filesystem" and "Generate Traffic" above). The daemon should print some messages as it discovers your devices:
pdsi7# /usr/sbin/blkmapd -f blkmapd: process_deviceinfo: 12 vols blkmapd: decode_blk_signature: si_comps[0]: bs_length 4, bs_string 0x14 blkmapd: decode_blk_signature: si_comps[1]: bs_length 32, bs_string APM000644032240000 blkmapd: read_cmp_blk_sig: /dev/sdn sig 0x14 at -65536 blkmapd: read_cmp_blk_sig: /dev/sdn sig APM000644032240000 at -65436 blkmapd: decode_blk_volume: simple 0 ... blkmapd: decode_blk_volume: stripe 10 nvols=10 unit=512 blkmapd: decode_blk_volume: concat 11 1 blkmapd: dm_device_create: 10 pnfs_vol_0 253:0 blkmapd: dm_device_create: 11 pnfs_vol_1 253:1
These messages will vary depending on your device topology. If you still don't have a pnfs mount, the daemon may have printed something useful, or you may find something in /var/log/messages.
