Fedora pNFS Client Setup

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
m (→Yum repository: typos)
(→Target and blkmapd setup for block layout client: pretty sig patch)
 
(26 intermediate revisions not shown)
Line 1: Line 1:
-
==Select Hardware==
 
-
<p>
 
-
Select hardware that is capable of running 64-bit os and at a minimum of two GigE copper NIC ports. Connect all necessary network ports to VLAN. For iSCSI target access, it is recommended that you use separate subnets for the backbone and iSCSI.
 
-
</p>
 
==Installing  Fedora==
==Installing  Fedora==
<p>The Install Guide at http://docs.fedoraproject.org/ describes
<p>The Install Guide at http://docs.fedoraproject.org/ describes
-
a numerous ways in which to install Fedora. Choose the best method for you.</p>
+
numerous ways in which to install Fedora. Choose the best method for you.</p>
<ul>
<ul>
<li>Select local disk and take default disk partitions for /boot, /swap LVM etc
<li>Select local disk and take default disk partitions for /boot, /swap LVM etc
Line 13: Line 9:
<li>Click on the "Software Development" button in the package install screen
<li>Click on the "Software Development" button in the package install screen
similar to [http://i1-news.softpedia-static.com/images/extra/LINUX/large/fedora10installguide-large_014.jpg this]
similar to [http://i1-news.softpedia-static.com/images/extra/LINUX/large/fedora10installguide-large_014.jpg this]
-
<li>Its also advisable to added in the 'Fedora' repository on the same page.
+
<li>Its also advisable to add in the 'Fedora' repository on the same page.
<li>Finally it's also a good idea to do a 'yum -y update' first thing after the  
<li>Finally it's also a good idea to do a 'yum -y update' first thing after the  
install completes. This will ensure you have the most up to date bits available.
install completes. This will ensure you have the most up to date bits available.
</ul>
</ul>
</ul>
</ul>
 +
==Installing pNFS Enabled Fedora kernel==
==Installing pNFS Enabled Fedora kernel==
 +
As of kernel 3.1 the block layout client is included in the standard kernel and you can skip this section.
 +
There are two way to install the pNFS enabled kernels. Either using
There are two way to install the pNFS enabled kernels. Either using
a yum repository or directly downloading.
a yum repository or directly downloading.
===Yum repository===
===Yum repository===
-
<p>For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.33) kernels
+
<p>For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.34) kernels
use the http://steved.fedorapeople.org/pnfs.repo</p>
use the http://steved.fedorapeople.org/pnfs.repo</p>
<p>For Fedora development kernels (kernel-2.6.32) repository
<p>For Fedora development kernels (kernel-2.6.32) repository
http://steved.fedorapeople.org/pnfs-rawhide.repo</p>
http://steved.fedorapeople.org/pnfs-rawhide.repo</p>
-
Download the appropriate repository into <tt>/etc/yum.repos.d</tt> directory.
+
Download the appropriate repository into <tt>/etc/yum.repos.d</tt> directory:
 +
 
 +
cd /etc/yum.repos.d; wget http://steved.fedorapeople.org/pnfs.repo
 +
 
Then use one of the following commands to install the kernel  
Then use one of the following commands to install the kernel  
-
of choice (including the dependencies):
+
of choice (including the dependencies). To install non-debug kernels:
-
+
-
To install non-debug kernels:
+
-
    yum --disablerepo='*' --enablerepo=pnfs install kernel\*
+
yum --disablerepo='*' --enablerepo=pnfs install kernel\*
To install debug enabled kernels:
To install debug enabled kernels:
-
    yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
+
yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
To install debuginfo rpms , which aid with debugging:
To install debuginfo rpms , which aid with debugging:
-
    yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
+
yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*
Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'
Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'
===Direct download===
===Direct download===
-
 
+
You can directly download the pNFS kernel rpms from <tt>http://steved.fedorapeople.org/repos/pnfs</tt>.
-
You can directly download the pNFS kernel rpms from http://steved.fedorapeople.org/repos/pnfs.
+
With the direct downloads you will need periodicity check for updates as well as  
With the direct downloads you will need periodicity check for updates as well as  
-
figure the dependencies.  
+
figure the dependencies.
 +
<b>If you use this method, and do not add the pnfs repo to yum, then a subsequent "yum update" may erase your pnfs changes and revert your system to pre-pnfs.</b> It may even render your system unusable.
-
==Load pNFS modules==
+
===Kernel source===
-
Load the needed modules with the following commands:
+
As of linux kernel 3.1, the block layout driver is included in the standard kernel. The latest pnfs source code, including bug fixes and experimental code that has not yet been merged upstream, is in <tt>git://linux-nfs.org/~bhalevy/linux-pnfs.git</tt> on the <tt>pnfs-all-latest</tt> branch.
-
    modprobe nfslayoutdriver
+
-
    modprobe blocklayoutdrive
+
-
To verify pNFS modules are loaded correctly do the following:
+
You can find the latest block layout kernel source in <tt>git://citi.umich.edu/projects/linux-pnfs-blk.git</tt>. This is for developers only.
-
    lsmod | grep nfslayout
+
 
-
    nfslayoutdriver        18423 0
+
==Target and blkmapd setup for block layout client==
-
    nfs                    353047 3 blocklayoutdriver,nfslayoutdriver
+
If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.
 +
 
 +
You need the pnfs version of nfs-utils. If you have already added the pnfs yum repo, you can get this just by doing a "yum update". You can also install an rpm from http://steved.fedorapeople.org/repos/pnfs or build it from the git source tree at <tt>git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git</tt>.
 +
 
 +
As of version 1.2.5, the standard nfs-utils package includes support for pNFS, including the block layout client, but you should apply this patch to prevent your logs from filling with garbage: [http://jim.rees.org/computers/0001-remove-pretty_sig.patch 0001-remove-pretty_sig.patch]
 +
 
 +
You will need to set up iscsi targets on your server, and any logins or permissions required by local policy. The procedure depends on the server.
 +
 
 +
You should lower the hung task timeout to aid in debugging:
 +
  sysctl -w kernel.hung_task_timeout_secs=10
 +
 
 +
Now connect to your iscsi targets, similar to:
 +
  iscsiadm -m discovery -t sendtargets -p <iscsi-server> -l
 +
 
 +
And start the block layout service, which will load kernel modules and start blkmapd:
 +
service blkmapd restart
 +
 
 +
If you get an error message "<tt>blkmapd: unrecognized service</tt>" you may be missing the init file. Install it from the CITI pnfs web site:
 +
wget -O /etc/rc.d/init.d/blkmapd http://www.citi.umich.edu/projects/nfsv4/pnfs/block/download/rh-init.txt
 +
chmod +x /etc/rc.d/init.d/blkmapd
==Mount Filesystem==
==Mount Filesystem==
Use the '-o minorversion=1' mount option when mounting the server,
Use the '-o minorversion=1' mount option when mounting the server,
similar to:
similar to:
-
    mount -t nfs4 -o minorversion=1 <server>:/export  /mnt
+
mount -t nfs4 -o minorversion=1 <server>:/export  /mnt
-
 
+
-
To Verify pNFS is up and working, grep for the word 'LAYOUT' in /proc/self/mountstat
+
-
    grep LAYOUT /proc/self/mountstat
+
==Generate Traffic==
==Generate Traffic==
Generate some I/O using “dd”  or  run “Connectathon ”. You  may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.
Generate some I/O using “dd”  or  run “Connectathon ”. You  may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.
 +
 +
To verify pNFS is up and working, grep for the word 'LAYOUT' in <tt>/proc/self/mountstat</tt>.
 +
You should see some non-zero values.
 +
<pre>
 +
fedora# grep LAYOUT /proc/self/mountstats
 +
        nfsv4:  bm0=0xfcff8fff,bm1=0x40f9bfff,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
 +
        PNFS_LAYOUTGET: 2561 2561 0 655616 256284 34 1698 2575
 +
        PNFS_LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
 +
        PNFS_LAYOUTRETURN: 1 1 0 252 88 0 0 1
 +
</pre>
 +
 +
==Unmount and disconnect==
 +
umount /mnt
 +
iscsiadm -m node -U all
 +
 +
==Troubleshooting==
 +
If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.
 +
 +
If this doesn't work for you, follow these steps to find the problem. First make sure your iscsi targets have been attached. For each target device you should see something like this in <tt>/var/log/messages</tt>:
 +
<pre>
 +
Mar  7 09:46:34 rhcl1 kernel: scsi 7:0:0:15: Direct-Access    DGC      RAID 5          0326 PQ: 0 ANSI: 4
 +
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: Attached scsi generic sg32 type 0
 +
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: [sdq] 1125628928 512-byte logical blocks: (576 GB/536 GiB)
 +
Mar  7 09:46:36 rhcl1 kernel: sd 7:0:0:15: [sdq] Write Protect is off
 +
Mar  7 09:46:37 rhcl1 kernel: sd 7:0:0:15: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 +
Mar  7 09:46:37 rhcl1 kernel: sdq: unknown partition table
 +
Mar  7 09:46:38 rhcl1 kernel: sd 7:0:0:15: [sdq] Attached SCSI disk
 +
</pre>
 +
You should also see the block devices in the <tt>/sys/block</tt> directory:
 +
<pre>
 +
pdsi7# ls /sys/block
 +
loop0  loop6  ram11  ram3  ram9  sdae  sdak  sdaq  sdb  sdh  sdn  sdt  sdz
 +
loop1  loop7  ram12  ram4  sda  sdaf  sdal  sdar  sdc  sdi  sdo  sdu
 +
loop2  md127  ram13  ram5  sdaa  sdag  sdam  sdas  sdd  sdj  sdp  sdv
 +
loop3  ram0  ram14  ram6  sdab  sdah  sdan  sdat  sde  sdk  sdq  sdw
 +
loop4  ram1  ram15  ram7  sdac  sdai  sdao  sdau  sdf  sdl  sdr  sdx
 +
loop5  ram10  ram2  ram8  sdad  sdaj  sdap  sdav  sdg  sdm  sds  sdy
 +
</pre>
 +
 +
Next load the kernel module:
 +
<pre>
 +
modprobe blocklayoutdriver
 +
pdsi7# modprobe blocklayoutdriver
 +
pdsi7#
 +
</pre>
 +
 +
Now run the daemon in foreground mode:
 +
<pre>
 +
pdsi7# /usr/sbin/blkmapd -f
 +
</pre>
 +
 +
Finally, run your mount command and verify that you have a pnfs mount (see "Mount Filesystem" and "Generate Traffic" above). The daemon should print some messages as it discovers your devices:
 +
<pre>
 +
pdsi7# /usr/sbin/blkmapd -f
 +
blkmapd: process_deviceinfo: 12 vols
 +
blkmapd: decode_blk_signature: si_comps[0]: bs_length 4, bs_string 0x14
 +
blkmapd: decode_blk_signature: si_comps[1]: bs_length 32, bs_string APM000644032240000
 +
blkmapd: read_cmp_blk_sig: /dev/sdn sig 0x14 at -65536
 +
blkmapd: read_cmp_blk_sig: /dev/sdn sig APM000644032240000 at -65436
 +
blkmapd: decode_blk_volume: simple 0
 +
...
 +
blkmapd: decode_blk_volume: stripe 10 nvols=10 unit=512
 +
blkmapd: decode_blk_volume: concat 11 1
 +
blkmapd: dm_device_create: 10 pnfs_vol_0 253:0
 +
blkmapd: dm_device_create: 11 pnfs_vol_1 253:1
 +
</pre>
 +
 +
These messages will vary depending on your device topology.
 +
If you still don't have a pnfs mount, the daemon may have printed something useful, or you may find something in <tt>/var/log/messages</tt>.

Latest revision as of 12:14, 15 May 2012

Contents

Installing Fedora

The Install Guide at http://docs.fedoraproject.org/ describes numerous ways in which to install Fedora. Choose the best method for you.

  • Select local disk and take default disk partitions for /boot, /swap LVM etc
  • If your block device is accessed through iSCSI then click on “Advanced disk configuration” tab and select iSCSI. If you are using FC you are recommended to unplug Fiber cable before installation and connect back before reboot. Do not initialize Block devices if you are using EMC Unified storage devices (Celerra NAS)
  • In order to run the Connectathon test suite you will need the “software development” package.
    • Click on the "Software Development" button in the package install screen similar to this
    • Its also advisable to add in the 'Fedora' repository on the same page.
    • Finally it's also a good idea to do a 'yum -y update' first thing after the install completes. This will ensure you have the most up to date bits available.

Installing pNFS Enabled Fedora kernel

As of kernel 3.1 the block layout client is included in the standard kernel and you can skip this section.

There are two way to install the pNFS enabled kernels. Either using a yum repository or directly downloading.

Yum repository

For Fedora 12 (kernel-2.6.32) and Fedora 13 (kernel-2.6.34) kernels use the http://steved.fedorapeople.org/pnfs.repo

For Fedora development kernels (kernel-2.6.32) repository http://steved.fedorapeople.org/pnfs-rawhide.repo

Download the appropriate repository into /etc/yum.repos.d directory: 

cd /etc/yum.repos.d; wget http://steved.fedorapeople.org/pnfs.repo

Then use one of the following commands to install the kernel of choice (including the dependencies). To install non-debug kernels: 

yum --disablerepo='*' --enablerepo=pnfs install kernel\*

To install debug enabled kernels: 

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

To install debuginfo rpms , which aid with debugging: 

yum --disablerepo='*' --enablerepo=pnfs-debug install kernel\*

Note: For development kernels use '--enablerepo=pnfs-rawhide-XXX'

Direct download

You can directly download the pNFS kernel rpms from http://steved.fedorapeople.org/repos/pnfs. With the direct downloads you will need periodicity check for updates as well as figure the dependencies. If you use this method, and do not add the pnfs repo to yum, then a subsequent "yum update" may erase your pnfs changes and revert your system to pre-pnfs. It may even render your system unusable.

Kernel source

As of linux kernel 3.1, the block layout driver is included in the standard kernel. The latest pnfs source code, including bug fixes and experimental code that has not yet been merged upstream, is in git://linux-nfs.org/~bhalevy/linux-pnfs.git on the pnfs-all-latest branch.

You can find the latest block layout kernel source in git://citi.umich.edu/projects/linux-pnfs-blk.git. This is for developers only.

Target and blkmapd setup for block layout client

If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.

You need the pnfs version of nfs-utils. If you have already added the pnfs yum repo, you can get this just by doing a "yum update". You can also install an rpm from http://steved.fedorapeople.org/repos/pnfs or build it from the git source tree at git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git.

As of version 1.2.5, the standard nfs-utils package includes support for pNFS, including the block layout client, but you should apply this patch to prevent your logs from filling with garbage: 0001-remove-pretty_sig.patch

You will need to set up iscsi targets on your server, and any logins or permissions required by local policy. The procedure depends on the server.

You should lower the hung task timeout to aid in debugging: 

sysctl -w kernel.hung_task_timeout_secs=10

Now connect to your iscsi targets, similar to: 

iscsiadm -m discovery -t sendtargets -p <iscsi-server> -l

And start the block layout service, which will load kernel modules and start blkmapd: 

service blkmapd restart

If you get an error message "blkmapd: unrecognized service" you may be missing the init file. Install it from the CITI pnfs web site: 

wget -O /etc/rc.d/init.d/blkmapd http://www.citi.umich.edu/projects/nfsv4/pnfs/block/download/rh-init.txt
chmod +x /etc/rc.d/init.d/blkmapd

Mount Filesystem

Use the '-o minorversion=1' mount option when mounting the server, similar to: 

mount -t nfs4 -o minorversion=1 <server>:/export  /mnt

Generate Traffic

Generate some I/O using “dd” or run “Connectathon ”. You may download Connectathon from http://www.connectathon.org. All tests are expected to pass without errors.

To verify pNFS is up and working, grep for the word 'LAYOUT' in /proc/self/mountstat. You should see some non-zero values. 

fedora# grep LAYOUT /proc/self/mountstats
        nfsv4:  bm0=0xfcff8fff,bm1=0x40f9bfff,acl=0x3,sessions,pnfs=LAYOUT_BLOCK_VOLUME
        PNFS_LAYOUTGET: 2561 2561 0 655616 256284 34 1698 2575
        PNFS_LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
        PNFS_LAYOUTRETURN: 1 1 0 252 88 0 0 1

Unmount and disconnect

umount /mnt
iscsiadm -m node -U all

Troubleshooting

If you are using file or object layouts, skip this section. If you are using block layouts with iscsi targets, follow these instructions.

If this doesn't work for you, follow these steps to find the problem. First make sure your iscsi targets have been attached. For each target device you should see something like this in /var/log/messages: 

Mar  7 09:46:34 rhcl1 kernel: scsi 7:0:0:15: Direct-Access     DGC      RAID 5           0326 PQ: 0 ANSI: 4
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: Attached scsi generic sg32 type 0
Mar  7 09:46:34 rhcl1 kernel: sd 7:0:0:15: [sdq] 1125628928 512-byte logical blocks: (576 GB/536 GiB)
Mar  7 09:46:36 rhcl1 kernel: sd 7:0:0:15: [sdq] Write Protect is off
Mar  7 09:46:37 rhcl1 kernel: sd 7:0:0:15: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Mar  7 09:46:37 rhcl1 kernel: sdq: unknown partition table
Mar  7 09:46:38 rhcl1 kernel: sd 7:0:0:15: [sdq] Attached SCSI disk

You should also see the block devices in the /sys/block directory: 

pdsi7# ls /sys/block
loop0  loop6  ram11  ram3  ram9  sdae  sdak  sdaq  sdb  sdh  sdn  sdt  sdz
loop1  loop7  ram12  ram4  sda   sdaf  sdal  sdar  sdc  sdi  sdo  sdu
loop2  md127  ram13  ram5  sdaa  sdag  sdam  sdas  sdd  sdj  sdp  sdv
loop3  ram0   ram14  ram6  sdab  sdah  sdan  sdat  sde  sdk  sdq  sdw
loop4  ram1   ram15  ram7  sdac  sdai  sdao  sdau  sdf  sdl  sdr  sdx
loop5  ram10  ram2   ram8  sdad  sdaj  sdap  sdav  sdg  sdm  sds  sdy

Next load the kernel module: 

modprobe blocklayoutdriver
pdsi7# modprobe blocklayoutdriver
pdsi7#

Now run the daemon in foreground mode: 

pdsi7# /usr/sbin/blkmapd -f

Finally, run your mount command and verify that you have a pnfs mount (see "Mount Filesystem" and "Generate Traffic" above). The daemon should print some messages as it discovers your devices: 

pdsi7# /usr/sbin/blkmapd -f
blkmapd: process_deviceinfo: 12 vols
blkmapd: decode_blk_signature: si_comps[0]: bs_length 4, bs_string 0x14
blkmapd: decode_blk_signature: si_comps[1]: bs_length 32, bs_string APM000644032240000
blkmapd: read_cmp_blk_sig: /dev/sdn sig 0x14 at -65536
blkmapd: read_cmp_blk_sig: /dev/sdn sig APM000644032240000 at -65436
blkmapd: decode_blk_volume: simple 0
...
blkmapd: decode_blk_volume: stripe 10 nvols=10 unit=512
blkmapd: decode_blk_volume: concat 11 1
blkmapd: dm_device_create: 10 pnfs_vol_0 253:0
blkmapd: dm_device_create: 11 pnfs_vol_1 253:1

These messages will vary depending on your device topology. If you still don't have a pnfs mount, the daemon may have printed something useful, or you may find something in /var/log/messages.

Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/Fedora_pNFS_Client_Setup"
Personal tools