PNFS Block Server Setup Instructions
From Linux NFS
BennyHalevy (Talk | contribs) |
(→blkmapd) |
||
(10 intermediate revisions not shown) | |||
Line 3: | Line 3: | ||
This page describes the setup of the pNFS Block Layout Server. This is based on the Rick McNeal's how to guide. Please note that Fedora 11 was used to setup the server, some of the content you see might be specific to Fedora ( for e.g yum). | This page describes the setup of the pNFS Block Layout Server. This is based on the Rick McNeal's how to guide. Please note that Fedora 11 was used to setup the server, some of the content you see might be specific to Fedora ( for e.g yum). | ||
- | + | Note that this is an early development prototype, and recently has not been actively maintained; thus this is recommended for developers only. | |
==Building the code== | ==Building the code== | ||
Line 15: | Line 15: | ||
'''git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git''' | '''git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git''' | ||
- | + | Use the pnfs-all-latest branch and configure the kernel as follows: | |
+ | |||
+ | CONFIG_NFSD=m | ||
+ | CONFIG_NFSD_V4=y | ||
+ | CONFIG_PNFSD=y | ||
+ | # CONFIG_PNFSD_LOCAL_EXPORT is not set | ||
+ | CONFIG_PNFSD_BLOCK=y | ||
+ | |||
+ | Note: CONFIG_PNFSD_LOCAL_EXPORT must be disabled for pnfsd-block since | ||
+ | it currently takes precedence over it in run-time. | ||
This page doesn't discuss anything about kernel compilation. | This page doesn't discuss anything about kernel compilation. | ||
- | '''2) Building the | + | '''2) Building the nfsutils and utils/blkmapd''' |
- | + | The nfs-utils-1.2.5-3.fc16.x86_64 Fedora RPM package already includes the blkmapd service | |
- | + | ||
- | + | Here the instructions for building the utilities for older distributions: | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
Obtain the "nfs-utils" source code. | Obtain the "nfs-utils" source code. | ||
- | '''git clone | + | '''git clone git://linux-nfs.org/~bhalevy/pnfs-nfs-utils.git''' |
Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several | Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several | ||
- | + | packages are required. I have either installed or updated the following packages. | |
- | yum | + | A one-liner command to install all prerequisites on Fedora 15: |
+ | yum install autoconf automake libtool libtirpc{,-devel} tcp_wrappers{,-devel} libevent{,-devel} \ | ||
+ | libnfsidmap{,-devel} openldap-devel libgssglue{,-devel} krb5-devel libblkid{,-devel} \ | ||
+ | device-mapper-devel libcap{,-devel} | ||
- | yum install libtirpc-devel | + | Or run selectively: |
+ | yum install autoconf automake | ||
+ | yum install libtool | ||
+ | yum install libtirpc-devel | ||
+ | yum install tcp_wrappers-devel | ||
+ | yum install libevent-devel | ||
+ | yum install libnfsidmap-devel | ||
+ | yum install nfs-utils-lib-devel (deprecated in Fedora 15) | ||
+ | yum install openldap-devel | ||
+ | yum install libgssglue-devel | ||
- | yum install | + | # Fedora 12 and later require also: |
+ | yum install libblkid-devel | ||
+ | yum install device-mapper-devel | ||
- | yum install | + | # Fedora 13 and later require also: |
+ | yum install krb5-devel | ||
- | + | == blkmapd == | |
- | + | The blkmapd daemon should be run on the pnfs client to map the block devices | |
+ | according the pnfs device information. | ||
+ | See utils/blkmapd/etc/blkmapd.conf and utils/blkmapd/etc/initd/initd.redhat for | ||
+ | more information about its setup. | ||
- | + | For more instructions on setting up the client, see [[Fedora pNFS Client Setup]]. | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
== Exporting the filesystem == | == Exporting the filesystem == | ||
- | For the block access to work properly the disks must have a signature. | + | For the block access to work properly the disks must have a signature. Partitioned the disks using "parted". Disks partitioned with "fdisk" doesn't have the signatures. |
I have followed the below mentioned steps. | I have followed the below mentioned steps. | ||
Line 119: | Line 99: | ||
I haven't setup the block storage and metadata server on the same machine. You may setup them on two different machines , but client and metadata sever should see the same disks. | I haven't setup the block storage and metadata server on the same machine. You may setup them on two different machines , but client and metadata sever should see the same disks. | ||
- | + | I have used iSCSI to setup the block storage , "scsi-target-utils" is required to setup the iscsi target. | |
- | I have used | + | One key thing is when adding a LUN to the target , don't add the disk partition (/dev/sdb1), instead add the entire disk(/dev/sdb). |
- | One key thing is when adding a LUN to the target , don't add the disk | + | |
The disk signatures are not visible when if you add the disk partetion to the target. | The disk signatures are not visible when if you add the disk partetion to the target. | ||
Line 128: | Line 107: | ||
/mnt *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs) | /mnt *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs) | ||
+ | |||
+ | == ctl == | ||
+ | |||
+ | The "ctl" daemon should be run on the pnfs server to map the block devices | ||
+ | according the pnfs device information. | ||
+ | |||
+ | Obtain the "ctl" source code. | ||
+ | '''git clone git://linux-nfs.org/~bhalevy/pnfsd-block-ctl.git''' | ||
+ | |||
+ | Simply run "make" to build the daemon. | ||
+ | |||
+ | Run the daemon on the server using the following command: | ||
+ | # ctl -u | ||
== How to Start the server == | == How to Start the server == | ||
- | I have used the following | + | I have used the following script to start the server |
#/bin/bash | #/bin/bash | ||
Line 159: | Line 151: | ||
== Mount from the client == | == Mount from the client == | ||
- | mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob | + | # modprobe blocklayoutdriver |
+ | # mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob | ||
== How to verify == | == How to verify == | ||
Line 165: | Line 158: | ||
- tcpdump/wireshark is the best way to see what is happening. | - tcpdump/wireshark is the best way to see what is happening. | ||
- The other way is after mounting the export,on the client check /proc/self/mountstats. | - The other way is after mounting the export,on the client check /proc/self/mountstats. | ||
+ | - nfsstat also provides operations counters | ||
+ | - echo 32767 > /proc/sys/sunrpc/nfs_debug (and nfsd_debug, correspondingly) provides debug printouts in /var/log/messages |
Latest revision as of 00:15, 16 March 2012
How to guide to setup the pNFS Block Layout server based on sPNFS
This page describes the setup of the pNFS Block Layout Server. This is based on the Rick McNeal's how to guide. Please note that Fedora 11 was used to setup the server, some of the content you see might be specific to Fedora ( for e.g yum).
Note that this is an early development prototype, and recently has not been actively maintained; thus this is recommended for developers only.
Contents |
Building the code
1) Building the kernel source
Obtain the code from Linux pNFS git. pNFS Block Layout server is currently a part of the pNFS git.
git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git
Use the pnfs-all-latest branch and configure the kernel as follows:
CONFIG_NFSD=m CONFIG_NFSD_V4=y CONFIG_PNFSD=y # CONFIG_PNFSD_LOCAL_EXPORT is not set CONFIG_PNFSD_BLOCK=y
Note: CONFIG_PNFSD_LOCAL_EXPORT must be disabled for pnfsd-block since it currently takes precedence over it in run-time.
This page doesn't discuss anything about kernel compilation.
2) Building the nfsutils and utils/blkmapd
The nfs-utils-1.2.5-3.fc16.x86_64 Fedora RPM package already includes the blkmapd service
Here the instructions for building the utilities for older distributions:
Obtain the "nfs-utils" source code.
git clone git://linux-nfs.org/~bhalevy/pnfs-nfs-utils.git
Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several packages are required. I have either installed or updated the following packages.
A one-liner command to install all prerequisites on Fedora 15:
yum install autoconf automake libtool libtirpc{,-devel} tcp_wrappers{,-devel} libevent{,-devel} \ libnfsidmap{,-devel} openldap-devel libgssglue{,-devel} krb5-devel libblkid{,-devel} \ device-mapper-devel libcap{,-devel}
Or run selectively:
yum install autoconf automake yum install libtool yum install libtirpc-devel yum install tcp_wrappers-devel yum install libevent-devel yum install libnfsidmap-devel yum install nfs-utils-lib-devel (deprecated in Fedora 15) yum install openldap-devel yum install libgssglue-devel
# Fedora 12 and later require also: yum install libblkid-devel yum install device-mapper-devel
# Fedora 13 and later require also: yum install krb5-devel
blkmapd
The blkmapd daemon should be run on the pnfs client to map the block devices according the pnfs device information. See utils/blkmapd/etc/blkmapd.conf and utils/blkmapd/etc/initd/initd.redhat for more information about its setup.
For more instructions on setting up the client, see Fedora pNFS Client Setup.
Exporting the filesystem
For the block access to work properly the disks must have a signature. Partitioned the disks using "parted". Disks partitioned with "fdisk" doesn't have the signatures.
I have followed the below mentioned steps.
- parted /dev/sdb
(parted) mklabel gpt (parted) mkpart 1 <Provide start and end of the partetions> (parted) print Model: VMware Virtual disk (scsi) Disk /dev/sdb: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 53.7GB 53.7GB ext3 1 msftres
I have tested with ext4 filesystem, create ext4 filesystem with 4K block size.
# mkfs.ext4 -b 4096 /dev/sdb1
Setting up the BLOCK storage / SAN
I haven't setup the block storage and metadata server on the same machine. You may setup them on two different machines , but client and metadata sever should see the same disks.
I have used iSCSI to setup the block storage , "scsi-target-utils" is required to setup the iscsi target. One key thing is when adding a LUN to the target , don't add the disk partition (/dev/sdb1), instead add the entire disk(/dev/sdb).
The disk signatures are not visible when if you add the disk partetion to the target.
Export Options
/mnt *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs)
ctl
The "ctl" daemon should be run on the pnfs server to map the block devices according the pnfs device information.
Obtain the "ctl" source code.
git clone git://linux-nfs.org/~bhalevy/pnfsd-block-ctl.git
Simply run "make" to build the daemon.
Run the daemon on the server using the following command:
# ctl -u
How to Start the server
I have used the following script to start the server
#/bin/bash # UMOUNT /mnt umount /mnt #start the service service tgtd restart sleep 8 # Create iSCSI target tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.1992-05.com.emc:openblock # Expose LUN as iSCSI target tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/sdb # Allow acces of all initiator tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL # show all the details tgtadm --lld iscsi --op show --mode target # mount the partetion mount /dev/sdb1 /mnt sleep 3 # start the nfs server service nfs restart sleep 3 # start the deamon cd <CTL_SRC>/ctl/ ./ctl -u &
Mount from the client
# modprobe blocklayoutdriver # mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob
How to verify
- tcpdump/wireshark is the best way to see what is happening. - The other way is after mounting the export,on the client check /proc/self/mountstats. - nfsstat also provides operations counters - echo 32767 > /proc/sys/sunrpc/nfs_debug (and nfsd_debug, correspondingly) provides debug printouts in /var/log/messages