GFS2 Setup Notes

From Linux NFS

You do not have permission to edit this page, for the following reason:

You can view and copy the source of this page:

==Initial install== ===Basics=== Started with fresh installs of RHEL5.0 on 4 nodes of mixed hardware, all attached to a shared MSA-1000 fibre channel 8-disk array (in two sets of 4, ~550GB total). * installed cluster and update RPMs from wendy cheng: ** <tt> cman-2.0.64-1.el5.x86_64.rpm </tt> ** <tt> cman-devel-2.0.64-1.el5.x86_64.rpm </tt> ** <tt> device-mapper-1.02.13-1.el5.x86_64.rpm </tt> ** <tt> gfs-utils-0.1.11-3.el5.x86_64.rpm </tt> ** <tt> gfs2-utils-0.1.25-1.el5.x86_64.rpm </tt> ** <tt> gnbd-1.1.5-1.el5.x86_64.rpm </tt> ''(unused?)'' ** <tt> kmod-gfs-0.1.16-5.2.6.18_8.1.4.el5.x86_64.rpm </tt> ** <tt> kmod-gnbd-0.1.3-4.2.6.18_8.1.4.el5.x86_64.rpm </tt> ''(unused?)'' ** <tt> lvm2-2.02.16-3.el5.x86_64.rpm </tt> ** <tt> lvm2-cluster-2.02.16-3.el5.x86_64.rpm </tt> ** <tt> openais-0.80.2-1.el5.x86_64.rpm </tt> ** <tt> openais-devel-0.80.2-1.el5.x86_64.rpm </tt> ** <tt> system-config-cluster-1.0.39-1.0.noarch.rpm </tt> ''(just a python frontend for several <tt>vg*</tt>, <tt>lv*</tt>, and <tt>pv*</tt> commands)'' ===Configuring <tt>cman</tt> and <tt>clvmd</tt>=== * '''cman''': at first I tried using <tt>system-config-cluster</tt> to set up <tt>cman</tt>, but given that I didn't have any complicated fencing or quorum-related needs, I basically just took a generic <tt>cluster.conf</tt> and edited it. My <tt>[http://www.citi.umich.edu/u/richterd/gfs2/cluster.conf cluster.conf]</tt> is real basic and has manual fencing set up to be a no-op (I'd get complaints from the daemons if I didn't have any fencing setup). ** distribute the new <tt>cluster.conf</tt> to all nodes; on the first run, you can just use <tt>scp</tt> or whatever. ** once the cluster's up, though, propagating and setting changes on all nodes takes two steps. From the node with the updated configuration, do: *** <tt>$ sudo ccs_tool update /path/to/new/cluster.conf</tt> ''(pushes to all nodes listed in conf file)'' *** <tt>$ sudo cman_tool version -r <new-version-number></tt> ''(a generation number to keep the nodes synched)'' * '''clvmd''': as before, I tried using <tt>system-config-lvm</tt> to set up <tt>clvmd</tt>, but it's not quite "there yet" -- it'd get wedged or go blind to clustered volumes at strange times. Again, tweaking a mostly-templated (and very well-commented) stock conf file wasn't hard; my <tt>[http://www.citi.umich.edu/u/richterd/gfs2/lvm.conf lvm.conf]</tt> is real simple. ''Note:'' btw, in my setup the MSA-1000 disk array is initially set up to do raid0 on the 8 disks in two groups of 4; my machines see 2 block devices, each with a capacity of ~270GB. ** create 1 physical linux (0x83) partition each, using whole "disk"; repeat for <tt>/dev/sdc</tt> *** <tt>$ sudo fdisk /dev/sdb</tt> ** create physical volumes with LVM2 metadata *** <tt>$ sudo pvcreate -M 2 /dev/sdb1</tt> *** <tt>$ sudo pvcreate -M 2 /dev/sdc1</tt> ** create a clustered volume group and add <tt>/dev/sdb1</tt> to it *** <tt>$ sudo vgcreate -M 2 -l 256 -p 256 -s 4m -c y VolGroupCluster /dev/sdb1</tt> *** <tt>$ sudo pvscan</tt> ''# (verify it worked)'' ** edit <tt>lvm.conf</tt> and make sure that "<tt>locking_type</tt>" is set to 3 (<tt>DLM</tt>). ** distribute <tt>lvm.conf</tt> to all the nodes ** start up both <tt>cman</tt> and <tt>clvmd</tt> everywhere. ''Note:'' fwiw, I use [https://computing.llnl.gov/linux/pdsh.html pdsh], the parallel distributed shell, to communicate to all nodes at once; I have mine use <tt>ssh</tt> for transport. E.g., from my .bashrc: *** <tt> $ alias start-cluster='for svc in cman clvmd ; do pdsh -w node[1-4] sudo service $svc start; done'</tt> ** add <tt>/dev/sdc1</tt> to the existing volume group (needs the daemons running) *** <tt>$ sudo vgextend VolGroupCluster /dev/sdc1</tt> *** <tt>$ sudo vgs</tt> ''# (verify that the "clustering" flag is set on the volgroup)'' ** create a logical volume using the whole volgroup *** <tt>$ sudo lvcreate -n ClusterVolume -l 138924 VolGroupCluster</tt> *** <tt>$ sudo lvdisplay -c -a</tt> ''# (verify that it worked)'' ** create a GFS2 filesystem therein *** <tt>$ sudo gfs2_mkfs -j 4 -p lock_dlm -t GFS2_Cluster:ClusterFS -O /dev/VolGroupCluster/ClusterVolume</tt> ** edit <tt>/etc/fstab</tt> to add a mountpoint, restart the daemons, and mount! ===Custom kernels=== Once the basics were going, I built some kernels and things more or less worked -- except I had a heck of a time getting the <tt>Qlogic</tt> firmware to load properly. I'm fine with building the <tt>initcramfs "initrds"</tt> by hand, ''but'' for the firmware in this setup; I don't know, I guess I'm a <tt>udev</tt> idiot or something. What I ended up doing was bogarting a vendor patch from Red Hat (bless their hearts ;) that side-stepped the issue and just built the blobs into the GFS kernel module. A [http://www.citi.umich.edu/u/richterd/gfs2/add-qlogic-firmware-blob--2.6.22.19.diff slightly-updated version against 2.6.22.19] is available. ==Upgrading GFS2 userland for kernels >2.6.18== Not too long after the initial install (which came with a 2.6.18-based kernel), I found that the in-kernel <tt>DLM</tt> (distributed lock manager) stuff changed recently and required a corresponding update to userspace <tt>LVM2</tt> (logical volume manager) tools. While Wendy Cheng had gotten things off the ground by giving me the bag of RPMs, we didn't get any RHN entitlements, so no updates = pain in the neck. I did finally manage to find a way to sneak RHEL5 packages out of RHN despite the lack of entitlement, but I had to do it by hand and I had to re-login for each package. Worse, when I finally did get the newest RPMs, they weren't even new enough anyway. Lesson learned: build from source. I wasn't sure that it was the best idea, but since I already had GFS2 working with the stock userland, I was skittish and didn't want to clobber the system RPMs so I installed under my home directory; worked fine. * got the newest packages: ** [http://sources.redhat.com/dm/ device-mapper.1.02.22] ** [http://www.openais.org/ openAIS] ''(get the stable/"whitetank" release)'' ** [ftp://sources.redhat.com/pub/cluster/releases/ cluster-2.01.00 tools] ** [http://sources.redhat.com/lvm2/ LVM2.2.02.28] ** <tt>libvolume_id-devel-095-14.5.el5.x86_64.rpm</tt> ''(bogarted from RHN)'' * <tt>export CLUSTER=/home/richterd/projects/nfs/CLUSTER; cd $CLUSTER</tt> * <tt>mkdir device-mapper-OBJ cluster-OBJ LVM2-OBJ</tt> * device-mapper: ** <tt>./configure --prefix=$CLUSTER/device-mapper-OBJ && make && sudo make install</tt> ** add <tt>$CLUSTER/device-mapper-OBJ/lib</tt> to <tt>/etc/ld.so.conf</tt> and rerun <tt>ldconfig</tt> * openAIS: ** edit the Makefile; set <tt>DESTDIR</tt> to the empty string ** <tt>make && sudo make install</tt> -- at some point, this clobbered some of the RPM stuff; meh. ** added <tt>/usr/lib64/openais</tt> to <tt>ld.so.conf</tt> and reran <tt>ldconfig</tt> ** ** '''update''': when i came back to this and was building on Fedora 9, i got complaints about <tt>struct ucred</tt> not being defined (see [http://sourceware.org/bugzilla/show_bug.cgi?id=6545 this bugreport]). I edited $OPENAIS/exec/Makefile and added <tt>-D_GNU_SOURCE</tt> to its CFLAGS and things seem copacetic. * libvolume_id-devel: ** <tt>sudo rpm -ivh libvolume_id-devel-095-14.5.el5.x86_64.rpm</tt> * cluster tools: ** <tt>./configure --prefix=$CLUSTER/cluster-OBJ --openaislibdir=/usr/lib64/openais --dlmincdir=/lib/modules/<kernel>/source/include</tt> *** '''update''': am now omitting a couple things (gfs doesn't build right any longer anyway)<tt>./configure --without_gnbd --without_gfs</tt> ** edit <tt>dlm/lib/Makefile</tt> and add: <tt>CFLAGS += -I$(dlmincdir)</tt> ** since I was doing my "trial" install, I added <tt>$CLUSTER/cluster-OBJ/usr/lib</tt> to <tt>ld.so.conf</tt> and reran <tt>ldconfig</tt>. I anticipate going back and installing things in real system locations now that I know things worked <tt>:)</tt> ** <tt>make && sudo make install</tt> * LVM2: ** <tt>./configure --prefix=$CLUSTER/LVM2-OBJ --with-lvm1=none --with-dmdir=$CLUSTER/device-mapper-OBJ --with-clvmd=cman</tt> ** edit <tt>make.tmpl</tt> and look for where the above <tt>dmdir</tt> is set; my <tt>configure</tt> screwed up and appended <tt>"/ioctl"</tt> to the end and I had to trim it. *** '''fix''': rather, first trim from <tt>make.tmpl.in</tt>, where it originates for whatever reason ** <tt>make && sudo make install</tt> .. at this point, I had a <tt>clvmd</tt> that linked against the right shared libraries and that could deal with the kernel's modified <tt>DLM</tt> setup. ==Troubleshooting the clustering flag== '''Problem 1:''' LVM changes don't appear to "take". Quoting from an email I found online (XXX: cite): Why aren't changes to my logical volume being picked up by the rest of the cluster? There's a little-known "clustering" flag for volume groups that should be set on when a cluster uses a shared volume. If that bit is not set, you can see strange lvm problems on your cluster. For example, if you extend a volume with lvresize and gfs_grow, the other nodes in the cluster will not be informed of the resize, and will likely crash when they try to access the volume. To check if the clustering flag is on for a volume group, use the "vgs" command and see if the "Attr" column shows a "c". If the attr column shows something like "wz--n-" the clustering flag is off for the volume group. If the "Attr" column shows something like "wz--nc" the clustering flag is on. To set the clustering flag on, use this command: <tt>vgchange -cy</tt> '''Problem 2:''' In the midst of adding a new node to the cluster, <tt>clvmd</tt> wouldn't start on other nodes and recognize the disk array. I tried the above <tt>vgchange -cy</tt> thing and screwed it up by making the ''local disk's'' VG '''clustered''' (ugh). [http://kbase.redhat.com/faq/FAQ_96_11024.shtm The problem] made sense, but the temporarily-changing-the-locking-type was what I was missing when I tried to undo my mistake. The fix: make sure uniform <tt>lvm.conf</tt>s are tweaked as per the link above and distributed to the cluster; start <tt>cman/clvmd</tt> everywhere; ''then'' use <tt>vgchange -cn VolGroup00</tt> to remove clustering flag (''<tt>VolGroup00</tt> is the local disk's VG, set up during the RHEL install''); ''then'' set the <tt>lvm.conf</tt> locking stuff back to "clustered" and redistribute to the cluster; ''then'' restart the daemons, mount, declare victory.

Return to GFS2 Setup Notes.