From Linux NFS

Revision as of 19:31, 15 December 2020

This is an attempt to provide pointers to the basic information necessary to start hacking the Linux NFS implementation. I assume that you know C and know the basics of administering a Linux box (so I assume, for example, that you know how to build and install a new kernel). I don't assume a knowledge of kernel internals.

setting up NFS

NFS is easy to set up and use; follow instructions for your distribution and play around a bit.

understanding NFSv4

The authoritative sources are RFC 7530 (NFSv4.0), RFC 5881 (NFSv4.1), and related documents from the NFSv4 IETF working group. Don't read them from start to finish! They're too long. But keep them on hand to refer to when you need to understand something specific.

The best way to watch NFSv4 at work is to run NFSv4 while watching your network with a packet sniffer. Use Wireshark: it's widely available and has up-to-date support for NFSv4. Once again, your traffic doesn't have to be going over a "real" network for this to work; if your client and server are on the same machine, just sniff the loopback interface ("lo").

Wireshark also has a companion program, tshark, with a text-only interface.

I usually adjust the Wireshark preferences to give the "Packet Details" panel the full height of the window. You may also need to set:

Protocols->TCP->"Allow subdissector to desegment TCP streams"
Protocols->IP ->"Reassemble fragmented IP datagrams"
Protocols->RPC->"Desegment all RPC-over-TCP messages"
Protocols->RPC->"Defragment all RPC-over-TCP messages"

In addition to providing filters in the capture dialog or (with the -f option) on the commandline, Wireshark also gives you some help constructing filters after-the-fact: right-click on an element in the middle pane and play around with the "prepare" and "match" menus. One additional hint: right-clicking on an element and then choosing "expand tree" recursively expands everything in that element.

reading kernel code

The best way to understand how some part of the kernel works is usually just to read the code.

The best way to get the code is to install git, then run:

# get linus's mainline tree:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
# if you want to see in-progress kernel work from other developers, you can add them with for example:
git remote add -f trond git://linux-nfs.org/pub/linux/nfs-2.6.git
git remote add -f bfields git://linux-nfs.org/~bfields/linux.git

Then you can check out different versions with:

git checkout v2.6.25
git checkout v2.6.26-rc1
git remote show trond    # what branches does Trond have?
git checkout trond/devel # Checkout the tip of Trond's "devel" branch

and download new updates with:

git fetch origin
git fetch trond
git fetch bfields

(But note this does not affect your working directory; if you want to see what's new on some branch, you'll need to run "git checkout" again.)

"git grep" is useful for finding your way around, but you may also want to set up a good text editor integrated with a database of code cross-references. I use cscope and vim. The cscope home page has instructions on using cscope with vim and emacs, and instructions on using cscope on a large project like the kernel without waiting forever for the indices to build. This allows you to follow the flow of control easily by popping quickly from the use of a function to its definition and back.

Take notes. As an example, I keep some outdated notes on the kernel. In many cases they're too rough to be of use to someone else, but they help me organize my thoughts while I'm learning something new.

It's easy to get lead astray if one attempts to understand large subsystems all at once. Instead, try to keep in mind one small goal (e.g., to fix a bug, to learn how to use a certain interface).

Robert Love's "Linux Kernel Development" gives a good overview if read side-by-side with the kernel code. "Linux Device Drivers" is also good, as is "Understanding the Linux Kernel". See also lwn.net's kernel coverage.

NFS Debugging

You can use the rpcdebug command (included in nfs-utils) to get additional debugging information dumped in your logs.

(To see the code that produces this, see include/linux/sunrpc/debug.h, include/linux/nfs_fs.h, include/linux/nfsd/debug.h, the NFSDDBG_FACILITY defines at the top of each .c file, and the dprintk()'s sprinkled throughout. generating patches

Patches are the basic unit of communication with other kernel hackers. They should be readable by humans, not just by the patch command. To this end:

• Use the "-u" option to diff. I also like the "-p" option, which adds a guess at the current C function before each chunk.
• Make patches that are short and that only do one thing. This may mean that, after writing a bunch of new code to implement a new feature, you need to spend some time breaking up the code into smaller patches which introduce the new feature in easier-to-understand chunks.
• When dealing with a long series of patches, make sure that each individual patch introduces no compile-time or run-time regressions.

See also Andrew Morton's The Perfect Patch, and see Documentation/SubmittingPatches in your friendly local kernel tree.

Generating the patches can be done with diff -urNp if you have two clean trees to compare; things get more complicated if you mistakenly do a "make" in one of the two trees, or if you start having to deal with long series of patches.

There are a several ways people deal with this. I use git, which is what the upstream kernel source is kept in. See the tutorial and user's manual for an introduction. Specifically, the chapter Rewriting history and maintaining patch series may be useful.

Mailing Lists

It's helpful to at least skim the following mailing lists:

• linux-kernel@vger.kernel.org: traffic is several hundred posts a day, so don't subscribe unless you have some filtering set up to help you cope.
• linux-nfs@vger.kernel.org: nfs/linux stuff.
• nfsv4@ietf.org: ietf NFSv4 working group.

Miscellaneous

Use the -s option to make; this eliminates most of its output so that you can see (potentially important) compiler warnings more easily.

GDB Tricks

When you encounter an OOPS, you'll get a Call trace like:

Call Trace:

[<ffffffff8110a3ef>] kfree+0x63/0xfc
[<ffffffffa011e18e>] nfs_free_parsed_mount_data+0x24/0xc1 [nfs]
[<ffffffffa0121743>] nfs_fs_mount+0x5ac/0x61c [nfs]
[<ffffffff81116a8d>] mount_fs+0x69/0x158
[<ffffffff810ea5ad>] ? __alloc_percpu+0x10/0x12
[<ffffffff8112b8a8>] vfs_kern_mount+0x65/0xc4
[<ffffffff8112bf07>] do_kern_mount+0x4d/0xdf
[<ffffffff8112d6b7>] do_mount+0x64b/0x6af
[<ffffffff8112cfd0>] ? copy_mount_options+0xcb/0x12e
[<ffffffff8112d81e>] sys_mount+0x88/0xc2
[<ffffffff81426be9>] system_call_fastpath+0x16/0x1b

There is a simple way to translate a <symbol>/<offset> to a line number:

• run gdb passing the correct kernel module (.ko) as the only argument
• use the "l * (<symbol>+<offset>)" command

$ gdb obj/fs/nfs/nfs.ko 

...

(gdb) l * (nfs_free_parsed_mount_data+0x24)
0x918e is in nfs_free_parsed_mount_data (/home/dros/build/src/fs/nfs/super.c:924).
919	
920	static void nfs_free_parsed_mount_data(struct nfs_parsed_mount_data *data)
921	{
922		if (data) {
923			kfree(data->client_address);
924			kfree(data->mount_server.hostname);
925			kfree(data->nfs_server.export_path);
926			kfree(data->nfs_server.hostname);
927			kfree(data->fscache_uniq);
928			security_free_mnt_opts(&data->lsm_opts);