From Linux NFS

Revision as of 18:50, 21 December 2012

Overview

Peer-to-peer pNFS is designed to solve the "boot storm" problem that happens when several clients in a cluster boot and attempt to read the same set of files from a single NFS server all at the same time. This could overload the server's bandwidth, slowing down operations on most client machines. The idea behind p2p NFS is to allow clients to act as an adhoc read-only pNFS data server that serves files out of their data cache. This should spread out network usage across all machines, rather than focusing all activity on a single node. Server and desired DS machines will need to be modified but any pNFS-enabled client already has the code required to read from adhoc DSs.

Related Documents

• draft-myklebust-nfsv4-pnfs-backend-protocol-01.txt
• RFC 5661

Dependencies

This design needs the following from others:

Assumptions

• Workload with large number of read-only files
• Enable the following .config options for the pNFS client and pNFS ds machines:
  • CONFIG_NFS_V4_1
  • CONFIG_PNFS_FILE_LAYOUT
• Enable the following .config option for the pNFS ds machine:
  • CONFIG_NFS_P2P
• Enable the following .config options for the pNFS server and pNFS ds machines:
  • CONFIG_PNFSD
  • CONFIG_PNFSD_LOCAL_EXPORT
  • CONFIG_PNFSD_P2P
• Install pnfs-nfs-utils on the pNFS server
• Add "pnfs" to the export options of a local filesystem on the pNFS server
• pNFS DS should have nfsd running, but does not need to edit /etc/exports to share files
• pnfsd needs to add "pnfs" export option to /etc/exports
• pnfsd also needs to have "fsid=0" as an export option, otherwise the path walking code will trigger an early UNREGISTER_DS.

Design

REGISTER_DS

• Server
  • Create a new struct pnfs_p2p_client to store information about the adhoc DS:
    • p2p client stateid
    • netid
    • ip address
    • MDS identifier
  • Store structure as part of the nfs4_client
  • Encode p2p client stateid as reply to client
• Client
  • Send REGISTER_DS call as part of nfs4_remote_mount()
    • Use REGISTER_DS_ALL so server knows we'll cache everything
    • Generate MDS identifier using cl_cb_ident and a static u32 counter

UNREGISTER_DS

• Server
  • Check that the nfs4_client has an associated pnfs_p2p_client
  • Check that the nfs4_client is using the correct stateid
  • Free memory allocated for struct pnfs_p2p_client structure during REGISTER_DS
  • Set pnfs_p2p_client pointer in nfs4_client to NULL
• Client
  • Send UNREGISTER_DS as part of nfs4_destroy_server()

LAYOUTGET

• Server
  • Edit pnfs_lexp_layout_get()
  • Set device id field in the layout to the clientid of the machine acting as the DS
  • Encode a filehandle with the DSs MDS ID prepended in filelayout_encode_layout()

• pnfsd
  • Prepend the MDS-ID to filehandles when turning them into a "p2p filehandle" to make conversions easier
  • Send CB_PROXY_REVOKE when client either expires or returns a layout

• p2pds
  • REGISTER_DS_ALL easier to implement
  • Check for delegation before servicing request
  • Store previous proxy open result with the layout, free when responding to a CB_PROXY_REVOKE
  • UNREGISTER_DS during nfs4_kill_server()

DESCRIBE YOUR DESIGN IN THIS SECTION

This section is typically the largest section. Since designs are highly specific, the template cannot provide much in the way of guidelines here. Information which is relevant to the sections below should not be discussed here.

This is the main place where customizing the template for each particular team can really pay off. Teams are encouraged to add a section for the design considerations their own particular area needs to address.

The Design specification describes how the functionality is implemented. Intended readers are:

• Engineering (current and future)
• QA; given this spec, QA should understand the design enough to be able to create white-box type tests for the various parts.

• Overall design
  This document should describe:
  • How it works, in detail.
  • Module breakdown
  • Major data paths through the code. (Referring to the use cases might be useful here)
  • Process structure.
  • Major data structures.
  • Concurrency, parallelism, and mutual exclusion.
  • Class hierarchy, if your design uses object-oriented notions of inheritance and polymorphism. This applies to, but is not limited to, development done in object-oriented languages such as C++ and Java.
  • A UML diagram may be the easiest and most precise way of describing the relationship between the various abstractions supported by your design.
  • Any state machines.
  • Resources used, how they’re controlled, what we do when we run out, recovery steps

• Versioning/compatibility
  • Any v4.1 / pNFS enabled client should be able to make use of adhoc data servers already, and not need special p2p extensions.
  • Clients wishing to act as a data server need CONFIG_NFS_P2P enabled
  • Servers wishing to track adhoc DSs need CONFIG_PNFSD_P2P enabled

• Documentation
  • I can write a Documentation/filesystems/nfs/peer_to_peer.txt file to give a brief overview of how p2pNFS is supposed to work and how users can configure it.
  • I can also copy the page to linux-nfs.org for "online documentation"

Feature Interaction Dependencies and Impacts

• nfsd <-> nfs
  • The machine acting as a pNFS DS needs to be running both the nfs server and the nfs client.

Performance

• Keep a per-file LRU list of clients that currently have the file cached to avoid redirecting all p2p activity to the same client for that file.

Scalability

The hope is that p2p NFS scales to hundreds and thousands of clients better than straight pnfs does. This can be tested by comparing read times for files of varying sizes both with and without p2p enabled. A handful of DSs and a large number of clients should be used to get a feel for how this would work in a data center.

Testing

• Basic proof-of-concept tests
  • 1 client, 1 DS, 1 server
  • Have DS and client rsync files from server
  • Maybe do a `git clone linux-src` instead?
  • Try exporting a /lib partition

• In-depth testing
  • NFSv4root with varying numbers of clients
    • NFSv4root doesn't work right now due to idmapping issues
  • More rsyncs / git clones with more clients

Open Issues

Approvals

Approvers

Reviewers