Testing forecast roadmap

From Linux NFS

Latest revision as of 22:52, 4 October 2005

Linux NFSv4 Testing Roadmap

As of May 2005, we've completed a detailed itemization of testing work necessary for NFSv4, in a spreadsheet "Test Matrix", identifying items as High, Medium, and Low.

This document seeks to divide the high priority work into achievable chunks that we as a community can strive to attain.

I tried to prioritize for things that look reasonably easy to complete in the near term, that have owners, and/or are most likely to reveal issues that developers want to know about.

Milestone 0: Developer Testing Support

These testing tasks seek to put into place tools and practices that will help support development activities (including testing) on an ongoing basis.

Functionality:

I.B.1   [X]  [OSDL]  Automate running applicable existing functionality
                     tests on codebase periodically:  cthon, pynfs,
                     iozone
I.B.3   [X]  [OSDL]  Automate cross-compile testing for major platforms
                     on each NFSv4 patchset
I.C.2   [ ]  [    ]  Test install on Fedora Core of NFSv4 server &
                     client, krb5, ldap, et al
I.E.9   [ ]  [    ]  Verify ability to force operations (close files,
                     change states, unmount)
I.G.5   [ ]  [Bull]  Test compilation and functionality on IA-64
I.G.7   [ ]  [Bull]  Test compilation and functionality on PPC-64
I.K.2   [ ]  [    ]  Check that NFS HOWTO is updated with sufficient
                     nfsv4 info

Interop:

II.A.1  [ ]  [Bull]  Kerberos (MIT) - verify basic functionality.  Mount
                     w/ krb5, etc.

Robustness:

III.A.3 [ ]  [OSDL]  Run NFS server for 2 wks with random configuration
                     changes
III.C.1 [ ]  [Bull]  Run LTP NFS fstress in a std config on each release

Performance:

IV.A.1  [ ]  [Bull]  Time to perform sequence of unique read/write
                     operations
IV.A.2  [ ]  [Bull]  Time to perform sequence of cacheable read/write
                     operations
IV.F.4  [ ]  [Bull]  Measure memory/network/CPU efficiency of client for
                     fixed workload
IV.G.3  [ ]  [Bull]  Measure effects of scaling up number of
                     connections with exportfs/mount.  Measure number of 
                     mounts per second on client and server.
IV.J.2  [ ]  [Bull]  Measure performance when operating with Kerberos 5
                     integrity and privacy

Security:

V.A.6   [ ]  [    ]  Audit the mountd code
V.B.4   [ ]  [OSDL]  Run sparse as regression test periodically
V.L.1   [ ]  [    ]  Ensure there is high level design documentation of
                     NFSv4 security
V.L.2   [ ]  [    ]  Ensure there is inline documentation for security
                     related code in kernel
V.L.3   [ ]  [    ]  Ensure there is inline documentation for GSS API
                     (libgssapi, librpcsecgss)

Misc:

*       [X]  [OSDL]  Establish bug tracking process for NFSv4

Milestone 1: Distro Quality Assurance

This milestone strives to attain a "Top Ten" list of testing that is felt will emphasize that NFSv4 is stable enough to be included in all enterprise distros. The focus is robustness, and items that would cause distros the most headaches if they're busted.

I.A.5   [ ]  [    ]  Protocol implementation interoperability between
                     Linux server and client
I.C.5   [ ]  [    ]  Test install on SLES of NFSv4 server & client,
                     krb5, ldap, et al
I.C.6   [ ]  [    ]  Test install on RHEL of NFSv4 server & client,
                     krb5, ldap, et al
I.H.1   [ ]  [    ]  Verify compatibility with glibc
I.H.9   [ ]  [ IBM]  Verify compatibility with automounter
I.H.11  [ ]  [    ]  Verify compatibility with basic system tools
                     (file utils, core utils, util-linux, mount, sar,
                     iostat, etc.)
I.K.4   [ ]  [    ]  Check that Network Admin Guide at tldp.org is
                     updated for nfsv4
II.E.2  [ ]  [    ]  Verify features of the Ext3 file system work under
                     NFSv4
III.A.1 [ ]  [Bull]  Run iozone for 2 wks on basic client/server
                     operations
III.A.2 [ ]  [    ]  Run automounter use case for 2 wks on amd, autofs,
                     and autong
III.A.4 [ ]  [    ]  Run connectathon locking tests against NFS server
                    for 2 weeks
III.E.1 [ ]  [OSDL]  Test short & long term local network failure
                     (unplugged cable, ifdown eth0, etc.)
III.E.2 [ ]  [OSDL]  Test short & long duration remote network partition
III.E.3 [ ]  [OSDL]  Test behavior during crash/reboot of server with
                     clients holding various states
V.D.1   [ ]  [    ]  Review Authentication/ACL feature design
V.E.1   [ ]  [    ]  Ensure a functionality test sufficiently tests
                     Authentication/ACL

Milestone 2: Drill Down for One Use Case

This milestone strives to push NFSv4 through one complete use case, to give it some thorough exercise and scaling, and to establish a "case study" we can use as an example of NFSv4's value.

I.E.5   [ ]  [    ]  Verify sufficient monitoring functionality within
                     Ethereal
I.F.1   [ ]  [    ]  Client notification to server of locking, write,
                     read, etc.
I.P.1   [ ]  [OSDL]  Database functionality on NFS
I.L.1   [ ]  [    ]  Test compatibility with TCP protocol
IV.A.7  [ ]  [    ]  NFSv3 vs. v4:  Metadata - open/close intensive
                     workload
IV.A.8  [ ]  [    ]  NFSv3 vs. v4:  Metadata - directory scanning
IV.A.9  [ ]  [    ]  NFSv3 vs. v4:  Metadata - create/delete
IV.A.10 [ ]  [    ]  NFSv3 vs. v4:  Metadata - changing attributes
                     (chown, chmod) while dir scanning
IV.E.1  [ ]  [Bull]  Test performance with large numbers of small (<4k)
                     files
IV.E.2  [ ]  [Bull]  Test performance with a few very large (>1G) files
IV.F.4  [ ]  [Bull]  Measure memory/network/CPU efficiency of client
                     for fixed workload
IV.G.1  [ ]  [    ]  Verify server scalability with clients generating
                     various basic requests (ACCESS, GETATTR, et al)
IV.G.2  [ ]  [    ]  Verify server scalability with clients using
                     compound requests
V.D.6   [ ]  [    ]  Penetration testing for client callback
                     implementation
V.E.6   [ ]  [    ]  Ensure a functionality test sufficiently covers
                     penetration testing for client callback
                     implementation

Milestone 3: Enterprise Usability

This milestone focuses on testing that verifies "basic" usability of NFSv4 in data centers. It strives to establish that NFSv4 is able to perform basic functions in enterprise-class deployments.

I.F.2   [ ]  [    ]  Reboot recovery
I.F.4   [ ]  [    ]  Open with shares / deny
I.F.5   [ ]  [    ]  Bumping a sequence ID
I.F.6   [ ]  [    ]  Network partition recovery
I.F.8   [ ]  [    ]  Locking  (Non-blocking locks)
I.H.3   [ ]  [    ]  Verify compatibility with Kerberos
I.H.4   [ ]  [    ]  Verify compatibility with Ipsec
I.H.5   [ ]  [    ]  Verify compatibility with POSIX ACLs
I.H.6   [ ]  [    ]  Verify compatibility with NFS ACLs
II.B.1  [ ]  [Bull]  Interoperability between 32-bit and 64-bit for
                     client and server:  IA32/AIX-PPC, IA32/Linux-PPC
II.B.2  [ ]  [Bull]  Interoperability between little endian and big
                     endian
II.C.1  [ ]  [Bull]  Interoperability for Linux IA-32 client ? Solaris
                     10 server
II.C.4  [ ]  [Bull]  Interoperability for Linux IA-32 client ? AIX 5.3
                     server
III.D.7 [ ]  [    ]  Test correctness of NFS client when backed by a
                     large (>100GB) cachefs
IV.E.5  [ ]  [    ]  NFS "Cluster" scenario with 1000+ clients and
                     several servers
V.A.7   [ ]  [    ]  Audit the RPC authentication code (gssd, authsys,
                     etc.)
V.D.2   [ ]  [    ]  Review each security flavor feature design:
                     Krb5, Spkm
V.E.2   [ ]  [    ]  Ensure a functionality test sufficiently covers
                     each security flavor

Milestone 4: Enterprise Readiness

This milestone builds on the previous one by ensuring that NFSv4 will behave well and perform robustly under typical enterprise-class loads.

III.B.2  [ ]  [    ]  Test stability of client in out of memory
                      situation
III.B.3  [ ]  [    ]  Test stability of client in out of disk space on
                      server situation
III.B.4  [ ]  [    ]  Test stability of client in out of inode situation
III.B.5  [ ]  [    ]  Test stability of client in out of swap space
                      situation
III.B.6  [ ]  [    ]  Test stability of server in out of pid situation
III.B.7  [ ]  [    ]  Test stability of server in out of memory
                      situation
III.B.8  [ ]  [    ]  Test stability of server in out of disk space
                      situation
III.B.9  [ ]  [    ]  Test stability of server in out of inode situation
III.B.10 [ ]  [    ]  Test stability of server in out of swap space
                      situation
III.D.1  [ ]  [    ]  Find maximum number of connections to Linux IA-32
                      server
III.D.4  [ ]  [Bull]  Find maximum number of mounted file systems on
                      client
III.E.4  [ ]  [    ]  Test multiple clients using, locking, etc. same
                      files
III.E.5  [ ]  [    ]  Test behavior of server with failed storage device
III.E.6  [ ]  [    ]  Test behavior during crash of client with open
                      delegations and locks
III.E.7  [ ]  [    ]  Test recovery from denied permission
III.E.8  [ ]  [    ]  Test recovery from JUKEBOX/DELAY
III.E.9  [ ]  [    ]  Test recovery from ESTALE
III.E.10 [ ]  [    ]  Test server callback mechanism (c.f. III.E.1,
                      III.E.2)
III.G.2  [ ]  [    ]  Test concurrent access tests for races in
                      automounter
IV.G.4   [ ]  [Bull]  Measure effects of increasing number of files
IV.G.5   [ ]  [Bull]  Measure effects of increasing file size
                      (with/without cache)
IV.G.6   [ ]  [Bull]  Measure effects when increasing size of
                      on-the-wire NFS read or write operations
V.J.1    [ ]  [Bull]  Identify security issues assuming attack from
                      client-side