Client sessions Implementation Issues

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(Add Known Bugs)
(Updated priorities after discussion with Trond Myklebust, Andy Adamson, Alexandros Batsakis, Mike Sager, and Ricardo Labiaga)
Line 2: Line 2:
=== Legend ===
=== Legend ===
-
Note: The labeling still needs to be reviewed by the v4.1 Linux community.
+
* (A) indicates the issue needs to be addressed prior to status change
-
* An (A) indicates the issue needs to be addressed prior to status change
+
* (B) indicates the issue can be deferred after status change
-
* A (B) indicates the issue can be deferred after status change
+
* (C) indicates the issue may not be addressed
-
It's only a first stab at this time.  In the near future we'll probably break the list into two sections: Items needed before the change of status and items needed after change of status.
+
== NFSv4.1 Sessions ==
== NFSv4.1 Sessions ==
=== Backchannel ===
=== Backchannel ===
-
* Duplicate Reply Cache                                   (B)
+
* Duplicate Reply Cache
-
** Not yet implemented
+
** Not yet implemented  (B)
-
** The backchannel currently only implements <i>idempotent</i> operations.
+
*** DRC needs to be implemented before we give the ability to reestablish the backchannel/ connection w/o breaking the session.  (B)
-
* Alternate connection for the backchannel                 (B)
+
** The backchannel currently only implements <i>idempotent</i> operations and operations that can be retried with no side effects.
-
** Not yet implemented
+
** As specified in 2.10.6.1.3, we should cache and return NFs4ERR_TOO_BIG_TO_CACHE - on a reply where <i>cache_this</i> is set to TRUE or NFS4ERR_RETRY_UNCACHED_REP if <i>cache_this</i> is set to FALSE.  This forces server to re-issue the request. Investigate if implementing is faster than returning the errors (A) 
 +
* CB_NOTIFY_LOCK
 +
** Implement Feature (C)
 +
* Alternate connection for the backchannel
 +
** Not yet implemented   (B)
** The backchannel can only currently be bound to the existing forechannel connection.
** The backchannel can only currently be bound to the existing forechannel connection.
** BIND_CONN_TO_SESSION (Separate Connection)              (B)
** BIND_CONN_TO_SESSION (Separate Connection)              (B)
Line 24: Line 27:
* Sequence Flag Processing
* Sequence Flag Processing
** The client does not yet implement the check on the following callback path related flags
** The client does not yet implement the check on the following callback path related flags
-
** SEQ4_STATUS_CB_PATH_DOWN                                (A)
+
** SEQ4_STATUS_CB_PATH_DOWN                                (B)
-
** SEQ4_STATUS_CB_PATH_DOWN_SESSION                        (A)
+
** SEQ4_STATUS_CB_PATH_DOWN_SESSION                        (B)
-
** SEQ4_STATUS_BACKCHANNEL_FAULT                          (A)
+
** SEQ4_STATUS_BACKCHANNEL_FAULT                          (B)
-
** Section 2.10.12.2.4 recommends                          (A)
+
*** What errors are fixable?  This becomes fixable when we add the ability to have multiple connections per session.
-
*** Provide a new connection and bind it to the session when the server indicates the backchannel is down
+
-
[ AB: I have implemented a first version of this, queued up for submission upstream]
+
* Inspect "Referring triples" to detect race with forechannel  
* Inspect "Referring triples" to detect race with forechannel  
** Section 2.10.6.3
** Section 2.10.6.3
-
** Not yet implemented
+
** Not yet implemented (A)
 +
*** Client can mark delegation state for returning  - return OK to delegation recall.  When the open finishes it immediately returns the delegation.
 +
** Later we'll have to do the same thing for layout_get/ layout_return
* Kerberos                                                  (B)
* Kerberos                                                  (B)
-
** Not yet implemented
+
** Not yet implemented   (B)
-
** Need to ensure krb5 forechannel with AUTH_SYS backchannel works  (A)
+
** Need to ensure krb5 forechannel with AUTH_SYS (or possibly AUTH_NULL) backchannel works  (A)
=== Slot Management/ Negotiation ===
=== Slot Management/ Negotiation ===
None of the following items have yet been implemented
None of the following items have yet been implemented
-
* Client CB_RECALL_SLOT (Handles server reducing slots)        (B?)
+
* Client CB_RECALL_SLOT (Handles server reducing slots)        (A)
-
* Server CB_RECALL_SLOT (Reduce slots)                        (B)
+
* Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B)
-
* Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B?)
+
* Define policy to size slot table (startup, congestion, etc)  (C)
-
* Define policy to size slot table (startup, congestion, etc)  (B)
+
* Statistics to monitor (B)                                      
-
* Statistics to monitor                                       (B)
+
* Destroy Session when not in use                              (A-)
-
* Destroy Session when not in use                              (B)
+
* Ensure client checks LEASE TIMEOUT after every clientid exchange (A)
-
* Verify we ask for LEASE TIMEOUT after every clientid exchange (A)
+
** [ Done.  This is being done in nfs4_proc_create_session() ]
 +
* Adjust to correct max-cachesize?            (A)
 +
** Does <i>open</i> require the largest reply size?  Is it sufficient to specify enough bytes for an <i>open</i> reply?
 +
** Do we need to cut down our current size request?
=== Connection Management ===
=== Connection Management ===
* Rebind session to a new connection (after loss of connection)
* Rebind session to a new connection (after loss of connection)
** BIND_CONN_TO_SESSION                                        (B)
** BIND_CONN_TO_SESSION                                        (B)
-
** Not yet impelemented - we currently destroy the session and create a new one
+
** Not yet implemented - we currently destroy the session and create a new one
=== Session Reestablishment ===
=== Session Reestablishment ===
* Need a thorough review of session and state recovery        (A)
* Need a thorough review of session and state recovery        (A)
-
* Need to verify that open state, locks, and delegations survive session reestablishment  (A)
 
=== SessionID Trunking ===
=== SessionID Trunking ===
Line 62: Line 67:
*** Not yet implemented
*** Not yet implemented
** Issue SEQUENCE with existing sessionID?
** Issue SEQUENCE with existing sessionID?
-
** IIRC, the spec states that a SEQUENCE op on a new connection causes the connection to be bound to the specified session
 
-
*** Not yet implemented
 
=== ClientID Trunking ===
=== ClientID Trunking ===
-
* Not yet implemented
+
* Not yet implemented   (B)
=== State Management===
=== State Management===
* State revocation handling
* State revocation handling
** Sequence status bits processing                            (A)
** Sequence status bits processing                            (A)
-
*** Not yet implemented
 
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED                      (B)
-
*** SEQ4_STATUS_EXPIRED_{ALL/SOME}_STATE_REVOKED             (A)
+
*** SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED           
-
**** Propagate error to app
+
**** Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
-
*** SEQ4_STATUS_ADMIN_STATE_REVOKED                          (A)
+
**** Propagate error to app  - SIGLOST (B)
-
*** SEQ4_STATUS_RECALLABLE_STATE_REVOKED                      (A)
+
**** Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
 +
*** SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED              
 +
**** Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
 +
**** Propagate error to app   - SIGLOST (B)
 +
**** Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
 +
*** SEQ4_STATUS_ADMIN_STATE_REVOKED                           
 +
**** Same as above for now (B)
 +
*** SEQ4_STATUS_RECALLABLE_STATE_REVOKED                       
 +
**** Same as above for now (B)
*** SEQ4_STATUS_LEASE_MOVE                                    (B?)
*** SEQ4_STATUS_LEASE_MOVE                                    (B?)
*** SEQ4_STATS_RESTART_RECLAIM_NEEDED                        (A)
*** SEQ4_STATS_RESTART_RECLAIM_NEEDED                        (A)
-
** TEST_STATEID                                              (B?)
+
** TEST_STATEID                                              (B)
-
*** Use to determine status of stateids
+
*** Not yet implemented
 +
** FREE_STATEID                                              (B)
*** Not yet implemented
*** Not yet implemented
-
** FREE_STATEID                                              (B?)
 
-
*** Use to tell server to free stateids after revocation
 
* Verify we use the correct stateid ordering (Section 8.2.4)  (A)
* Verify we use the correct stateid ordering (Section 8.2.4)  (A)
* Ensure Close with most recent stateid (not v4.1 specific)  (A)
* Ensure Close with most recent stateid (not v4.1 specific)  (A)
-
* Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2)  (B?)
+
* Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2)  (B)
* Verify locks and delegations survive session reestablishment (A)
* Verify locks and delegations survive session reestablishment (A)
=== State Reclaim ===
=== State Reclaim ===
-
* Wait for outstanding RPCs (Section 8.4.2.1)                (B)
+
* RECLAIM_COMPLETE                                            (A)
-
* LOCK with RECLAIM                                          (A?)
+
** Not yet implemented
-
* OPEN with CLAIM_PREVIOUS                                    (B?)
+
*** Issue after establishing a new clientid even if the server didn't reboot
-
* RECLAIM_COMPLETE                                            (B?)
+
*** Code server or pyNFS server to accept request 
-
* Lock recovery when eir_server_owner is different (Section 8.4.2.1)  (B?)
+
*** Update wireshark to understand new OP                      (B)
 +
* Lock recovery when eir_server_owner is different (Section 8.4.2.1)  (B)
 +
** Only needed for migration???
** Verify client attempts lock recovery when eir_server_scope is same
** Verify client attempts lock recovery when eir_server_scope is same
Line 105: Line 116:
=== Error Handling Review ===
=== Error Handling Review ===
-
* Thorough error handling inspection and testing            (A)
+
* Error mapping problems
-
 
+
** Code Inspection (A)
 +
** Change the place in the kernel code where we map the errors?  (B)
 +
*** Is it doable?  Are we mixing RPC errors and NFS errors?
 +
** Code inspection  (A)
 +
* pyNFS server changes to accommodate returning random errors  (A)
 +
** Framework  (A)
 +
** Operations  (A)
 +
* pyNFS regression tests for ongoing development    (B)
=== COMPOUND and CB_COMPOUND ===
=== COMPOUND and CB_COMPOUND ===
Line 113: Line 131:
*** Known problem where max sizes does not allow for compound operation header
*** Known problem where max sizes does not allow for compound operation header
*** Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4)  (B)
*** Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4)  (B)
 +
* Mount negotiation
 +
** Verify server allowed values in CREATE_SESSION is reasonable for us to proceed
 +
** Client requests reasonable values, then checks for the bare minimum (A)
 +
 +
=== Minor Version Negotiation ===
 +
* Drop down to lower version if failed v4.1 mount        (B)
 +
* Pass something to user-land specifying the error      (A)
 +
** [ Done.  EPROTONOSUPPORT is already being returned to user-land ]
 +
 +
== Misc Functionality ==
 +
 +
* File Delegations
 +
** OPEN4_SHARE_ACCESS_WANT_NO_DELEG                          (C)
 +
*** Define Policy/ Implications/ When to use each kind?      (C)
 +
** CLAIM_DELEG_CUR_FH                                (C)
 +
** WANT_DELEGATION                                        (C)
 +
** CB_PUSH_DELEG (C)
 +
** CB_RECALLABLE_OBJ_AVAIL (C)
 +
** CB_GETATTR (A)
 +
*** [ done.  Checked the spec and CB_GETATTR is the same for 4.0 and 4.1. ]
 +
** CB_NOTIFY (B)
 +
** CB_RECALL_ANY (A)
 +
*** Can we track the number of processes that have the delegation so we don't return the most "popular" one.    (B)
 +
 +
* Directory Delegations
 +
** Do they really buy us anything investigation (C)
 +
** GET_DIR_DELEGATION (C)
 +
 +
* Security, Kerberos and RPCSEC_GSS
 +
** SECINFO (B)
 +
*** Necessary for migration in the future
 +
** SECINFO_NO_NAME  (B)
 +
 +
* Implementation ID                                        (B)
 +
[ AB: I have implemented a first version of this, queued up for submission upstream]
 +
 +
* Named Attributes                                          (C)
 +
** Not yet supported in v4 either
 +
 +
* Persistent Session
 +
** OPEN - EXCLUSIVE4_1  (A)
 +
** GUARDED (if persistent session)      (A)
 +
** Check for Persistent Session flag during CREATE_SESSION response (A)
 +
       
 +
* OPEN4_RESULT_PRESERVE_UNLINKED (B)
 +
 +
* Lock Notification
 +
** OPEN4_RESULT_MAY_NOTIFY_LOCK (open flag)  (B)
 +
** CB_NOTIFY_LOCK  (B)
 +
 +
* ACL changes: dacl, sacl, inheritance  (C)
-
== Known Bugs ==
+
* Data Retention                                          (C)
-
Should file bugzillas for these and refer the BZ# here instead
+
-
#When trying to umount a server that is down and the process is interrupted with a CNTL-C leads to an oops
+
-
#*This happens in NFSv4.1 due to destroy_session / sequence sync issues.
+
-
#** Alexandros sent a patch that Trond wants to be revisited
+
-
# If rsize, wsize are specified at mount time:
+
-
#* the request_sz, response_sz take the wsize, rsize values incorrectly
+
-
#* options don't propagate to CREATE_SESSION with DSs (pNFS)
+
-
#* Alexandros will be submitting a patch soon
+
-
# If backchannel connection timeouts, upon re-establishment CB_SEQUENCE is out of order
+
-
# Mount two different file systems from the same servers
+
-
#* Does not reuse the nfs_client structure therefore failing to reuse the existing session
+
-
#* Alexandros: (related) mount nfs41_server:/exp1 /mnt, ..., mount nfs41_server:/exp2 /mnt2. New session overrides old
+
-
# The spec says that the program version number for the backchannel must be set to 4 (!= v4.0 that the spec doesn't say anything AFICS). Currenly we set it to 1. What about NFSv4.0 ?
+
-
# Sequence Flooding: the NFSv4.1 client keeps queuing SEQUENCE operations when the server is down (and eventually we run out of slots). Same for NFSv4.0 and RENEWs (minus the slot issue)
+
-
#*Alexandros working on a patch
+

Revision as of 08:11, 3 December 2009

The client forechannel and backchannel functionality has been integrated into Linux-2.6.31. The server forechannel functionality has been integrated into Linux-2.6.30. The server backchannel functionality has been integrated into Linux-2.6.32. This document highlights functionality that is not yet fully implemented, not fully tested, or which status needs to be checked. It also provides a list of known issues/ bugs. This list of issues and pending functionality should be addressed before the NFS v4.1 client can be changed from Developer to Experimental. This change will allow distros to more comfortably include the functionality in their releases.

Contents

Legend

  • (A) indicates the issue needs to be addressed prior to status change
  • (B) indicates the issue can be deferred after status change
  • (C) indicates the issue may not be addressed

NFSv4.1 Sessions

Backchannel

  • Duplicate Reply Cache
    • Not yet implemented (B)
      • DRC needs to be implemented before we give the ability to reestablish the backchannel/ connection w/o breaking the session. (B)
    • The backchannel currently only implements idempotent operations and operations that can be retried with no side effects.
    • As specified in 2.10.6.1.3, we should cache and return NFs4ERR_TOO_BIG_TO_CACHE - on a reply where cache_this is set to TRUE or NFS4ERR_RETRY_UNCACHED_REP if cache_this is set to FALSE. This forces server to re-issue the request. Investigate if implementing is faster than returning the errors (A)
  • CB_NOTIFY_LOCK
    • Implement Feature (C)
  • Alternate connection for the backchannel
    • Not yet implemented (B)
    • The backchannel can only currently be bound to the existing forechannel connection.
    • BIND_CONN_TO_SESSION (Separate Connection) (B)
      • Not yet implemented.
      • The workaround is for the client to destroy and create a new session to reestablish the backchannel.
    • BACKCHANNEL_CTL (B)
      • Not yet implemented
      • Provide alternate Backchannel program number
      • Provide Kerberos (not yet supported) Principals for Backchannel
  • Sequence Flag Processing
    • The client does not yet implement the check on the following callback path related flags
    • SEQ4_STATUS_CB_PATH_DOWN (B)
    • SEQ4_STATUS_CB_PATH_DOWN_SESSION (B)
    • SEQ4_STATUS_BACKCHANNEL_FAULT (B)
      • What errors are fixable? This becomes fixable when we add the ability to have multiple connections per session.
  • Inspect "Referring triples" to detect race with forechannel
    • Section 2.10.6.3
    • Not yet implemented (A)
      • Client can mark delegation state for returning - return OK to delegation recall. When the open finishes it immediately returns the delegation.
    • Later we'll have to do the same thing for layout_get/ layout_return
  • Kerberos (B)
    • Not yet implemented (B)
    • Need to ensure krb5 forechannel with AUTH_SYS (or possibly AUTH_NULL) backchannel works (A)

Slot Management/ Negotiation

None of the following items have yet been implemented

  • Client CB_RECALL_SLOT (Handles server reducing slots) (A)
  • Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B)
  • Define policy to size slot table (startup, congestion, etc) (C)
  • Statistics to monitor (B)
  • Destroy Session when not in use (A-)
  • Ensure client checks LEASE TIMEOUT after every clientid exchange (A)
    • [ Done. This is being done in nfs4_proc_create_session() ]
  • Adjust to correct max-cachesize? (A)
    • Does open require the largest reply size? Is it sufficient to specify enough bytes for an open reply?
    • Do we need to cut down our current size request?

Connection Management

  • Rebind session to a new connection (after loss of connection)
    • BIND_CONN_TO_SESSION (B)
    • Not yet implemented - we currently destroy the session and create a new one

Session Reestablishment

  • Need a thorough review of session and state recovery (A)

SessionID Trunking

Increases the I/O pipe and the number of slots

  • Bind a new connection to an existing session (B)
    • BIND_CONN_TO_SESSION (B)
      • Not yet implemented
    • Issue SEQUENCE with existing sessionID?

ClientID Trunking

  • Not yet implemented (B)

State Management

  • State revocation handling
    • Sequence status bits processing (A)
      • SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING (B)
      • SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED (B)
      • SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED
        • Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
        • Propagate error to app - SIGLOST (B)
        • Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
      • SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED
        • Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
        • Propagate error to app - SIGLOST (B)
        • Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
      • SEQ4_STATUS_ADMIN_STATE_REVOKED
        • Same as above for now (B)
      • SEQ4_STATUS_RECALLABLE_STATE_REVOKED
        • Same as above for now (B)
      • SEQ4_STATUS_LEASE_MOVE (B?)
      • SEQ4_STATS_RESTART_RECLAIM_NEEDED (A)
    • TEST_STATEID (B)
      • Not yet implemented
    • FREE_STATEID (B)
      • Not yet implemented
  • Verify we use the correct stateid ordering (Section 8.2.4) (A)
  • Ensure Close with most recent stateid (not v4.1 specific) (A)
  • Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2) (B)
  • Verify locks and delegations survive session reestablishment (A)

State Reclaim

  • RECLAIM_COMPLETE (A)
    • Not yet implemented
      • Issue after establishing a new clientid even if the server didn't reboot
      • Code server or pyNFS server to accept request
      • Update wireshark to understand new OP (B)
  • Lock recovery when eir_server_owner is different (Section 8.4.2.1) (B)
    • Only needed for migration???
    • Verify client attempts lock recovery when eir_server_scope is same

State Protection

  • SSV Support (for trunking and reconnection) (B)
    • SET_SSV
    • GET_SSV
  • Mach creds (B)

Error Handling Review

  • Error mapping problems
    • Code Inspection (A)
    • Change the place in the kernel code where we map the errors? (B)
      • Is it doable? Are we mixing RPC errors and NFS errors?
    • Code inspection (A)
  • pyNFS server changes to accommodate returning random errors (A)
    • Framework (A)
    • Operations (A)
  • pyNFS regression tests for ongoing development (B)

COMPOUND and CB_COMPOUND

  • Correct use of max sizes (A)
    • Client should take care to use correct request and response max sizes
      • Known problem where max sizes does not allow for compound operation header
      • Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4) (B)
  • Mount negotiation
    • Verify server allowed values in CREATE_SESSION is reasonable for us to proceed
    • Client requests reasonable values, then checks for the bare minimum (A)

Minor Version Negotiation

  • Drop down to lower version if failed v4.1 mount (B)
  • Pass something to user-land specifying the error (A)
    • [ Done. EPROTONOSUPPORT is already being returned to user-land ]

Misc Functionality

  • File Delegations
    • OPEN4_SHARE_ACCESS_WANT_NO_DELEG (C)
      • Define Policy/ Implications/ When to use each kind? (C)
    • CLAIM_DELEG_CUR_FH (C)
    • WANT_DELEGATION (C)
    • CB_PUSH_DELEG (C)
    • CB_RECALLABLE_OBJ_AVAIL (C)
    • CB_GETATTR (A)
      • [ done. Checked the spec and CB_GETATTR is the same for 4.0 and 4.1. ]
    • CB_NOTIFY (B)
    • CB_RECALL_ANY (A)
      • Can we track the number of processes that have the delegation so we don't return the most "popular" one. (B)
  • Directory Delegations
    • Do they really buy us anything investigation (C)
    • GET_DIR_DELEGATION (C)
  • Security, Kerberos and RPCSEC_GSS
    • SECINFO (B)
      • Necessary for migration in the future
    • SECINFO_NO_NAME (B)
  • Implementation ID (B)

[ AB: I have implemented a first version of this, queued up for submission upstream]

  • Named Attributes (C)
    • Not yet supported in v4 either
  • Persistent Session
    • OPEN - EXCLUSIVE4_1 (A)
    • GUARDED (if persistent session) (A)
    • Check for Persistent Session flag during CREATE_SESSION response (A)
  • OPEN4_RESULT_PRESERVE_UNLINKED (B)
  • Lock Notification
    • OPEN4_RESULT_MAY_NOTIFY_LOCK (open flag) (B)
    • CB_NOTIFY_LOCK (B)
  • ACL changes: dacl, sacl, inheritance (C)
  • Data Retention (C)
Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/Client_sessions_Implementation_Issues"
Personal tools