Client sessions Implementation Issues

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(Initial description of open issues in the NFSv4.1 client sessions implementation)
 
(2 intermediate revisions not shown)
Line 1: Line 1:
-
The client forechannel and backchannel functionality has been integrated into Linux-2.6.31.  The server forechannel functionality has been integrated into Linux-2.6.30. The server backchannel functionality has been integrated into Linux-2.6.32. This document highlights functionality that is not yet fully implemented, not fully tested, or which status needs to be checked.  It also provides a list of known issues/ bugs.  This list of issues and pending functionality should be addressed before the NFS v4.1 client can be changed from <i>Developer</i> to <i>Experimental</i>.  This change will allow distros to more comfortably include the functionality in their releases.
+
The client forechannel and backchannel functionality is included as of kernel version 2.6.31.  However, the version in 2.6.31 deviates from [http://tools.ietf.org/search/rfc5661 rfc 5661] in several important ways and is considered useful for developers only.
 +
 
 +
This document lists known issues in that initial implementation, including those which had to be addressed before the NFSv4.1 client could be changed from <i>Developer Only</i> to <i>Experimental</i>, allowing distros to more comfortably include the functionality in their releases.
 +
 
 +
The issues labelled (A) below have since been addressed, and the NFSv4.1 client is labelled "Experimental" as of 2.6.36.
=== Legend ===
=== Legend ===
-
Note: The labeling still needs to be reviewed by the v4.1 Linux community.
+
* (A) indicates the issue needs to be addressed prior to status change
-
* An (A) indicates the issue needs to be addressed prior to status change
+
* (B) indicates the issue can be deferred after status change
-
* A (B) indicates the issue can be deferred after status change
+
* (C) indicates the issue may not be addressed
-
It's only a first stab at this time.  In the near future we'll probably break the list into two sections: Items needed before the change of status and items needed after change of status.
+
== NFSv4.1 Sessions ==
== NFSv4.1 Sessions ==
=== Backchannel ===
=== Backchannel ===
-
* Duplicate Reply Cache                                   (B)
+
* Duplicate Reply Cache
-
** Not yet implemented
+
** Not yet implemented  (B)
-
** The backchannel currently only implements <i>idempotent</i> operations.
+
*** DRC needs to be implemented before we give the ability to reestablish the backchannel/ connection w/o breaking the session.  (B)
-
* Alternate connection for the backchannel                 (B)
+
** The backchannel currently only implements <i>idempotent</i> operations and operations that can be retried with no side effects.
-
** Not yet implemented
+
** As specified in 2.10.6.1.3, we should cache and return NFs4ERR_TOO_BIG_TO_CACHE - on a reply where <i>cache_this</i> is set to TRUE or NFS4ERR_RETRY_UNCACHED_REP if <i>cache_this</i> is set to FALSE.  This forces server to re-issue the request. Investigate if implementing is faster than returning the errors (A) 
 +
* CB_NOTIFY_LOCK
 +
** Implement Feature (C)
 +
* Alternate connection for the backchannel
 +
** Not yet implemented   (B)
** The backchannel can only currently be bound to the existing forechannel connection.
** The backchannel can only currently be bound to the existing forechannel connection.
** BIND_CONN_TO_SESSION (Separate Connection)              (B)
** BIND_CONN_TO_SESSION (Separate Connection)              (B)
Line 24: Line 31:
* Sequence Flag Processing
* Sequence Flag Processing
** The client does not yet implement the check on the following callback path related flags
** The client does not yet implement the check on the following callback path related flags
-
** SEQ4_STATUS_CB_PATH_DOWN                                (A)
+
** SEQ4_STATUS_CB_PATH_DOWN                                (B)
-
** SEQ4_STATUS_CB_PATH_DOWN_SESSION                        (A)
+
** SEQ4_STATUS_CB_PATH_DOWN_SESSION                        (B)
-
** SEQ4_STATUS_BACKCHANNEL_FAULT                          (A)
+
** SEQ4_STATUS_BACKCHANNEL_FAULT                          (B)
-
** Section 2.10.12.2.4 recommends                          (A)
+
*** What errors are fixable?  This becomes fixable when we add the ability to have multiple connections per session.
-
*** Provide a new connection and bind it to the session when the server indicates the backchannel is down
+
-
[ AB: I have implemented a first version of this, queued up for submission upstream]
+
* Inspect "Referring triples" to detect race with forechannel  
* Inspect "Referring triples" to detect race with forechannel  
** Section 2.10.6.3
** Section 2.10.6.3
-
** Not yet implemented
+
** Not yet implemented (A)
 +
*** Client can mark delegation state for returning  - return OK to delegation recall.  When the open finishes it immediately returns the delegation.
 +
** Later we'll have to do the same thing for layout_get/ layout_return
* Kerberos                                                  (B)
* Kerberos                                                  (B)
-
** Not yet implemented
+
** Not yet implemented   (B)
-
** Need to ensure krb5 forechannel with AUTH_SYS backchannel works  (A)
+
** Need to ensure krb5 forechannel with AUTH_SYS (or possibly AUTH_NULL) backchannel works  (A)
=== Slot Management/ Negotiation ===
=== Slot Management/ Negotiation ===
None of the following items have yet been implemented
None of the following items have yet been implemented
-
* Client CB_RECALL_SLOT (Handles server reducing slots)        (B?)
+
* Client CB_RECALL_SLOT (Handles server reducing slots)        (A)
-
* Server CB_RECALL_SLOT (Reduce slots)                        (B)
+
* Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B)
-
* Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B?)
+
* Define policy to size slot table (startup, congestion, etc)  (C)
-
* Define policy to size slot table (startup, congestion, etc)  (B)
+
* Statistics to monitor (B)                                      
-
* Statistics to monitor                                       (B)
+
* Destroy Session when not in use                              (A-)
-
* Destroy Session when not in use                              (B)
+
* Ensure client checks LEASE TIMEOUT after every clientid exchange (A)
-
* Verify we ask for LEASE TIMEOUT after every clientid exchange (A)
+
** [ Done.  This is being done in nfs4_proc_create_session() ]
 +
* Adjust to correct max-cachesize?            (A)
 +
** Does <i>open</i> require the largest reply size?  Is it sufficient to specify enough bytes for an <i>open</i> reply?
 +
** Do we need to cut down our current size request?
=== Connection Management ===
=== Connection Management ===
* Rebind session to a new connection (after loss of connection)
* Rebind session to a new connection (after loss of connection)
** BIND_CONN_TO_SESSION                                        (B)
** BIND_CONN_TO_SESSION                                        (B)
-
** Not yet impelemented - we currently destroy the session and create a new one
+
** Not yet implemented - we currently destroy the session and create a new one
=== Session Reestablishment ===
=== Session Reestablishment ===
* Need a thorough review of session and state recovery        (A)
* Need a thorough review of session and state recovery        (A)
-
* Need to verify that open state, locks, and delegations survive session reestablishment  (A)
 
=== SessionID Trunking ===
=== SessionID Trunking ===
Line 62: Line 71:
*** Not yet implemented
*** Not yet implemented
** Issue SEQUENCE with existing sessionID?
** Issue SEQUENCE with existing sessionID?
-
** IIRC, the spec states that a SEQUENCE op on a new connection causes the connection to be bound to the specified session
 
-
*** Not yet implemented
 
=== ClientID Trunking ===
=== ClientID Trunking ===
-
* Not yet implemented
+
* Not yet implemented   (B)
=== State Management===
=== State Management===
* State revocation handling
* State revocation handling
** Sequence status bits processing                            (A)
** Sequence status bits processing                            (A)
-
*** Not yet implemented
 
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED                      (B)
*** SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED                      (B)
-
*** SEQ4_STATUS_EXPIRED_{ALL/SOME}_STATE_REVOKED             (A)
+
*** SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED           
-
**** Propagate error to app
+
**** Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
-
*** SEQ4_STATUS_ADMIN_STATE_REVOKED                          (A)
+
**** Propagate error to app  - SIGLOST (B)
-
*** SEQ4_STATUS_RECALLABLE_STATE_REVOKED                      (A)
+
**** Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
 +
*** SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED              
 +
**** Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
 +
**** Propagate error to app   - SIGLOST (B)
 +
**** Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
 +
*** SEQ4_STATUS_ADMIN_STATE_REVOKED                           
 +
**** Same as above for now (B)
 +
*** SEQ4_STATUS_RECALLABLE_STATE_REVOKED                       
 +
**** Same as above for now (B)
*** SEQ4_STATUS_LEASE_MOVE                                    (B?)
*** SEQ4_STATUS_LEASE_MOVE                                    (B?)
*** SEQ4_STATS_RESTART_RECLAIM_NEEDED                        (A)
*** SEQ4_STATS_RESTART_RECLAIM_NEEDED                        (A)
-
** TEST_STATEID                                              (B?)
+
** TEST_STATEID                                              (B)
-
*** Use to determine status of stateids
+
*** Not yet implemented
 +
** FREE_STATEID                                              (B)
*** Not yet implemented
*** Not yet implemented
-
** FREE_STATEID                                              (B?)
 
-
*** Use to tell server to free stateids after revocation
 
* Verify we use the correct stateid ordering (Section 8.2.4)  (A)
* Verify we use the correct stateid ordering (Section 8.2.4)  (A)
* Ensure Close with most recent stateid (not v4.1 specific)  (A)
* Ensure Close with most recent stateid (not v4.1 specific)  (A)
-
* Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2)  (B?)
+
* Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2)  (B)
* Verify locks and delegations survive session reestablishment (A)
* Verify locks and delegations survive session reestablishment (A)
=== State Reclaim ===
=== State Reclaim ===
-
* Wait for outstanding RPCs (Section 8.4.2.1)                (B)
+
* RECLAIM_COMPLETE                                            (A)
-
* LOCK with RECLAIM                                          (A?)
+
** Not yet implemented
-
* OPEN with CLAIM_PREVIOUS                                    (B?)
+
*** Issue after establishing a new clientid even if the server didn't reboot
-
* RECLAIM_COMPLETE                                            (B?)
+
*** Code server or pyNFS server to accept request 
-
* Lock recovery when eir_server_owner is different (Section 8.4.2.1)  (B?)
+
*** Update wireshark to understand new OP                      (B)
 +
* Lock recovery when eir_server_owner is different (Section 8.4.2.1)  (B)
 +
** Only needed for migration???
** Verify client attempts lock recovery when eir_server_scope is same
** Verify client attempts lock recovery when eir_server_scope is same
Line 105: Line 120:
=== Error Handling Review ===
=== Error Handling Review ===
-
* Thorough error handling inspection and testing            (A)
+
* Error mapping problems
-
 
+
** Code Inspection (A)
 +
** Change the place in the kernel code where we map the errors?  (B)
 +
*** Is it doable?  Are we mixing RPC errors and NFS errors?
 +
** Code inspection  (A)
 +
* pyNFS server changes to accommodate returning random errors  (A)
 +
** Framework  (A)
 +
** Operations  (A)
 +
* pyNFS regression tests for ongoing development    (B)
=== COMPOUND and CB_COMPOUND ===
=== COMPOUND and CB_COMPOUND ===
Line 113: Line 135:
*** Known problem where max sizes does not allow for compound operation header
*** Known problem where max sizes does not allow for compound operation header
*** Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4)  (B)
*** Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4)  (B)
 +
* Mount negotiation
 +
** Verify server allowed values in CREATE_SESSION is reasonable for us to proceed
 +
** Client requests reasonable values, then checks for the bare minimum (A)
 +
 +
=== Minor Version Negotiation ===
 +
* Drop down to lower version if failed v4.1 mount        (B)
 +
* Pass something to user-land specifying the error      (A)
 +
** [ Done.  EPROTONOSUPPORT is already being returned to user-land ]
 +
 +
== Misc Functionality ==
 +
 +
* File Delegations
 +
** OPEN4_SHARE_ACCESS_WANT_NO_DELEG                          (C)
 +
*** Define Policy/ Implications/ When to use each kind?      (C)
 +
** CLAIM_DELEG_CUR_FH                                (C)
 +
** WANT_DELEGATION                                        (C)
 +
** CB_PUSH_DELEG (C)
 +
** CB_RECALLABLE_OBJ_AVAIL (C)
 +
** CB_GETATTR (A)
 +
*** [ done.  Checked the spec and CB_GETATTR is the same for 4.0 and 4.1. ]
 +
** CB_NOTIFY (B)
 +
** CB_RECALL_ANY (A)
 +
*** Can we track the number of processes that have the delegation so we don't return the most "popular" one.    (B)
 +
 +
* Directory Delegations
 +
** Do they really buy us anything investigation (C)
 +
** GET_DIR_DELEGATION (C)
 +
 +
* Security, Kerberos and RPCSEC_GSS
 +
** SECINFO (B)
 +
*** Necessary for migration in the future
 +
** SECINFO_NO_NAME  (B)
 +
 +
* Implementation ID                                        (B)
 +
[ AB: I have implemented a first version of this, queued up for submission upstream]
 +
 +
* Named Attributes                                          (C)
 +
** Not yet supported in v4 either
 +
 +
* Persistent Session
 +
** OPEN - EXCLUSIVE4_1  (A)
 +
** GUARDED (if persistent session)      (A)
 +
** Check for Persistent Session flag during CREATE_SESSION response (A)
 +
       
 +
* OPEN4_RESULT_PRESERVE_UNLINKED (B)
 +
 +
* Lock Notification
 +
** OPEN4_RESULT_MAY_NOTIFY_LOCK (open flag)  (B)
 +
** CB_NOTIFY_LOCK  (B)
 +
 +
* ACL changes: dacl, sacl, inheritance  (C)
 +
 +
* Data Retention                                          (C)

Latest revision as of 23:51, 18 December 2010

The client forechannel and backchannel functionality is included as of kernel version 2.6.31. However, the version in 2.6.31 deviates from rfc 5661 in several important ways and is considered useful for developers only.

This document lists known issues in that initial implementation, including those which had to be addressed before the NFSv4.1 client could be changed from Developer Only to Experimental, allowing distros to more comfortably include the functionality in their releases.

The issues labelled (A) below have since been addressed, and the NFSv4.1 client is labelled "Experimental" as of 2.6.36.

Contents

Legend

  • (A) indicates the issue needs to be addressed prior to status change
  • (B) indicates the issue can be deferred after status change
  • (C) indicates the issue may not be addressed

NFSv4.1 Sessions

Backchannel

  • Duplicate Reply Cache
    • Not yet implemented (B)
      • DRC needs to be implemented before we give the ability to reestablish the backchannel/ connection w/o breaking the session. (B)
    • The backchannel currently only implements idempotent operations and operations that can be retried with no side effects.
    • As specified in 2.10.6.1.3, we should cache and return NFs4ERR_TOO_BIG_TO_CACHE - on a reply where cache_this is set to TRUE or NFS4ERR_RETRY_UNCACHED_REP if cache_this is set to FALSE. This forces server to re-issue the request. Investigate if implementing is faster than returning the errors (A)
  • CB_NOTIFY_LOCK
    • Implement Feature (C)
  • Alternate connection for the backchannel
    • Not yet implemented (B)
    • The backchannel can only currently be bound to the existing forechannel connection.
    • BIND_CONN_TO_SESSION (Separate Connection) (B)
      • Not yet implemented.
      • The workaround is for the client to destroy and create a new session to reestablish the backchannel.
    • BACKCHANNEL_CTL (B)
      • Not yet implemented
      • Provide alternate Backchannel program number
      • Provide Kerberos (not yet supported) Principals for Backchannel
  • Sequence Flag Processing
    • The client does not yet implement the check on the following callback path related flags
    • SEQ4_STATUS_CB_PATH_DOWN (B)
    • SEQ4_STATUS_CB_PATH_DOWN_SESSION (B)
    • SEQ4_STATUS_BACKCHANNEL_FAULT (B)
      • What errors are fixable? This becomes fixable when we add the ability to have multiple connections per session.
  • Inspect "Referring triples" to detect race with forechannel
    • Section 2.10.6.3
    • Not yet implemented (A)
      • Client can mark delegation state for returning - return OK to delegation recall. When the open finishes it immediately returns the delegation.
    • Later we'll have to do the same thing for layout_get/ layout_return
  • Kerberos (B)
    • Not yet implemented (B)
    • Need to ensure krb5 forechannel with AUTH_SYS (or possibly AUTH_NULL) backchannel works (A)

Slot Management/ Negotiation

None of the following items have yet been implemented

  • Client CB_RECALL_SLOT (Handles server reducing slots) (A)
  • Client needs to provide indication of "highest_slotid" and comply with "target" and "enforced highest_slotid" in SEQUENCE OP (B)
  • Define policy to size slot table (startup, congestion, etc) (C)
  • Statistics to monitor (B)
  • Destroy Session when not in use (A-)
  • Ensure client checks LEASE TIMEOUT after every clientid exchange (A)
    • [ Done. This is being done in nfs4_proc_create_session() ]
  • Adjust to correct max-cachesize? (A)
    • Does open require the largest reply size? Is it sufficient to specify enough bytes for an open reply?
    • Do we need to cut down our current size request?

Connection Management

  • Rebind session to a new connection (after loss of connection)
    • BIND_CONN_TO_SESSION (B)
    • Not yet implemented - we currently destroy the session and create a new one

Session Reestablishment

  • Need a thorough review of session and state recovery (A)

SessionID Trunking

Increases the I/O pipe and the number of slots

  • Bind a new connection to an existing session (B)
    • BIND_CONN_TO_SESSION (B)
      • Not yet implemented
    • Issue SEQUENCE with existing sessionID?

ClientID Trunking

  • Not yet implemented (B)

State Management

  • State revocation handling
    • Sequence status bits processing (A)
      • SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING (B)
      • SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED (B)
      • SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED
        • Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
        • Propagate error to app - SIGLOST (B)
        • Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
      • SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED
        • Set LEASE_EXPIRED flag in state manager to force it to reclaim the locks (A)
        • Propagate error to app - SIGLOST (B)
        • Linux 4.0 tries to reclaim the lock instead of notifying the app - needs to be fixed altogether (B)
      • SEQ4_STATUS_ADMIN_STATE_REVOKED
        • Same as above for now (B)
      • SEQ4_STATUS_RECALLABLE_STATE_REVOKED
        • Same as above for now (B)
      • SEQ4_STATUS_LEASE_MOVE (B?)
      • SEQ4_STATS_RESTART_RECLAIM_NEEDED (A)
    • TEST_STATEID (B)
      • Not yet implemented
    • FREE_STATEID (B)
      • Not yet implemented
  • Verify we use the correct stateid ordering (Section 8.2.4) (A)
  • Ensure Close with most recent stateid (not v4.1 specific) (A)
  • Backchannel must check for zero seqid in stateid callbacks (Section 8.2.2) (B)
  • Verify locks and delegations survive session reestablishment (A)

State Reclaim

  • RECLAIM_COMPLETE (A)
    • Not yet implemented
      • Issue after establishing a new clientid even if the server didn't reboot
      • Code server or pyNFS server to accept request
      • Update wireshark to understand new OP (B)
  • Lock recovery when eir_server_owner is different (Section 8.4.2.1) (B)
    • Only needed for migration???
    • Verify client attempts lock recovery when eir_server_scope is same

State Protection

  • SSV Support (for trunking and reconnection) (B)
    • SET_SSV
    • GET_SSV
  • Mach creds (B)

Error Handling Review

  • Error mapping problems
    • Code Inspection (A)
    • Change the place in the kernel code where we map the errors? (B)
      • Is it doable? Are we mixing RPC errors and NFS errors?
    • Code inspection (A)
  • pyNFS server changes to accommodate returning random errors (A)
    • Framework (A)
    • Operations (A)
  • pyNFS regression tests for ongoing development (B)

COMPOUND and CB_COMPOUND

  • Correct use of max sizes (A)
    • Client should take care to use correct request and response max sizes
      • Known problem where max sizes does not allow for compound operation header
      • Audit client to ensure proper GETFH usage after FH modifying ops (Section 2.10.6.4) (B)
  • Mount negotiation
    • Verify server allowed values in CREATE_SESSION is reasonable for us to proceed
    • Client requests reasonable values, then checks for the bare minimum (A)

Minor Version Negotiation

  • Drop down to lower version if failed v4.1 mount (B)
  • Pass something to user-land specifying the error (A)
    • [ Done. EPROTONOSUPPORT is already being returned to user-land ]

Misc Functionality

  • File Delegations
    • OPEN4_SHARE_ACCESS_WANT_NO_DELEG (C)
      • Define Policy/ Implications/ When to use each kind? (C)
    • CLAIM_DELEG_CUR_FH (C)
    • WANT_DELEGATION (C)
    • CB_PUSH_DELEG (C)
    • CB_RECALLABLE_OBJ_AVAIL (C)
    • CB_GETATTR (A)
      • [ done. Checked the spec and CB_GETATTR is the same for 4.0 and 4.1. ]
    • CB_NOTIFY (B)
    • CB_RECALL_ANY (A)
      • Can we track the number of processes that have the delegation so we don't return the most "popular" one. (B)
  • Directory Delegations
    • Do they really buy us anything investigation (C)
    • GET_DIR_DELEGATION (C)
  • Security, Kerberos and RPCSEC_GSS
    • SECINFO (B)
      • Necessary for migration in the future
    • SECINFO_NO_NAME (B)
  • Implementation ID (B)

[ AB: I have implemented a first version of this, queued up for submission upstream]

  • Named Attributes (C)
    • Not yet supported in v4 either
  • Persistent Session
    • OPEN - EXCLUSIVE4_1 (A)
    • GUARDED (if persistent session) (A)
    • Check for Persistent Session flag during CREATE_SESSION response (A)
  • OPEN4_RESULT_PRESERVE_UNLINKED (B)
  • Lock Notification
    • OPEN4_RESULT_MAY_NOTIFY_LOCK (open flag) (B)
    • CB_NOTIFY_LOCK (B)
  • ACL changes: dacl, sacl, inheritance (C)
  • Data Retention (C)
Retrieved from "http://wiki.linux-nfs.org/wiki/index.php/Client_sessions_Implementation_Issues"
Personal tools