From Linux NFS

---

Revision as of 19:06, 3 December 2012

3 December 2012	DRAFT	bjschuma@netapp.com

Contents 1 Overview 2 Related Documents 3 Dependencies 3.1 This design needs the following from others: 3.2 Assumptions 4 Design 4.1 Considerations for Data ONTAP 4.1.1 Resource Requirements 4.1.1.1 Memory Requirements 4.1.1.1.1 Scaling 4.1.1.1.2 Kernel Code 4.1.1.1.3 User-Level Code 4.1.1.2 CPU / Scheduling Requirements 4.1.1.3 Image Requirements 4.1.2 Virtual Appliance Requirements 4.1.3 Build Architecture and Build Impact 4.1.3.1 Components and Dependencies 5 RAS (Reliability, Availability, Serviceability) 5.1 Reliability 5.2 Availability 5.3 Serviceability 5.4 Robustness 5.5 Testability 6 Multitenancy Considerations 7 Feature Interaction Dependencies and Impacts 8 Performance 9 Scalability 10 Open Issues 11 Revision History 12 Approvals 12.1 Approvers 12.2 Reviewers

Overview

The design specification covers the internal details of a module. This includes anything that doesn’t have an effect on the interaction model presented by the Functional Spec (FS) or Architecture Spec (AS).

The target audience for this document is:

• Development – Current and future: be thinking of the new engineer who’s been assigned a burt in this module
• QA – Given this DS, QA should understand the design enough to be able to create white-box type tests for the various parts.

Describe the work concisely but well enough that a reader not on your team will understand at a high level what you're doing, how you're doing it, why you're doing it, who should care enough to read further and why. Be sure to highlight any key interactions with other components of the system.

Provide enough context to make the rest of this document meaningful.

Related Documents

• draft-myklebust-nfsv4-pnfs-backend-protocol-01.txt
• RFC 5661

Dependencies

This design needs the following from others:

Item	Description of Dependency or Issue	Affected Group	Contact
1	This is the reference implementation that the p2p work goes on top of. git://git.linux-nfs.org/projects/bhalevy/linux-pnfs.git	Bryan Schumaker	Benny Halevy
2	pNFS nfs utils needs to be installed on the NFSD server so it can export a filesystem over pNFS. git://git.linux-nfs.org/projects/bhalevy/pnfs-nfs-utils.git	Bryan Schumaker	Benny Halevy

Assumptions

• Enable the following .config options:
  • CONFIG_NFS_V4_1
  • CONFIG_PNFS_FILE_LAYOUT
  • CONFIG_NFS_P2P
  • CONFIG_PNFSD
  • CONFIG_PNFSD_LOCAL_EXPORT
  • CONFIG_PNFSD_P2P
• Install pnfs-nfs-utils on the pNFS server
• Add "pnfs" to the export options of a local filesystem

Design

DESCRIBE YOUR DESIGN IN THIS SECTION

This section is typically the largest section. Since designs are highly specific, the template cannot provide much in the way of guidelines here. Information which is relevant to the sections below should not be discussed here.

This is the main place where customizing the template for each particular team can really pay off. Teams are encouraged to add a section for the design considerations their own particular area needs to address.

The Design specification describes how the functionality is implemented. Intended readers are:

• Engineering (current and future)
• QA; given this spec, QA should understand the design enough to be able to create white-box type tests for the various parts.

• Overall design
  This document should describe:
  • How it works, in detail.
  • Module breakdown
  • Major data paths through the code. (Referring to the use cases might be useful here)
  • Process structure.
  • Major data structures.
  • Concurrency, parallelism, and mutual exclusion.
  • Class hierarchy, if your design uses object-oriented notions of inheritance and polymorphism. This applies to, but is not limited to, development done in object-oriented languages such as C++ and Java.
  • A UML diagram may be the easiest and most precise way of describing the relationship between the various abstractions supported by your design.
  • Any state machines.
  • What persistent storage is used? For Data ONTAP this might be files in the root, rdb databases, registry entries, and the like. For other products, it might be a client filesystem, a NetApp system somewhere, or dedicated hardware. What happens when (not if) these are lost due to failure or hardware replacement?
  • Resources used, how they’re controlled, what we do when we run out, recovery steps
  • What languages are involved.
  • Document how the consistency model is maintained. (NG, CFO, consistency points, etc.)

• Licenses
  • Describe how licenses are used, especially if the license checking must be done before the licensing infrastructure is initialized.

• Upgrade/revert
  • Describe how upgrade and revert work.
  • Discuss how these modules interact with CFO and SFO, data motion, and data replication.

• Install/uninstall
  • Describe how the product is installed and uninstalled.

• Versioning/compatibility
  • Describe how the versioning checks are implemented.
  • If wire- or disk-layout is important, discuss tools (like IDL’s) used to achieve that.

• Internationalization/language support

• Branding and brand or vendor-neutral implementation.

• Configurations
  • Describe algorithms related to the platform or architecture type.
  • Describe algorithms affected by user configuration.

• Packaging
  • Does it change the build/release/install process in any way (e.g. adds new build types, new build steps, new build files, new files to be shipped in the tar bundle, etc.) If so, describe how these are implemented.

• Online documentation
  • Describe implementations of documentation of any form (for example, tools which process commentary and create other documents)

Considerations for Data ONTAP

Fill in this section if and only if this design specification applies to Data ONTAP.

Resource Requirements

If this feature impacts Data ONTAP resource utilization (increase or decrease), then the applicable subsections must be filled-out, and the specification must be reviewed and approved by a representative from dl-resource-review.

If this information was already provided in the Functional Specification, and has remained unchanged, then simply provide a pointer to the Resource Requirement Section in the Functional Specification and no further resource review is necessary. If, however, the resource requirements have changed in some way, then: show all requirements here, highlight the differences, and have them reviewed and approved by a Resource Requirement Reviewer.

Memory Requirements

DESCRIBE WHAT THE MEMORY IS/WAS BEING USED FOR IN THIS SECTION.

If this feature consumes more memory or results in less memory utilization, then this section must be filled out.

The memory of primary interest is that used for data, for example: key data structures, buffer/table allocations, message allocations, preexisting resource resizing. Although other notable usage may be included if known. This section is not meant to account for every byte, but rather to obtain reasonable estimates and ensure you're thinking through various system design aspects.

If the feature results in reducing memory utilization, then only answer the applicable questions. The spec will still have to be reviewed by Resource Review to allow for the proper tracking of the memory reduction.

Scaling

Feature memory sizing should scale with the amount of physical memory available in the platform.

In the table below, show the minimum and maximum bound of this feature's memory requirements for each listed platform class.

Platform	RAM	NRAM	MIN Size	MAX Size	Reason for Range
VSim	2GB	32MB
FAS3040, FAS3140, VSA	4GB	512MB
FAS3210, VSA	4GB	640MB
FAS2240(Vespa)	6GB	768MB
FAS3070, FAS3160,FAS3240	8GB	1GB
Buell	8GB	4GB
FAS6040	16GB	512MB
FAS3170, FAS6040	16GB	2G
FAS3270	20GB	2G
FAS6080, FAS6070	32GB	2G
Bimota-L	32GB	4G
FAS6280	96GB	4GB
Revolution	128GB	16GB

Note: Not all product platforms are represented above, but the table shows the major classes to consider, and will show the feature memory trend. In addition, for some platforms the RAM size shown is decreased to account for NVMEM usage.

• If the memory requirements are not sized differently for each platform class, explain why.

• For a given platform, describe the conditions under which memory utilization is increased or decreased during system operation.

• If memory is never freed, explain why.

• Initialization and conditional allocation: The initial allotment of memory for a given feature should only be allocated when the feature is activated. If this is not the case, or if the the feature is allocated unconditionally at boot, explain why.

• If memory size in future platforms is doubled or quadrupled, how is this feature impacted?

Kernel Code

Data ONTAP has several kernel-level memory allocators. In the table below, show which allocator(s) your feature intends to allocate its memory from. A given feature may use memory from a single or multiple memory sources. List the amount of memory being allocated from each.

If you are unfamiliar with the tradeoffs of using each allocator, then show your memory requirement in the "I don't know" section, and you will be advised by your Resource Requirement Reviewer during the spec review.

Memory Pool	MAX Size	Reason why this allocator was chosen
Static allocation: BSS data segment for global and static variables
SK standard memory allocator: sk_kmem_alloc
SK early memory allocator (direct mem, physically contiguous): sk_allocate_memory
WAFL: wafl_steal_buf
FreeBSD memory allocator: malloc
NVRAM/NVMEM
Heap Segment (HSEG)
I don't know

Note: HSEG is not a memory allocator. It is a pool of virtual addresses, which is used for stitching together discontiguous physical pages, to create a virtually contiguous memory region. Allocating >4k chunks from the SK allocators results in the equivalent amount of HSEG consumption.

• If SK memory allocators are being used, is the memory allocated before or after WAFL is initialized?
• Does the memory have to be physically contiguous? If so, explain why.
• Are the individual memory allocations >4k?

User-Level Code

• Does the application need to start before swap (aka, WAFL) is online? If so, why?

The following questions may be answered after the application is developed, at which point it can be measured using the "top" and "swapinfo" commands. If identified after the formal specification review is complete, be sure to notify dl-resource-review.

• Specify the working set size of the application: amount of physical memory needed by the application under normal operating conditions. The rest of the application's memory exists in swap or is never loaded. This value can be learned using the resident (RES) statistic provided by the FreeBSD "top" command.
• Specify total virtual memory size of the application. This size can be learned using the virtual (SIZE) statistic provided by the FreeBSD "top" command.
• Swap impact: if the application swaps, how does it affect overall system performance or the feature's response time? For example, if VLDB swaps under normal conditions, the system performance degrades dramatically. The same is true of Anti-Virus.

CPU / Scheduling Requirements

This section is not meant to account for every CPU cycle, but rather to obtain reasonable estimates and ensure you're thinking through various system design aspects.

• Do you expect this feature to impact CPU utilization? If yes, explain.
• Is the CPU utilization expected to be different for different platforms? If yes, explain.
• How does this feature handle different system core counts (e.g., 1, 2, 4, 64)?
• How many threads does this feature add to or remove from the system?
• Does this feature run its threads in user-space or in the kernel, or both?
• If in the kernel, what CSMP domain will the code run in? If a new domain is being added, explain why.
• If not the default, what priorities are your threads running at and why.
• Are your threads long-running threads (run for long periods of time without yielding the processor)? If yes, explain why.

Image Requirements

The following questions may be answered after the application is developed, and file sizes can be measured. If identified after the formal specification review is complete, be sure to notify dl-resource-review.

How much space is your feature going to consume in the various filesystems:

Resource	Size	Reason
CFCARD: rootfs
CFCARD: platfs
CFCARD: mroot
CFCARD: other
NVRAM: varfs
I don't know

All images which are needed to boot the system reside in rootfs, while all others belong in mroot.

• If your image is not resident in mroot, explain why.

Virtual Appliance Requirements

Data ONTAP-v technology enables Data ONTAP to execute within a virtual machine for execution in virtual environments.

• Are any Virtual Storage Appliance (VSA) changes or special considerations required?
  • If so, describe and add dl-vsa-dev to the approver list.

• Are any Simulator/VSim changes or special considerations required?
  • If so, describe and add dl-vsim-dev to the approver list.

• If your feature has platform specific limits, define the limits for the virtual platforms as well.

Build Architecture and Build Impact

• How will your code be built (e.g., bedrock, or other means)?

• Are you just adding significant new code to existing files? If so, please explain why the new code should not be placed in separate file(s). (Creating too many files is bad but 3000+ line source files are not good either.)

• Are there parts of your new software that should not be branched by release? (for example -- firmware, error codes) If so, specify the details.

Components and Dependencies

Adding unnecessary build dependencies between Data ONTAP components is easy to do, difficult to detect and undo after the fact and slows down builds.

• Are you creating a new component? Are you using standard subcomponents?

• Explain how the code will publish (export) public APIs and data structures (if any).

• What components do you expect to depend on?

• What components do you expect will depend on you?

RAS (Reliability, Availability, Serviceability)

Please refer to RAS Basics for more background on what is meant by "RAS."

Reliability

Availability

• Discuss how the product interacts with the takeover and giveback mechanisms of CFO (HA), or the ONTAP-NG clustering mechanism, as appropriate.
• Discuss NDU (non-disruptive upgrade) upgrades for Data ONTAP. How does this get implemented in this product/feature?

Serviceability

• Discuss unusual implementations related to diagnosibility, logging, tracing.

Robustness

• If important data is held in files, describe what happens if the file is deleted, renamed, etc.
• If important data is in files (or metafiles), describe what happens if the file is corrupted. Does wack fix it? Can it be deleted and recreated? Does the setup step fix it?

Testability

• Discuss unusual implementations related to testing
• What would be required to achieve complete code coverage?

Multitenancy Considerations

Are there any Secure Multi-tenancy (Vserver and Delegated administration)design considerations for this feature? If so, consider which services, protocols, policies, schedules or manageable objects will need to be Vserverized. It is recommended that security implications also be considered.

Feature Interaction Dependencies and Impacts

Please review the Feature Reference List and denote any dependencies in this section.

Performance

Describe what if any aspects of the design impact the performance?

• What bottlenecks, limitations, or unpredictable performance effects may result from the design, and why?
• Discuss resource limitations and sizing issues as they apply to performance.

Scalability

Provide details about how scalability goals identified in the related Architecture and Functional Specifications will be met.

Provide descriptions of data structures, algorithms, and programmatic interfaces between Data ONTAP components, or between client and server, which are needed to achieve a scalable solution.

For example, fast lookup of a logical object may involve replacing use of a linear based search, with use of a hash table or btree based search.

Open Issues

Record in this section issues that you are aware of, but which are not yet resolved in the specification. If you discover issues after the specification is approved, you may record them here, and then re-review the specification after you address the issues.

Item	Date	Name	Issue	Resolution	Date Resolved
1	Date the issue was raised.	Who raised it?	Describe the issue.	Describe what you did to resolve the issue.	Date

Revision History

The entries below are for this template itself. Replace them with the history of changes to your specification.

Version	Date	Name	Change
1.0	3 Feb 2006	Garth Rodericks	Initial version adapted from ONTAP template
2.0	24 Feb 2006	Becca Beaman	Integrated comments from first review.
2.1	8 Mar 2006	Becca Beaman	Added note on customization to main section moved Revision History to end.
2.2	14 Apr 2006	Garth Rodericks	Updated formatting to match other docs. Eliminated title page and TOC.
2.3	3 May 2006	Garth Rodericks	Added TOC to section.
2.4	22 Jun 2006	Garth Rodericks	Moved related docs to follow overview.
2.5	9 Oct 2006	Becca Beaman	Changed “Features” section to “Design,” Added text “DESCRIBE YOUR DESIGN IN THIS SECTION.”
2.6	13 Dec 2006	Garth Rodericks	Updated instruction text formatting to show in blue italics and added missing Revision History table.
2.7	23 Feb 2007	Charlie Hedstrom	Update copyright year, correct WIKI instructions.
2.8	12 Jun 2007	Kim Merriman	Added Document Status section
2.9	22 August 2007	Becca Beaman	Added pointer to "RAS Basics" in RAS section.
2.10	10 September 2007	Brian Hackworth	Removed the unneeded "Document Status" section. Split Dependencies into two tables: incoming and outgoing.
2.11	10 September 2007	Brian Hackworth	Added "Assumptions" sub-section in Dependencies.
2.12	12 September 2007	Garth Rodericks	Updated formatting at top of document, updated link to authoritative Word version, and eliminated duplicate text from top of document that's already in Objective section.
2.13	27 September 2007	Brian Hackworth	Added some more detailed things to think about with respect to memory budgets in Data ONTAP.
2.14	2 May 2008	Brian Hackworth	Added to the Approvals section a description of "required reviewers" as distinct from approvers.
2.15	9 October 2008	Garth Rodericks	Added information classification and keyword to comply with NetApp Information Security policy.
2.16	2 October 2009	Brian Hackworth	Added Open Issues section and Target Approval Date.
2.17	8 April 2010	Brian Hackworth	Added mention of branding in Design section.
2.18	9 December 2010	Joe CaraDonna, Eric Hamilton	Rework for Data ONTAP 8 and TAB++ review process.
2.19	23 December 2010	Joe CaraDonna	Further refined ONTAP Resource Requirements section.
2.20	16 March 2011	Eric Hamilton	Added Snaplock section.
2.21	17 May 2011	Kathy Coencas	Added instruction for Spec Tool users to replace the approver table with a link to the approver list in Spec Tool
2.22	19 May 2011	Kathy Coencas	Added link to Spec Tool to instruction for Spec Tool users to replace the approver table with a link to the approver list in Spec Tool
2.23	05 July 2011	Kathy Coencas	Added Scalability sections/instructions and highlighted instructions to add a link to approvers in the spec tool
2.24	21 December 2011	Kathy Coencas	Removed IE as a mandatory approver of the Design Spec
2.25	25 January 2012	Kathy Coencas and Vanesa Knisley	Updated section 9 based on approved change request, to add Feature Interaction Dependencies and Impacts.
2.26	19 March 2012	Vanesa Knisley	Updated Section 8 based on approved change request, to add more detail
2.27	9 August 2012	Vanesa Knisley	Added Section 9 based on approved change request, to match AS and FS

Approvals

Approvers

Record here the names of the individuals who must approve the specification. When they approve the specification, add the date of their approval in the last column. If your specification is in Spec Tool, replace the table below with a link to the approver list in Spec Tool.

Guidelines for approvers:

• A Technical Director for your project area is the primary approver, and verifies that the specification is complete, adequately addresses the problem space, is consistent with the architecture for the product, is consistent with existing products and features, and adequately addresses dependencies with other projects. Additionally, the TD should verify that the specification has been reviewed both within the project team and with any other teams with dependencies, and that comments raised during reviews have been incorporated into the specification.
• If the specification calls out Dependencies with other groups, include the Technical Directors (or delegates) from those groups as approvers.
• The Product Manager verifies that the specification adequately addresses the requirements from the Engineering Requirements and Response Document.
• The Quality Assurance approver verifies that the specification defines the features and behaviors in sufficient detail to begin work on test planning. This should include the format and content of all inputs and outputs that are user visible.
• For any Data ONTAP design spec, the Technical Advisory Board uses this checklist to verify that the design is consistent with the overall architecture of Data ONTAP. The TAB may also identify additional reviewers and groups that should be consulted.
• The Resource Requirement Reviewer verifies that the specification defines the resource requirements of the feature in sufficient detail to assess whether the target platforms can support it, and assists in planning for system growth. A Resource Requirement Reviewer can be assigned by contacting dl-resource-review.
• If there are other areas of expertise that the project team desires input from (for example, a review of the User Interface sections by someone with UI expertise), or if the specification is complex, feel free to include additional approvers as needed.

Name	Role	Target Approval Date	Approval Date
Name	Technical Director, or delegate	Date	Date
Name	Technical Director, or delegate for any dependent groups	Date	Date
Name	Product Manager	Date	Date
Name	Quality Assurance	Date	Date
Name	Resource Requirement Reviewer	Date	Date
Name	Technical Advisory Board Member	Date	Date

Reviewers

Reviewers are those people who should be informed of the feature, but who are not required to officially approve it. Normally, these are people you depend on, or who depend on you, and are called out here to make sure they're aware of the dependency. Record here the names of the individuals who should review the specification, and upon completion add the date in the last column. If your specification is in Spec Tool, replace the table below with a link to the approver list in Spec Tool.

Name	Role	Target Approval Date	Approval Date
Name	-	Date	Date
Name	-	Date	Date
Name	-	Date	Date