Alternate Data Streams
From Linux NFS
| (4 intermediate revisions not shown) | |||
| Line 5: | Line 5: | ||
Extended attributes are smaller. (XXX: maximum size varies, maybe around 64K on Linux?) The contents can only be replaced atomically, using special system calls; see [http://man7.org/linux/man-pages/man7/xattr.7.html xattr(7)]. See https://en.wikipedia.org/wiki/Extended_file_attributes for other platforms. | Extended attributes are smaller. (XXX: maximum size varies, maybe around 64K on Linux?) The contents can only be replaced atomically, using special system calls; see [http://man7.org/linux/man-pages/man7/xattr.7.html xattr(7)]. See https://en.wikipedia.org/wiki/Extended_file_attributes for other platforms. | ||
| - | Alternate data streams (or named attributes) are files that happen to be looked up in a different way. You can hold references to them and read and write to them as you would a file, and they can be much larger. The lookup interface differs. NFSv4 provides an [http://tools.ietf.org/html/rfc7530#section-16.17 OPENATTR operation] which takes a file and returns a directory filehandle; READDIR and LOOKUP on that filehandle can be used to get the filehandles of the alternate data streams. On Solaris openat with the O_XATTR flag performs a similar operation. (See [http://docs.oracle.com/cd/E19253-01/816-5175/6mbba7f02/ fsattr(5)], which confusingly calls these "extended attributes".) Windows seems to use a colon character to separate filename from ADS name, with ":" otherwise not allowed to appear in path names. For OSX, see https://en.wikipedia.org/wiki/Resource_fork. See also https://en.wikipedia.org/wiki/Fork_%28file_system%29 ? | + | Alternate data streams (or named attributes) are files that happen to be looked up in a different way. You can hold references to them and read and write to them as you would a file, and they can be much larger. The lookup interface differs. NFSv4 provides an [http://tools.ietf.org/html/rfc7530#section-16.17 OPENATTR operation] which takes a file and returns a directory filehandle; READDIR and LOOKUP on that filehandle can be used to get the filehandles of the alternate data streams. On Solaris openat with the O_XATTR flag performs a similar operation. (See [http://docs.oracle.com/cd/E19253-01/816-5175/6mbba7f02/ fsattr(5)], which confusingly calls these "extended attributes".) Windows seems to use a colon character to separate filename from ADS name, with ":" otherwise not allowed to appear in path names. For OSX, see https://en.wikipedia.org/wiki/Resource_fork (and note: "However, as forks can make it difficult to exchange files with other operating systems, this feature is not in common use. Even in OS X, resource forks are seldom used anymore."). See also https://en.wikipedia.org/wiki/Fork_%28file_system%29 ? |
Linux has support for extended attributes, but not for alternate data streams or NFSv4 named attributes (either on client or server). | Linux has support for extended attributes, but not for alternate data streams or NFSv4 named attributes (either on client or server). | ||
| - | + | The NFSv4 protocol supports "user." extended attributes, see [https://tools.ietf.org/html/rfc8276 RFC 8276], as do recent Linux clients and servers. Note "user." attributes are given meaning only by the applications that use them. The xattr interface is also used on Linux to get and set ACLs and security labels, but that has nothing to do with "user." extended attributes. | |
| - | + | ||
| - | + | ||
| - | The main motivation for implementing alternate data streams on Linux would probably be | + | People have occasionally suggested implementing these interfaces in terms of each other. (E.g. implementing the extended attribute interface on the NFS client by using named attributes under the cover, or implementing named attributes on the server by using the exported filesystem's extended attribute support.) But the above differences make conversion messy: e.g. write to an extended attribute must be implemented as a read-modify-write operation with setxattr and getxattr. |
| + | |||
| + | The main motivation for implementing alternate data streams on Linux would probably be Windows and OSX compatibility: for example, Samba (and to a lesser extent NFSv4) servers want to store them for Windows clients, and we might want to be able to access them on NTFS filesystems. | ||
Implementing alternate data streams would mean designing and implementing the interface to look them up. | Implementing alternate data streams would mean designing and implementing the interface to look them up. | ||
| Line 25: | Line 25: | ||
More recent discussion prompted by an attempt to provide an interface to ZFSOnLinux (a problematic project due to licensing issues): | More recent discussion prompted by an attempt to provide an interface to ZFSOnLinux (a problematic project due to licensing issues): | ||
| - | + | https://en.wikipedia.org/wiki/Resource_fork | |
Latest revision as of 19:04, 14 December 2020
The NFSv4 protocol includes "named attributes" (RFC 7530 section 5.3). Microsoft uses the term "Alternate Data Streams" (ADSs) for a similar NTFS feature. (See, for example, https://blogs.technet.microsoft.com/askcore/2013/03/24/alternate-data-streams-in-ntfs/).
They are sometimes confused with the Linux "extended attribute" feature, but the two are quite different:
Extended attributes are smaller. (XXX: maximum size varies, maybe around 64K on Linux?) The contents can only be replaced atomically, using special system calls; see xattr(7). See https://en.wikipedia.org/wiki/Extended_file_attributes for other platforms.
Alternate data streams (or named attributes) are files that happen to be looked up in a different way. You can hold references to them and read and write to them as you would a file, and they can be much larger. The lookup interface differs. NFSv4 provides an OPENATTR operation which takes a file and returns a directory filehandle; READDIR and LOOKUP on that filehandle can be used to get the filehandles of the alternate data streams. On Solaris openat with the O_XATTR flag performs a similar operation. (See fsattr(5), which confusingly calls these "extended attributes".) Windows seems to use a colon character to separate filename from ADS name, with ":" otherwise not allowed to appear in path names. For OSX, see https://en.wikipedia.org/wiki/Resource_fork (and note: "However, as forks can make it difficult to exchange files with other operating systems, this feature is not in common use. Even in OS X, resource forks are seldom used anymore."). See also https://en.wikipedia.org/wiki/Fork_%28file_system%29 ?
Linux has support for extended attributes, but not for alternate data streams or NFSv4 named attributes (either on client or server).
The NFSv4 protocol supports "user." extended attributes, see RFC 8276, as do recent Linux clients and servers. Note "user." attributes are given meaning only by the applications that use them. The xattr interface is also used on Linux to get and set ACLs and security labels, but that has nothing to do with "user." extended attributes.
People have occasionally suggested implementing these interfaces in terms of each other. (E.g. implementing the extended attribute interface on the NFS client by using named attributes under the cover, or implementing named attributes on the server by using the exported filesystem's extended attribute support.) But the above differences make conversion messy: e.g. write to an extended attribute must be implemented as a read-modify-write operation with setxattr and getxattr.
The main motivation for implementing alternate data streams on Linux would probably be Windows and OSX compatibility: for example, Samba (and to a lesser extent NFSv4) servers want to store them for Windows clients, and we might want to be able to access them on NTFS filesystems.
Implementing alternate data streams would mean designing and implementing the interface to look them up.
Reiserfs previously attempted to implement a similar feature (incorrectly), which promoted a previous thread "possible design issues for hybrids" that may be useful:
http://mid.gmane.org/20040826212853.GA21964@parcelfarce.linux.theplanet.co.uk
(Note on the use of the word "hybrid" here: allowing lookups of named attributes associated with a file is a lot like looking up names in a directory, making the file somewhat of a hybrid of a file and a directory.)
More recent discussion prompted by an attempt to provide an interface to ZFSOnLinux (a problematic project due to licensing issues):
