http://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&feed=atom&action=historyServer Side Copy - Revision history2024-03-29T11:24:35ZRevision history for this page on the wikiMediaWiki 1.16.5http://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5365&oldid=prevAmschuma at 18:18, 9 August 20132013-08-09T18:18:03Z<p></p>
<table style="background-color: white; color:black;">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr valign='top'>
<td colspan='2' style="background-color: white; color:black;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black;">Revision as of 18:18, 9 August 2013</td>
</tr><tr><td colspan="2" class="diff-lineno">Line 65:</td>
<td colspan="2" class="diff-lineno">Line 65:</td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"><div>Rather than returning with an error if the filesystem doesn't support the copy_range file operation, instead fall back on do_splice_direct() to copy the data.</div></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"><div>Rather than returning with an error if the filesystem doesn't support the copy_range file operation, instead fall back on do_splice_direct() to copy the data.</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div>Also do the fallback if the filesystem returns -ENOTSUPP to the VFS so NFS v4 and v4.1 can use the copy operation too.</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div>Also do the fallback if the filesystem returns -ENOTSUPP to the VFS so NFS v4 and v4.1 can use the copy operation too <ins class="diffchange diffchange-inline">without having to include a "../internal.h" an export a vfs function to modules</ins>.</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
</table>Amschumahttp://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5364&oldid=prevAmschuma at 18:16, 9 August 20132013-08-09T18:16:50Z<p></p>
<a href="http://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5364&oldid=5363">Show changes</a>Amschumahttp://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5363&oldid=prevAmschuma at 18:02, 9 August 20132013-08-09T18:02:22Z<p></p>
<table style="background-color: white; color:black;">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr valign='top'>
<td colspan='2' style="background-color: white; color:black;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black;">Revision as of 18:02, 9 August 2013</td>
</tr><tr><td colspan="2" class="diff-lineno">Line 21:</td>
<td colspan="2" class="diff-lineno">Line 21:</td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline">2. </del>Argument</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline">== </ins>Argument <ins class="diffchange diffchange-inline">==</ins></div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div> </div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>struct COPY4args {</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>struct COPY4args {</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>/* SAVED_FH: source file */</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>/* SAVED_FH: source file */</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>/* CURRENT_FH: destination file or */</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>/* CURRENT_FH: destination file or */</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>/* directory */</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>/* directory */</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>stateid4 ca_src_stateid;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>stateid4 ca_src_stateid;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>stateid4 ca_dst_stateid;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>stateid4 ca_dst_stateid;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>offset4 ca_src_offset;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>offset4 ca_src_offset;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>offset4 ca_dst_offset;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>offset4 ca_dst_offset;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>length4 ca_count;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>length4 ca_count;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>uint32_t ca_flags;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>uint32_t ca_flags;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>component4 ca_destination;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>component4 ca_destination;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>netloc4 ca_source_server<>;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>netloc4 ca_source_server<>;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>};</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>};</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
</table>Amschumahttp://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5362&oldid=prevAmschuma at 18:00, 9 August 20132013-08-09T18:00:33Z<p></p>
<table style="background-color: white; color:black;">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr valign='top'>
<td colspan='2' style="background-color: white; color:black;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black;">Revision as of 18:00, 9 August 2013</td>
</tr><tr><td colspan="2" class="diff-lineno">Line 6:</td>
<td colspan="2" class="diff-lineno">Line 6:</td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline">=</del>== Data type reference <del class="diffchange diffchange-inline">=</del>==</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div>== Data type reference ==</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>typedef uint64_t length4</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>typedef uint64_t length4</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>typedef uint64_t offset4</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>typedef uint64_t offset4</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div> </div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins></div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>const COPY4_GUARDED = 0x00000001;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>const COPY4_GUARDED = 0x00000001;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>const COPY4_METADATA = 0x00000002;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>const COPY4_METADATA = 0x00000002;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div> </div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins></div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>struct write_response4 {</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>struct write_response4 {</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>stateid4 wr_callback_id<1>;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>stateid4 wr_callback_id<1>;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>count4 wr_count;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>count4 wr_count;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>stable_how4 wr_committed;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>stable_how4 wr_committed;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>verifier4 wr_writeverf;</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>verifier4 wr_writeverf;</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div><del class="diffchange diffchange-inline"> </del>};</div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div><ins class="diffchange diffchange-inline"> </ins>};</div></td></tr>
<tr><td class='diff-marker'>-</td><td style="background: #ffa; color:black; font-size: smaller;"><div> </div></td><td class='diff-marker'>+</td><td style="background: #cfc; color:black; font-size: smaller;"><div></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td><td class='diff-marker'> </td><td style="background: #eee; color:black; font-size: smaller;"></td></tr>
</table>Amschumahttp://wiki.linux-nfs.org/wiki/index.php?title=Server_Side_Copy&diff=5361&oldid=prevAmschuma: Created page with "The server-side copy feature provides a mechanism for the NFS client to perform a file copy on the server without the data being transmitted back and forth over the network. Wit..."2013-08-09T17:50:00Z<p>Created page with "The server-side copy feature provides a mechanism for the NFS client to perform a file copy on the server without the data being transmitted back and forth over the network. Wit..."</p>
<p><b>New page</b></p><div>The server-side copy feature provides a mechanism for the NFS client to perform a file copy on the server without the data being transmitted back and forth over the network. Without this feature, an NFS client copies data from one location to another by reading the data from the server over the network, and then writing the data back over the network to the server. Using this server-side copy operation, the client is able to instruct the server to copy the data locally without the data being sent back and forth over the network unnecessarily.<br />
<br />
The main usecase is for virtual machine migration between servers operating over NFS. Another use is for copying large files from one directory on a server to a different directory on the same server.<br />
<br />
The application calling the copyfile() system call is in charge of opening and closing file descriptors before and after the copy call. If a file can't be opened, it can't be copied.<br />
<br />
<br />
=== Data type reference ===<br />
typedef uint64_t length4<br />
typedef uint64_t offset4<br />
<br />
const COPY4_GUARDED = 0x00000001;<br />
const COPY4_METADATA = 0x00000002;<br />
<br />
struct write_response4 {<br />
stateid4 wr_callback_id<1>;<br />
count4 wr_count;<br />
stable_how4 wr_committed;<br />
verifier4 wr_writeverf;<br />
};<br />
<br />
<br />
<br />
2. Argument<br />
<br />
struct COPY4args {<br />
/* SAVED_FH: source file */<br />
/* CURRENT_FH: destination file or */<br />
/* directory */<br />
stateid4 ca_src_stateid;<br />
stateid4 ca_dst_stateid;<br />
offset4 ca_src_offset;<br />
offset4 ca_dst_offset;<br />
length4 ca_count;<br />
uint32_t ca_flags;<br />
component4 ca_destination;<br />
netloc4 ca_source_server<>;<br />
};<br />
<br />
<br />
<br />
3. Result<br />
<br />
union COPY4res switch (nfsstat4 cr_status) {<br />
case NFS4_OK:<br />
write_response4 resok4;<br />
default:<br />
length4 cr_bytes_copied;<br />
};<br />
<br />
<br />
<br />
4. Copy range system call<br />
<br />
ssize_t vfs_copy_range(struct file *file_in, loff_t pos_in,<br />
struct file *file_out, loff_t pos_out,<br />
size_t count);<br />
<br />
struct file_operations {<br />
<br />
<snip><br />
<br />
ssize_t (*copy_range)(struct file *, loff_t, struct file *, loff_t, size_t);<br />
};<br />
<br />
4.1 My modifications<br />
<br />
Rather than returning with an error if the filesystem doesn't support the<br />
copy_range file operation, instead fall back on do_splice_direct() to<br />
copy the data.<br />
<br />
Also do the fallback if the filesystem returns -ENOTSUPP to the VFS so<br />
NFS v4 and v4.1 can use the copy operation too.<br />
<br />
<br />
<br />
5. Synchronous copy<br />
<br />
Synchronous copy is significantly easier to implement, so it is a good<br />
milestone for implementing the entire copy operation. Later patches can<br />
expand on the sync code to make the copy asynchronous.<br />
<br />
5.1. Client<br />
<br />
- Enable the copy_range operation for v4.2, return -ENOTSUPP for v4 and v4.1.<br />
- Prefer a lock stateid if the file has one, otherwise use the open id.<br />
- Send the compound:<br />
SEQUENCE<br />
PUTFH /* source */<br />
SAVEFH<br />
PUTFH /* destination */<br />
COPY<br />
- Don't need to worry about the server to server case, this may be removed<br />
from the RFC and the vfs_copy_range() function gives an error if the file<br />
moves to a different superblock.<br />
- The server will tell us the number of bytes copied, return this to the<br />
vfs_copy_range() function.<br />
<br />
5.2. Server<br />
<br />
- OP_COPY op_flags should mimic the flags set in OP_WRITE.<br />
- Use nfs_preprocess_stateid_op() to find files associated with both<br />
the CURRENT_FH and the SAVED_FH.<br />
- Call the vfs_copy_range() function with arguments provided by the client.<br />
- Only copy the first 1 GB (1073741842 bytes) of the requested range to<br />
avoid holding an RPC slot for too long.<br />
- Call vfs_fsync_range() after the data is copied and set stable_how to<br />
NFS_FILE_SYNC.<br />
- Return an empty stateid list to the client to show that the copy was<br />
done synchronously.<br />
<br />
<br />
<br />
6. Asynchronous copy<br />
<br />
Asynchronous copy will free up RPC slots, since the server can do the copy<br />
on its own time and simply notify the client once it's done. To be spec<br />
compliant, the OFFLOAD_STATUS and OFFLOAD_ABORT operations also need to be<br />
implemented, but since I mostly want to prepare the client for this case<br />
the server patch may be submitted at a later time.<br />
<br />
6.1. Client<br />
<br />
- Keep a list of offloads that we know are in flight. Use a spinlock to<br />
protect list access.<br />
- Use a struct completion to put the thread to sleep until the callback<br />
comes in if we detect an async copy.<br />
- Watch for the OP_CB_OFFLOAD callback from the server.<br />
- Match the callback stateid to a stateid on the offload waitlist.<br />
- Call complete() on the completion struct.<br />
- Return bytes_copied from the callback data and not the COPY reply data.<br />
<br />
6.2. Server<br />
<br />
- Remove the 1GB copy cap.<br />
- Schedule the copy to run later using a work struct.<br />
- Need to allocate a new structure to pass to the delayed function so the<br />
main thread can be deallocated.<br />
- Need to initialize a new stateid to represent the copy and return it to<br />
the client so it knows to expect the callback.<br />
- Call CB_OFFLOAD after completing the copy.<br />
- Free the stateid during nfsd4_cb_offload_release()<br />
<br />
<br />
7. Userspace test program<br />
<br />
My test program is similar to cp. Run it as `nfscopy.py file1 file2` to<br />
make a copy of file1 named file2. If an entire file cannot be copied in<br />
one call then copy will be called again with the range set to the remaining<br />
data.<br />
<br />
7.1 Code<br />
<br />
#!/usr/bin/python<br />
<br />
import sys<br />
import os<br />
from ctypes import *<br />
<br />
libc = CDLL("libc.so.6")<br />
SYS_COPY_RANGE = 314<br />
<br />
def copyfile(f_in, f_out):<br />
INTP = POINTER(c_int)<br />
size = os.stat(f_in.fileno()).st_size<br />
copied = 0<br />
<br />
while size != 0:<br />
pos = c_int(copied)<br />
addr = addressof(pos)<br />
offset = cast(addr, INTP)<br />
print("Offset: %s size: %s" % (copied, size))<br />
<br />
ret = libc.syscall(c_int(SYS_COPY_RANGE),<br />
c_int(f_in.fileno()), offset,<br />
c_int(f_out.fileno()), offset,<br />
c_int(size)) # count<br />
copied = copied + ret;<br />
size = size - ret;<br />
print("SYS_COPY_RANGE returns: %s" % ret)<br />
print("Total: %s" % copied)<br />
if ret < 0:<br />
return ret<br />
if ret == 0:<br />
return ret<br />
return ret<br />
<br />
<br />
if len(sys.argv) != 3:<br />
print("Usage: " + sys.argv[0] + " src dst")<br />
sys.exit(1)<br />
<br />
if not os.path.exists(sys.argv[1]):<br />
print("ERROR: " + sys.argv[1] + " does not exist :(")<br />
sys.exit(1)<br />
<br />
f_in = open(sys.argv[1], "r")<br />
f_out = open(sys.argv[2], "w")<br />
copyfile(f_in, f_out)</div>Amschuma