[gridftp-user] globus-url-copy with udt and sshftp ..
graham at vpac.org
Sun Apr 26 19:39:36 CDT 2009
We've been using globus-url-copy (from globus-4.2.1) with udt and sshftp
for copying large numbers of 300-600M files from country-NSW to Perth
We've found that it's popular with our customers, because there's no
requirement for grid certificates. But it has some shortcomings:
1) It takes forever to build;
2) It doesn't preserve timestamps;
3) It tends to freeze, often writing strings of null characters to
stdout if we're using "-vb". We suspect that this occurs when an
excessive number of udp packets get dropped.
One work-around for 3) is to compute how long a transfer "should" take,
then use something like twice that time as a limit with the "-t"
parameter. But the difficulty here is the meaning of "should".
What we've found to be more satisfactory is to run globus-url-copy as a
background process, with a foreground monitor process which kills it if
the time between sensible outputs using "-vb" exceeds a nominated
interval (default: 60 secs).
So our what we'd like to see in a new version is:
a) An option to quickly build simple executables for "file:" and
"sshftp:" URLs only .. a bit like 'bbcp';
b) An option with the above to preserve timestamps;
c) An option to specify a "no-progress" timeout in seconds.
Any chance of this happening?
Australian Research Collaboration Service
Victorian Partnership for Advanced Computing
More information about the gridftp-user