[gridftp-user] globus-url-copy with udt and sshftp ..

[Graham Jenkins]

We've been using globus-url-copy (from globus-4.2.1) with udt and sshftp
for copying large numbers of 300-600M files from country-NSW to Perth
WA.

We've found that it's popular with our customers, because there's no
requirement for grid certificates. But it has some shortcomings:

1) It takes forever to build;

2) It doesn't preserve timestamps;

3) It tends to freeze, often writing strings of null characters to
stdout if we're using "-vb".  We suspect that this occurs when an
excessive number of udp packets get dropped.


One work-around for 3) is to compute how long a transfer "should" take,
then use something like twice that time as a limit with the "-t"
parameter.  But the difficulty here is the meaning of "should".

What we've found to be more satisfactory is to run globus-url-copy as a
background process, with a foreground monitor process which kills it if
the time between sensible outputs using "-vb" exceeds a nominated
interval (default: 60 secs).

So our what we'd like to see in a new version is:

a) An option to quickly build simple executables for "file:" and
"sshftp:" URLs only .. a bit like 'bbcp';

b) An option with the above to preserve timestamps;

c) An option to specify a "no-progress" timeout in seconds.


Any chance of this happening?

-- 
Graham Jenkins
Australian Research Collaboration Service
Victorian Partnership for Advanced Computing
(+613) 9925-4862