[gridftp-user] popen Usage in GT 4.2.1 GridFTP ..

[John Bresnahan]

First off, the popen feature is considered experimental, so it is outside of the norm of supported 
features to say the least.

programs like tar will not work with parallel streams because they need data to arrive in order.  To 
make them work you will need to add the ordering driver to the destinations fsstack.

Graham Jenkins wrote:
> In an effort to speed copying directories containing large numbers of
> 300Mb files to a a remote destination, I've been trying the tar/untar
> solution suggested at: http://www.mcs.anl.gov/~bresnaha/Stretch/
> 
> What I really want to do is something like:
> 
>  globus-url-copy -vb -udt -pp -p 4 \
>   -src-fsstack popen:argv=#/bin/tar#chCf#/data/ftp/tmp/d1#-#. \
>   -dst-fsstack popen:argv=#/bin/tar#xCf#/var/tmp/graham#- \
>    file:///x sshftp://srbdev.vpac.org/y
> 
> The first problem is that the example only uses 'ftp://localhost:5000/y'
> as a destination. So the example hereunder works fine.
> 
> But it doesn't work if I include a '-p 4' parameter.  And it doesn't
> work if I start the server on a remote host.  And I'm totally in the
> dark about how to make it work using a 'file' pseudo-source and a
> 'sshftp' pseudo-destination.
> 
> Anybody able to enlighten me .. please?
> 
> --
> globus-gridftp-server -p 41000 -dc-whitelist udt,gsi,tcp \
>   -fs-whitelist popen,file,ordering -aa &
> sleep 2
> globus-url-copy -vb -udt -pp \
>   -src-fsstack popen:argv=#/bin/tar#chCf#/data/ftp/tmp/d1#-#. \
>   -dst-fsstack popen:argv=#/bin/tar#xCf#/var/tmp/graham#- \
>   ftp://localhost:41000/x  ftp://localhost:41000/y
> wait