[gridway-user] Local-remote path differences
Enis Afgan
afgane10 at hotmail.com
Sat Aug 26 16:46:38 CDT 2006
Hello José,Thank you for yor time to asnwer my last question. Unfortunately, I am still not able to succesfuly complete my task. I am unsure of the methods in question (namely, setInputPath and setInputFile) and how do the paths specified actually map onto underlying resources in terms of them being remote or local.If I use setInputPath method to set the input variable STDIN_FILE in the job template file, the file cannot be found on my local system by GridWay job. Lower is given a cutout form the job.log file where it shows the file cannot be found on the local machine. In your earlier response, you said to give file location relative to the working directory. How can I find out what that working directory is? I am executing the application in ~/gw/examples/drmaa (and the input query files are located in ~/gw/examples/drmaa/queries) and, regardles of me specifying the path in either ~/gw/examples/drmaa/gueries or just ./queries format, I get the same respone: File cannot be coppied. If you could please let me know how some of these paths relate to each other w.r.t. local and remote resources and given job, I would appreciate it.EnisSat Aug 26 16:31:03 2006 [DM][I]: ----------------------------------------------------------Sat Aug 26 16:31:03 2006 [DM][I]: New state is PENDING.Sat Aug 26 16:31:07 2006 [DM][I]: New state is PROLOG.Sat Aug 26 16:31:07 2006 [TM][I]: Creating remote job working directory:Sat Aug 26 16:31:07 2006 [TM][I]: Target url: gsiftp://titanic.hpcl.cis.uab.edu/~/.gw_afgane_2/.Sat Aug 26 16:31:07 2006 [TM][I]: Remote job directory created.Sat Aug 26 16:31:07 2006 [TM][I]: Staging input files:Sat Aug 26 16:31:07 2006 [TM][I]: Source: /home/afgane/gw/examples/drmaa.Sat Aug 26 16:31:07 2006 [TM][I]: Copying file file:///usr/local/gw5.0/var/2/job.env.Sat Aug 26 16:31:07 2006 [TM][W]: Skipping file /home/mpiblast/toolkit/ncbi/build/blastall, absolute path.Sat Aug 26 16:31:07 2006 [TM][I]: Copying file ~/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_${TASK_ID}.Sat Aug 26 16:31:07 2006 [TM][I]: Copying file file:///usr/local/gw5.0/scripts/wrapper.sh.Sat Aug 26 16:31:08 2006 [TM][I]: File file:///usr/local/gw5.0/var/2/job.env copied.Sat Aug 26 16:31:08 2006 [TM][I]: Retrying copy of file ~/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_${TASK_ID}.Sat Aug 26 16:31:08 2006 [TM][I]: File file:///usr/local/gw5.0/scripts/wrapper.sh copied.Sat Aug 26 16:31:08 2006 [TM][I]: Retrying copy of file ~/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_${TASK_ID}.Sat Aug 26 16:31:08 2006 [TM][E]: Copy of file ~/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_${TASK_ID} failed.Sat Aug 26 16:31:08 2006 [TM][E]: Some input files were not copied.Sat Aug 26 16:31:08 2006 [TM][W]: Removing remote directory:Sat Aug 26 16:31:08 2006 [TM][W]: Target url: gsiftp://titanic.hpcl.cis.uab.edu/~/.gw_afgane_2/.Sat Aug 26 16:31:08 2006 [TM][E]: Could not remove remote dir (ftp_client_delete/rmdir command.), you may need to remove it manually.Sat Aug 26 16:31:08 2006 [DM][I]: Prolog failed:Sat Aug 26 16:31:08 2006 [DM][I]: Total time : 1Sat Aug 26 16:31:08 2006 [DM][I]: New state is FAILED.Date: Fri, 25 Aug 2006 16:15:14 +0200From: jherrera at fdi.ucm.esSubject: Re: [gridway-user] Local-remote path differencesTo: afgane10 at hotmail.com; gridway-user at globus.org
Dear Enis.
There are two possible solutions:
If you use the file file:///home/afgane/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_1
as input file of the "/home/mpiblast/toolkit/ncbi/build/blastall" executable you
should use the method setInputFiles.
Or, if you use this file as standard input you
should use the method setInputPath, but in this case the file file:///home/afgane/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_1 should
be relative to the working
directory.
Regards,
José Herrera.
----- Original Message -----
From:
Enis
Afgan
To: gridway-user at globus.org
Sent: Friday, August 25, 2006 12:32
AM
Subject: [gridway-user] Local-remote path
differences
Hello,
I am using DRMAA standard and
GridWay 5 implementation to get a version of BLAST running on some of our
resources. Even though most of the tasks are actually surprisingly simple to
accomplish, I am running into trouble when trying to execute an array of jobs.
In the crudest form, the flow of the execution should be the following:
1. using a perl script, fragment
the original query input file (receive number of
fragments)
2. create job template file with
received info
3. submit array of
jobs
4. do some
postprocessing
The algorithm and my
implementation work for a single job submission under the constraint that I
manually copy the input file to the execution host beforehand and specify
STDIN_FILE to point to the location I copied the file to.
If I try executing an array of
jobs, the fragments of the original query input file (at least theoretically)
do get copied to the remote host, but it appears as if the path for the input
file(s) is set to the path on local host and thus is BLAST does not work
properly when invoked on remote machine.
Here is the java code used to
generate the job template:
jt =
session.createJobTemplate();
jt.setWorkingDirectory(java.lang.System.getProperty("user.dir"));
jt.setJobName("BLAST_gw");
jt.setRemoteCommand("/home/mpiblast/toolkit/ncbi/build/blastall");
jt.setArgs(new String[] {"-p
blastp","-d
/home/mpiblast/toolkit/test/yeast.nt"});
jt.setInputPath
("file:///home/afgane/gw/examples/drmaa/queries/"+fragFileNameBase+SessionImpl.DRMAA_GW_TASK_ID);
jt.setOutputPath("first100.out." +
SessionImpl.DRMAA_GW_JOB_ID);
jt.setErrorPath ("first100.err." +
SessionImpl.DRMAA_GW_JOB_ID);
jt.setRequirements ("HOSTNAME=
\"titanic.hpcl.cis.uab.edu\";");
Here is the generated job template
file:
#This file was automatically
generated by the GridWay DRMAA library
EXECUTABLE=/home/mpiblast/toolkit/ncbi/build/blastall
ARGUMENTS= -p blastp -d
/home/mpiblast/toolkit/test/yeast.nt
STDOUT_FILE=first100.out.${JOB_ID}
STDIN_FILE=file:///home/afgane/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_${TASK_ID}
STDERR_FILE=first100.err.${JOB_ID}
REQUIREMENTS=HOSTNAME=
"titanic.hpcl.cis.uab.edu";
RESCHEDULE_ON_FAILURE=no
NUMBER_OF_RETRIES=3
Here is the contents of job.env on
execution host:
export
GW_CPULOAD_THRESHOLD="50"
export
GW_OS_NAME="Linux"
export
GW_OS_VERSION="2.4.20-8smp"
export
GW_CPU_MODEL="x86"
export
GW_CPU_MHZ="2657"
export
GW_MEM_MB="23"
export
GW_DISK_MB="11164"
export
GW_ARCH="x86"
export
GW_EXECUTABLE="/home/mpiblast/toolkit/ncbi/build/blastall"
export GW_ARGUMENTS="-p blastp -d
/home/mpiblast/toolkit/test/yeast.nt"
export
GW_STDIN_FILE=file:///home/afgane/gw/examples/drmaa/queries/mpiblast_qFileFrag_1r1_1
The input file gets copied from
the local machine to the remote one, but it appears from the job.env file that
the path of the local machine is being used on the remote machine and is thus
not working.
I also do not quite understand
setInputFiles and setInputPath methods. If I use setInputPath, STDIN_FILE
variable is set in the job template file. If I use setInputFiles, INPUT_FILES
variable in the job template is set. When using setInputPath function, why is
STDIN_FILE variable set in the job template file? Also, when using
setInputPath, is that the path for all the input files on local machine or on
remote? If it is for the local, where/how is path on the remote machine
specified? Why can I not specify the path on the local machine using
setInputPath and then specify actual input files using setInputFiles, so when
the files get copied form specified local directory to a temporary directory
on the remote machine, only file input names are used and the path on the
remote machine is set to the directory generated for that job by
GW?
I hope my problem is
understandable and I would truly appreciate any insight into above mentioned
issues and topics,
Enis
_________________________________________________________________
Express yourself with gadgets on Windows Live Spaces
http://discoverspaces.live.com?source=hmtag1&loc=us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.globus.org/pipermail/gridway-user/attachments/20060826/80ef1c66/attachment.htm>
More information about the gridway-user
mailing list