[workspace-user] Credential errors in workspaces with file staging
Duncan Penfold-Brown
dpb at uvic.ca
Tue Apr 17 17:35:11 CDT 2007
Hello again -
I have another quick question, this time about workspaces with file
staging specifications. I am attempting to create a workspace and stage
a file from the execute machine (in this case, the workspace services
head node) to another node on the same network. I have installed the RFT
file staging plugin (done as globus using 'ant deploy') on the services
node, and have enabled rft file staging in the jndi-config.xml. I have
also enabled all settings in the worksp.conf file that are required for
certificate staging.
I have tested staging with two different commands. The first attempts to
delegate a credential using the 'workspace' command parameters
--delegate and --delegateXf. Here is the command:
workspace --file
/usr/local/globus-4/share/workspace_client/tests/tty-stage01.epr
--metadata
/usr/local/globus-4/share/workspace_client/tests/workspace-metadata-tty1-staging.xml
--request
/usr/local/globus-4/share/workspace_client/tests/deployment-request.xml
--optional
/usr/local/globus-4/share/workspace_client/tests/propagation-staging-cmdCred.xml
--delegate
https://machine.location:8443/wsrf/services/DelegationFactoryService
--delegateXf
-shttps://machine.location:8443/wsrf/services/WorkspaceFactoryService
When I do this, the workspace begins to boot normally, but then I run
into the deployment message "- State changed: Unstaged --> Corrupted".
This is what the container.log contains regarding the workspace create
(note the long outpout):
2007-04-17 15:31:33,320 INFO impls.WorkspaceResourceImpl
[Timer-0,setOpsEnabled:398] [WORKSPACE-EVENT][id-1]: WS-operations enabled
2007-04-17 15:31:33,329 DEBUG async.RequestDispatch
[Timer-0,initialize:55] intializing RequestDispatch
2007-04-17 15:31:33,333 DEBUG async.RequestDispatch [Timer-0,start:119]
starting up request handler with 1 threads
2007-04-17 15:31:33,336 INFO staging.Stage
[Thread-16_WorkspTaskThrd,_execute:108] [WORKSPACE-EVENT][id-1]:
Requesting transfer of gsiftp://gridsn/share/workspace/rootfs_staged to
gsiftp://gsn-wn1/opt/workspace/images via service
https://142.104.60.52:8443/wsrf/services/ReliableFileTransferFactoryService
2007-04-17 15:31:33,339 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,stage:148] received staging request:
StagingRequest{service=https://142.104.60.52:8443/wsrf/services/ReliableFileTransferFactoryService,
sourceURL=gsiftp://gridsn/share/workspace/rootfs_staged,
destURL=gsiftp://gsn-wn1/opt/workspace/images,
stagingCredential=Address:
https://142.104.60.52:8443/wsrf/services/DelegationService
Reference property[0]:
<ns1:DelegationKey
xmlns:ns1="http://www.globus.org/08/2004/delegationService">6483dd00-ed33-11db-a2a0-ee652e2df710</ns1:DelegationKey>
, transferCredential=Address:
https://142.104.60.52:8443/wsrf/services/DelegationService
Reference property[0]:
<ns1:DelegationKey
xmlns:ns1="http://www.globus.org/08/2004/delegationService">6483dd00-ed33-11db-a2a0-ee652e2df710</ns1:DelegationKey>
, delegDN=/C=CA/O=Grid/OU=phys.uvic.ca/CN=Duncan Penfold-Brown}
2007-04-17 15:31:33,446 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,stage:170] calling
RFTF.createReliableFileTransfer()
2007-04-17 15:31:34,885 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,stage:177] RFT service URL:
https://142.104.60.52:8443/wsrf/services/ReliableFileTransferService
2007-04-17 15:31:35,349 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,startTransferJob:240] calling RFT.start()
2007-04-17 15:31:35,742 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,startTransferJob:243] Leaving startTransferJob()
2007-04-17 15:31:39,202 ERROR service.TransferWork
[WorkThread-20,run:714] Terminal transfer error:
Error authenticating user at source/dest hostServer refused performing
the request. Custom message: Server refused GSSAPI authentication.
(errorcode 1) [Nested exception message: Custom message: Unexpected
reply: 530-globus_xio: Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.] [Caused by: Server refused performing the request. Custom
message: Server refused GSSAPI authentication. (error code 1) [Nested
exception message: Custom message: Unexpected reply: 530-globus_xio:
Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.]]
Error authenticating user at source/dest hostServer refused performing
the request. Custom message: Server refused GSSAPI authentication.
(errorcode 1) [Nested exception message: Custom message: Unexpected
reply: 530-globus_xio: Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.]
. Caused by
org.globus.ftp.exception.ServerException: Server refused performing the
request. Custom message: Server refused GSSAPI authentication. (error
code 1) [Nested exception message: Custom message: Unexpected reply:
530-globus_xio: Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.]. Nested exception is
org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message:
Unexpected reply: 530-globus_xio: Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.
at
org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:166)
at org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:99)
at org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:84)
at
org.globus.transfer.reliable.service.TransferClient.authenticateDestination(TransferClient.java:557)
at
org.globus.transfer.reliable.service.TransferClient.authenticate(TransferClient.java:530)
at
org.globus.transfer.reliable.service.TransferWork.getNewClient(TransferWork.java:436)
at
org.globus.transfer.reliable.service.TransferWork.getTransferClient(TransferWork.java:373)
at
org.globus.transfer.reliable.service.TransferWork.run(TransferWork.java:684)
at
org.globus.wsrf.impl.work.WorkManagerImpl$WorkWrapper.run(WorkManagerImpl.java:355)
at java.lang.Thread.run(Thread.java:595)
2007-04-17 15:31:39,711 ERROR impls.StatefulResourceImpl
[ServiceThread-19,notify:146] Problem moving [id-1] to state 'Unpropagated'
java.lang.Exception: Error authenticating user at source/dest hostServer
refused performing the request. Custom message: Server refused GSSAPI
authentication. (error code 1) [Nested exception message: Custom
message: Unexpected reply: 530-globus_xio: Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.] [Caused by: Server refused performing the request. Custom
message: Server refused GSSAPI authentication. (error code 1) [Nested
exception message: Custom message: Unexpected reply: 530-globus_xio:
Server side credential failure
530-globus_credential: Error reading user credential: Can't open bio
stream for key file: /etc/grid-security/hostkey.pem for reading
530-OpenSSL Error: bss_file.c:109: in library: BIO routines, function
BIO_new_file: system lib
530-OpenSSL Error: bss_file.c:104: in library: system library, function
fopen: Permission denied fopen('/etc/grid-security/hostkey.pem','r')
530 End.]]
at
org.globus.workspace.staging.rft.StagingListener.deliver(StagingListener.java:89)
at
org.globus.wsrf.impl.notification.NotificationConsumerProvider.notify(NotificationConsumerProvider.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
at
org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
at
org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:147)
at
org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
2007-04-17 15:31:39,733 WARN impls.StateTransition
[ServiceThread-19,corrupted:239] Workspace was corrupted (when moving to
state Unpropagated): can not change state anymore unless workspace is
going to be destroyed
------------------------------------------------------------------------------------------------------------
The second command makes use of the credential fields in the
optional.xml file, in which I specify the path to an epr credential file
that I create with 'globus-credential-delegate'. Here is the command:
workspace --file
/usr/local/globus-4/share/workspace_client/tests/tty-stage01.epr
--metadata
/usr/local/globus-4/share/workspace_client/tests/workspace-metadata-tty1-staging.xml
--request
/usr/local/globus-4/share/workspace_client/tests/deployment-request.xml
--optional
/usr/local/globus-4/share/workspace_client/tests/propagation-staging-fileCred.xml
-shttps://machine.location:8443/wsrf/services/WorkspaceFactoryService
After getting the same deployment message, "- State changed: Unstaged
--> Corrupted", I get different error output from this command (again, a
lot of output):
2007-04-17 13:04:26,313 DEBUG async.RequestDispatch
[Timer-0,initialize:55] intializing RequestDispatch
2007-04-17 13:04:26,318 DEBUG async.RequestDispatch [Timer-0,start:119]
starting up request handler with 1 threads
2007-04-17 13:04:26,320 INFO staging.Stage
[Thread-16_WorkspTaskThrd,_execute:108] [WORKSPACE-EVENT][id-1]:
Requesting transfer of gsiftp://gridsn/share/workspace/rootfs_staged to
gsiftp://gsn-wn1/opt/workspace/images via service
https://142.104.60.52:8443/wsrf/services/ReliableFileTransferFactoryService
2007-04-17 13:04:26,369 DEBUG rft.RFTAdapter
[Thread-16_WorkspTaskThrd,stage:148] received staging request:
StagingRequest{service=https://142.104.60.52:8443/wsrf/services/ReliableFileTransferFactoryService,
sourceURL=gsiftp://gridsn/share/workspace/rootfs_staged,
destURL=gsiftp://gsn-wn1/opt/workspace/images,
stagingCredential=Extensibility Element[0]:
<error converting: Fatal Error: URI=null Line=1: Content is not allowed
in prolog.>
, transferCredential=Extensibility Element[0]:
<error converting: Fatal Error: URI=null Line=1: Content is not allowed
in prolog.>
, delegDN=/C=CA/O=Grid/OU=phys.uvic.ca/CN=Duncan Penfold-Brown}
2007-04-17 13:04:26,496 ERROR impls.StatefulResourceImpl
[Thread-16_WorkspTaskThrd,notify:146] Problem moving [id-1] to state
'Unpropagated'
java.lang.RuntimeException: Error starting a transfer with RFT
at
org.globus.workspace.staging.rft.RFTAdapter.stage(RFTAdapter.java:186)
at
org.globus.workspace.service.impls.staging.Stage._execute(Stage.java:164)
at
org.globus.workspace.service.impls.staging.Stage.execute(Stage.java:50)
at
org.globus.workspace.service.impls.async.WorkspaceThread.run(WorkspaceThread.java:56)
Caused by: java.rmi.RemoteException: Problem with credential -> GSS:
null; nested exception is:
java.lang.NullPointerException
at
org.globus.workspace.staging.rft.RFTAdapter.setStagingStubSecurity(RFTAdapter.java:302)
at
org.globus.workspace.staging.rft.RFTAdapter.stage(RFTAdapter.java:158)
... 3 more
Caused by: java.lang.NullPointerException
at
org.globus.delegation.DelegationUtil.getDelegationResource(DelegationUtil.java:199)
at
org.globus.workspace.staging.rft.RFTAdapter.setStagingStubSecurity(RFTAdapter.java:294)
... 4 more
2007-04-17 13:04:26,500 WARN impls.StateTransition
[Thread-16_WorkspTaskThrd,corrupted:239] Workspace was corrupted (when
moving to state Unpropagated): can not change state anymore unless
workspace is going to be destroyed
I've been tinkering with variations of the command and credential
specifications for quite some time now, and have not been able to figure
out what might be going wrong. The information in the container.log is
the only information that I can find on the errors, as services doesn't
get to the point of ssh'ing a command to a hypervisor node. Any help is
again appreciated.
Thanks,
Duncan Penfold-Brown
University of Victoria, CA
More information about the workspace-user
mailing list