[Globus] Help shape Globus' future: A call for use cases

[Ian Foster]

Summary

Over the past few months, within the context of the NSF Community  
Driven Improvement of Globus Software (CDIGS: www.cdigs.org) project,  
we have begun re-evaluating how we should invest toward Globus'  
future.  We believe we are starting to understand alternatives and  
trade-offs.  Of course, our understanding of the needs of the Globus  
community is imperfect, and we do not want to make decisions without  
broader consultation. Thus, we would now like to engage with the  
Globus community to help us work through this evaluation.

A key concern at this point in the process is that we may not have a  
complete enough set of uses cases that represent what people are doing  
with Globus, or hope to do with Globus.  Thus, we ask that you help us  
by providing a description of how you use (or hope to use) Globus,  
along with contact information so that we can follow up with you to  
clarify details and/or participate in subsequent stages of this  
effort. We provide below more details on what we are looking for.

Now the longer version...


Where are we?

We have had tremendous successes with Globus over the past 13 years,  
along with some trying times.  A challenging issue over the years has  
been adapting to the changing technology landscape.  Globus Toolkit  
version 4 (GT4) was introduced approximately 5 years ago, and much has  
changed since then.  So what are the circumstances that factor into  
our decisions on how to evolve?

Users: Many healthy, active, growing communities are relying on Globus  
for their day-to-day work, including TeraGrid, OSG, ESG, LIGO, caBIG,  
and many others.  (Based on our anonymized usage reporting, it is  
clear there are also many more Globus users than those we know  
about.)  These communities challenge us with the constant, healthy  
tension between demands for stability and new functionality.  There  
are also new, large communities that are adopting Globus, such as BIRN  
and other biomedical communities, which will continue to press these  
sometimes conflicting demands.

Funding: We are fortunate to have significant NSF CDIGS funding for a  
couple more years, which allows us to focus on support and development  
of Globus to meet user needs.  We also have a rich mix of grants for a  
wide range of activities, ranging from computer science research, to  
infrastructure deployment and support, to application integration, to  
complete application development.  However, these efforts are of  
course finite in duration, and we must be constantly looking to  
develop new programs to carry Globus forward.

Technology trends: The technology landscape in which we operate  
continues to evolve at a dizzying pace, with technologies and  
approaches such as Cloud Computing, Software-as-a-Service (SaaS),  
OpenID, OpenCL, multi-core, etc.  In the meantime, technology trends  
that we have embraced in Globus, such as SOA and Web services,  
continue to mature, for example with the widespread adoption of  
lighter-weight REST approaches.

Aging foundations: Globus relies heavily on open source components  
from other communities.  Some component choices, such as OpenSSL, have  
stood the test of time.  Others have not: in particular, on the Java  
web services front (e.g. GT4 Java Core), we find ourselves highly  
dependent on libraries such as Apache Axis v1.x and PureTLS, which  
most of the rest of the world has moved away from.  Thus, not only do  
we have to support these technologies entirely on our own, but it  
becomes increasingly difficult to support our users and our own newer  
activities that (rightly) want to exploit new open source software  
libraries and approaches.  These aging GT4 Java Core foundations are  
also inhibiting our ability to effectively address a variety of issues  
(e.g., scalability, REST support) in other Globus components, such as  
GRAM4 (aka WS-GRAM), RFT, and MDS4.

Community: The Globus Alliance (dev.globus.org) community of  
developers has made many valuable contributions that have seen wide  
adoption, such as MyProxy, GridShib, Gaards, and GridWay.  However,  
the dev.globus.org community has not helped significantly with the  
sustainability of existing components by bringing new developers into  
to the community who add features or maintain those components.
	

Where do we go from here?

Thanks in particular to our NSF CDIGS funding, we have some time and  
resources to address these issues, and we will not abruptly leave our  
users out in the cold now or in coming years.  One CDIGS mandate is to  
continue supporting existing users, which we will clearly do.  So  
regardless of what decisions are made, we will continue to support our  
GT4 users as long as we have demand and funding such as CDIGS to do  
so.  But CDIGS also has two other mandates: helping new users, and  
working to ensure Globus' sustainability.  And, of course, we have  
many other projects that are contributing to development of new  
functionality in Globus, which we can leverage as well.

With all of this in mind, our intent is to evolve Globus based on  
three principles:

Embrace widely adopted and community supported software and standards:  
When GT4 Java Core was created over 5 years ago, Apache Axis 1.x was  
the right choice, but as discussed above the world has moved on. (You  
One might argue whether WSRF was the right choice at the time, but  
there is strong support for managing/representing state as a first  
class part of the web services model, as evidenced by adoption of WSRF  
in large communities such as caBIG, and the popularity of REST  
approaches throughout the Web 2.0 community.) We have begun evaluation  
and prototyping efforts around the many alternative web services  
toolkits now available, with goals of: being as conservative as  
possible in our choice of technologies (e.g., looking for toolkits  
with robust communities); being as flexible as possible in what  
requirements we place on our users (e.g., looking to support multiple  
web services toolkits); doing as little "invention" or variance from  
the standard technology as possible (so we do not have to ship our own  
modified version, and plug cleanly into existing systems); while  
preserving core notions that Globus users have found compelling such  
as rich security and first class management and representation of state.

Maintain backward compatibility as much as possible: We recognize that  
protocol incompatibilities have been a recurring headache for our  
community in the past.  Sometimes these are unavoidable as one  
progresses forward -- particularly on the tight budget of a non-profit  
development group such as ours.  But our decisions will be strongly  
influenced by our ability to maintain backward compatibility.  For  
example, one consideration in evaluating web services toolkits is  
whether we can provide WSRF (GT4.0 and/or GT4.2) backward  
compatibility while simultaneously allowing evolution to newer SOAP-  
and REST-based protocols.

Address long-term sustainability: As discussed above, one key aspect  
of sustainability is moving Globus onto newer open source components  
that enjoy wide support from a much broader community than just  
Globus.  Additionally, we have been considering: options for  
simplifying Globus, such as removing little-used features which  
present code maintenance, stability, and/or scalability issues (e.g.,  
GRAM2&4 job rendezvous, GRAM2 stdout/err streaming); reinvesting  
resources into GRAM2 since it is still heavily used; ceasing  
additional development on some components such as GT4 Java Core, GRAM4  
and RFT (e.g., only providing bug fixes, and working with users to  
migrate to other options); and re-implementing components in a more  
maintainable fashion (e.g., RFT).   These examples are merely options  
we are considering -- at this point no firm decisions have been made.

Of particular concern is how to move forward with GT4 Java Core, and  
the GT4 services that are built on it.  Some of our thoughts are:

GT4 Java Core: As discussed above, we are investigating various open  
source web services frameworks.  There are three primary compatibility  
concerns: maintaining optional WSRF protocol compatibility while  
allowing for migration to non-WSRF protocols; porting existing code  
that uses GT4 Java Core to new frameworks; and adopting approaches  
that allows for easier protocol transitions in the future.

GRAM: A large part of the community is still using GRAM2, and GRAM4  
(aka WS-GRAM) has sustainability issues due to its use of GT4 Java  
Core.  We are considering re-investing in GRAM2 to solve some known  
issues (e.g. scalability).

RFT: While many people have expressed interest in RFT, we have had  
limited success with it for a variety of reasons.  RFT also has  
sustainability issues due to its use of GT4 Java Core.  We are  
experimenting with a replacement for RFT which is implemented using a  
Software-as-a-Service model, and are now looking for users interested  
in trying it out and giving feedback.  Please contact Lisa Childers if  
you are interested.

GridFTP: Address some persistent usability concerns, so that GridFTP  
can be downloaded and operated "out of the box" with little or no  
configuration.

MDS4: Various communities use MDS for various purposes, including for  
a service registry (e.g. caBIG, Teragrid), and for monitoring  
services.  Many good options for monitoring have emerged in the open  
source community in recent years, but there still appears to be a need  
for service registries.  We are considering focusing our MDS energies  
more on the service registry use cases.

Security: Many communities use Globus for its powerful and flexible  
security (GSI).  There are already activities underway for integrating  
GSI with other technologies such as Shibboleth, SAML, XACML, and group  
membership services (e.g. VOMS).  However, we also need to understand  
the role and potential integration of GSI in the context of emerging  
technologies such as OpenID and OAuth.

RLS: RLS is used by several important Globus communities. We will  
implement a number of community-requested features, including a  
pluggable back end, improved logging, and more streamlined startup.


How you can help

We need to hear from our users directly about how you use, or want to  
use, Globus.  It is only through a clear understanding of these use  
cases that we can be confident of our decisions.  What we need is a  
detailed description of how you use Globus, including:
	what for (e.g., a science gateway, a data replication system for a  
science experiment, etc.),
	what version,
	what components,
	what features of those components (e.g., the GRAM2 features you rely  
on),
	what clients (e.g., jGlobus for GRAM and GridFTP, Condor-G with GRAM2  
or 4, etc.),
	what sort of performance or scalability issues you are having or  
would like to see,
	etc.
Please email the following people (copied on this email) with your use  
cases for the following components of Globus:
	GRAM 2, GRAM 4, C security: Stu Martin
	GridFTP, XIO: Raj Kettimuthu
	Reliable File Transfer (RFT): Lisa Childers
	GT4 Java Core, Java security: Ravi Madduri
	Service registries (MDS4): Lee Liming
	Replica Location Service (RLS): Ann Chervenak
Please include appropriate contact information, so that they can  
follow up for clarification or more details if necessary.


Next steps

In the coming months you can expect more communication from us as we  
work through this activity.  We will likely produce one or more  
technical whitepapers that describe our proposed directions, on which  
we will be looking for feedback.  We will also produce prototypes that  
you can try to provide early feedback.  We may also try to organize a  
workshop in Chicago in the near future so that we can discuss these  
directions in person -- please let us know if you would be interested  
in attending such a workshop.

Viva Globus!

Thanks in advance for your valuable input.

The Globus Team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.globus.org/pipermail/announce/attachments/20090331/0cac1ffe/attachment.htm>