Reliable File Transfer:

Lessons Learned
Bill Allcock, ANL
Ravi Madduri, ANL
Overview
Quick Overview of the Service
Design Issues
Lifetime Management
Virtualization
Service Data Elements
Implementation Issues
SOAP Processing Issues
Standardization
Avoid Language specific Types
Anyhelper API
Fault Tolerance

RFT in Action
Registry
* The scenarios in this presentation are offered as examples and are not prescriptive
1. A Grid Service
Container is
started up; It
contains an RFT
Factory service;
The RFT Factory
service registers
itself

RFT Factory
Grid Service Container
Client
RFT in Action
Registry
* The scenarios in this presentation are offered as examples and are not prescriptive
2. From a known
registry, the client
discovers a factory
by querying the
Service data of the
registry

RFT Factory
Grid Service Container
Client
RFT in Action
3. The client
calls the
createService
operation on the
factory and
passes in a
TransferRequest
RFT Factory
Grid Service Container
* The scenarios in this presentation are offered as examples and are not prescriptive
Client
RFT in Action
RFT Factory
Grid Service Container
RFT Service Instance
- Start the Instance
- Deserialize XML to Java
- Write Request via JDBC
- Persist Service State
4. The instance
is started, and
the factory
returns a
locater
* The scenarios in this presentation are offered as examples and are not prescriptive
Client
RFT in Action
RFT Factory
Grid Service Container
RFT Service Instance
- Start the Instance
- Deserialize XML to Java
- Write Request via JDBC
- Persist Service State
5. Client calls
Start(),
subscribes to
notificaitons,
etc.
* The scenarios in this presentation are offered as examples and are not prescriptive
RFT in Action
Service is OGSI
compliant
Uses existing
GridFTP (non-OGSI)
protocols and tools
to execute 3
rd
Party
Transfer for the user
Provides extensive
state transition
notification
GridFTP
Server
GridFTP
Server
RFT Service
Instance
* The scenarios in this presentation are offered as examples and are not prescriptive
Lifetime Management
Lifetime management is a key aspect of
OGSI
Was not intuitively clear how to handle this
for disconnected services
Our (perhaps not optimal) solution is to
give it an indefinite lifetime
Should there be an activity monitor?
Does that really solve the problem?
Any other ideas?
Virtualization
Another Key Design Issue in services
We virtualize data movement
It works, LBL and ANL have interoperable
implementation
Need to standardize
Data Services Virtualization from DAIS
Should we pass around GSHs of file services
rather than URLs?
Granularity
Single file .vs. Multi-file .vs. service
composition
Service Data Elements
A huge improvement over the non-OGSI
services
Information Services are (should be)
baked in to the services
Defined both push (event notification) and
pull (full transfer status) SDEs
Need to be cognizant of size, frequency,
and performance of notifications
Lots of interesting possibilities: bandwidth,
errors, network status, etc..
Implementation Issues
SOAP Deserialization
Deserialization can be a HUGE issue. Our
original (very simple) XML could take up to
30 minutes to process.
A straight forward change reduced that
drastically.
SOAP engine also needs to be looked at.
Standardization
Critical for success
Will be moving to the OGSI-Agreement
interface
Implementation Issues (cont.)
Language Specific Data Types
Avoid them (I.e. Java vector type)
A python based service would not be able to
deserialize that.
AnyHelper API
Will deserialize any general XML blob that
utilizes basic types.
Implementation Issues (cont.)
Fault Tolerance
Multiple levels of Fault tolerance
GridFTP will handle remote failures
RFT provides fault tolerance of request via
JDBC compliant database
Service container provides instance fault
tolerance
We write only the primary key into the
wsdd file to avoid slowing down container
restart.