IBM FSS

WebSphere Transformation Extender

WebSphere’s Universal Transformation Engine for the Enterprise

Kimberly Drouot
WebSphere Transformation Extender Technical Lead – SW IOT
kimberly.drouot@fr.ibm.com

© 2006 IBM Corporation

IBM Software Group | WebSphere software

What Does IBM WebSphere TX Do?

It takes any kinds of data from their native forms

ANY Data

Processes them together, natively, with NO CODE

Many-to-many Integration

And outputs them into their native target formats

ANY Data

Ex. - Hierarchical Data , Binary Data, Packed Data, Tabular Data, Relational Data, Nested Structures, Mixed-Type Data, and on and on…

1
 IBM Software Group | WebSphere software

Transformation Vision - Simplified

Transform Anything

Transform Anywhere

Everyone Transforms

IBM Software Group | WebSphere software

The Need for Transformation

“Transformation is the hard, ugly part of integration”

-Ted Friedman, Research VP, Gartner

"Organizations pursuing service-oriented integration involving document-oriented,

proprietary, legacy or modern XML-oriented data standards… should consider
products capable of automating complex data transformation as an important building
block of their integration architecture."

Massimo Pezzini, VP and Distinguished Analyst, Gartner

“Two of the biggest and most significant challenges of dealing with data are
transformation and semantics.”

Zapthink

“There are many integration styles and challenges, each with their own unique
demands; what is consistent across every one of them is the need for data
transformation and mediation.”

IBM

2
 IBM Software Group | WebSphere software

The Challenge and Transformation

“About 35 percent of an enterprise’s software budget is
spent on maintaining the multitude of point-to-point
application links already in place. — Gartner Group

Resources Legacy Auto Home Life Business

Partner …
and IT Assets Claims
Database
Claims Claims Insurance
Claims Claims
System System
(insurance company) System

1. Each system has its own data definitions, business rules and interfaces
2. Providing granularity to application’s business functions requires ability to maintain
contextual usage and data meaning across systems
3. Interactions between applications expose data and business rule anomalies which
result in high maintenance costs and inflexibility to change
4. The Validation and Use Rules must align with the Transformation needs and execute in
the same unit of work with Transformation
5. Ever increasing integration logic points with duplicate or slightly modified templates
rob from anticipated benefits and efficiencies

IBM Software Group | WebSphere software

Real World Transformation Demands Hard/Complex

Demand a Broad Range of Capabilities Processing

Interdependent
Almost all projects that require these today, do Data
so via programming, even if they are using
middleware as the platform (ie Java, C++, etc)

Only WTX can provide all of these…

…AND DOES SO WITHOUT CODING ! . Many
to
Many

Many
to
Any
Any
To
Any to Any Many
Easy/Simple

3
 IBM Software Group | WebSphere software

What is the problem of Transformation

Processing Information with all the business rules and usage mechanisms in tact…
BINARY TABLE COPYBOOK
00011110010010011010010 Make Model PKG Extended_Feature
01
03
TP-API-CB.
TP-AGAPI-CB.
10010010010010010000011 s 05
88
TP-AGAPI-REQUEST PIC X(40).
TP-AGAPI-INITIALIZE-REQUEST
CASH RECONCILATION
11010100101010110010010 Ford Prefect 34890 2984782q,
93847920,
88
VALUE 'INITIALIZE-MAPPING'.
TP-AGAPI-PERFORM-MAPPING
VALUE 'PERFORM-MAPPING'.
27 files
10010010010010001010010 3438084 88 TP-AGAPI-FINISH-MAPPING
VALUE 'FINISH-MAPPING'. 10029847 - $100,000,000.00
05 TP-AGAPI-VERSION PIC X(04).
10010010101010101000100 Ford Prefect 34890 2984782q,
93847920,
88
88
TP-AGAPI-VERSION-VALID
TP-AGAPI-VERSION-0100
VALUES ARE '0100' '0200'.
VALUE '0100'.
13948589 - $679,495,094.98
11100010100010010001001 3438084
05
88 TP-AGAPI-VERSION-0200
TP-AGAPI-RESPONSE.
VALUE '0200'.
13950967 - $588,345,058.00
10 TP-AGAPI-RESPONSE-CODE PIC 9(04) COMP.
00100100100100101010010 Ford Prefect 34890 2984782q,
93847920,
88 TP-AGAPI-ALL-OKAY
88 TP-AGAPI-REQUEST-ERROR
VALUE 0.
VALUE 1.
13950968 - $000,000,000.00
01010100100010010010010 3438084 88 TP-AGAPI-INITIALIZE-ERROR
88 TP-AGAPI-MAP-ERROR
VALUE 2.
VALUE 3. 14001321 - $098,957,038.12
88 TP-AGAPI-FINISH-ERROR VALUE 4.
01001001110001010010101 Ford Prefect 34890 2984782q, 88 TP-AGAPI-UNKNOWN-LOOP-ID VALUE 5.

01010101010010010101001
93847920,
3438084
88 TP-AGAPI-NO-ALGORITHM
88 TP-AGAPI-NO-PARTNER
VALUE 6.
VALUES 7 15. AAA: HT4459
88 TP-AGAPI-NO-APPLICATION VALUE 8.

00100010010100101000010 Ford Prefect 34890 2984782q,

88 TP-AGAPI-ALGORITHM-IO-ERROR VALUE 9.
88 TP-AGAPI-FATAL-GATEWAY-ERROR VALUE 10.
AAA: B33566
93847920,

10101010001001010001001 3438084
88 TP-AGAPI-GATEWAY-WRITE-ERROR VALUE 11.
88 TP-AGAPI-PARTNER-IO-ERROR VALUE 12. AAA: C4058G
88 TP-AGAPI-BAD-VERSION VALUE 13.

01001010100101010100101 Ford Prefect 34890 2984782q,

88 TP-AGAPI-BAD-NUMERIC-TYPE
88 TP-AGAPI-NO-ALIAS
VALUE 14.
VALUE 15.
AAB: 948409
93847920,

01100101000000000001111 3438084 10
05
88 TP-AGAPI-ACCESS-ERROR
TP-AGAPI-RESPONSE-MESSAGE
TP-AGAPI-APPLICATION-ID
VALUE 16.
PIC X(80).
PIC X(10).
AAB: 874931
03
10010010010101110010010 01

01010101001010101101111
DDA Application Updates
STREAM PROPRIETARY EBCDIC

<MSG 19934749>
DUNS 0123 <ACCT BAL RPT>
OH,257*IN,142*MI,1 F046000INV for 2 <ACCTNUM><“14001321”>
54*WI,80*MT,5*ID,8 months 120799 ãÖ@Ô…™ƒ £– <ENDBALANCE><” $098,957,038.12”>
<DEBIT><“103048382$394,394.00”>
*WY,3*CO,21*NM,8 12 718-339-1700I43989D-2
120799999-b
™z@Ö <DEBIT><“103048383$001,293.65”>
<CREDIT><“987463921$928,943.67”>
*AZ,15*UT,13*NV,4* 003000010000 •¥…™£@”…@£– </ACCTNUM>
MN,48*MO,67*ND,9 DUNS 4445 P55590
@ÅÂÃÄÉÃ </MSG>
120799499QR
*SD,9*KS,27 000004004000

413 msgs

IBM Software Group | WebSphere software

What Makes WebSphere TX so Different?

Code-Free Design and Deployment

In-Process Data Validation

High Throughput Execution of Complex
Transformations and Enhancements

One Engine, Multiple Deployment Options

4
 IBM Software Group | WebSphere software

Examples of Complex Transformation Challenges Addressed

by WebSphere TX

Data Enhancement

Lookups

Data Logic and Routing

Data Validation

Context Based Data and Usage Rules

Many to Many Transformation

Single-Transaction, interdependent data sets, conversions and logic

Mixed Data and Source/Target Types

Dependent Result Sets, Nested Structure Dependencies

Complex Data Transformation

Nested, Semi-structured And Hierachical Data Types

Dependent Inputs And Outputs

Binary, Packed, EBCDIC, ASCII, Mixed Character Data

IBM Software Group | WebSphere software

One Engine, Multiple Deployment Options

WebSphere TX is the one transformation engine that can be reused
across all applications and infrastructures in the enterprise

Deploy the same WebSphere transformation asset across:

ESB
Unix
30+ Adapters
WebSphere Message Broker

SOA
Linux
ETL
WebSphere ESB

Web Services
Windows
Data quality
WebSphere Process Server

SOAP
z/OS
Enterprise
WebSphere Partner Gateway

EJB – Batch Applications
WebSphere Application Server

HTTP – CICS
Application
WebSphere DataStage
Servers

Event driven – IMS

Batch – USS

Java

C/C++

COBOL

5
 IBM Software Group | WebSphere software

WebSphere TX is further enhanced by Vertical Product

Packs
Financial Services Health Care

SWIFTNet
HIPAA

SWIFTNet Funds
NCPDP

SEPA*
HL7

Many Services Based Solutions
Many Services Based Solutions

FIX
HealthCare Hub

FedWire, CHIPs
Payment Processing

NACHA
Clearing House Processing

BAI, BAI2

AL3

ACORD

Etc.

CPG, Manufacturing Embedded Software

ERP/CRM Integration
Integration to the Enterprise as

EDI (ANSI X12) an embedded service

TRADACOMS
Telecom

ODETTE
Enterprise Apps

EDIFACT
Shop Floor Systems, etc…

Many Services Based Solutions

IBM Software Group | WebSphere software

WebSphere TX – V…
V8.1
A Long And Storied Road V8.0
V7.5
Proven V6.7
…
Q3 2006
V6.5
July 2005
V6.0
Reliable V5.0
June 2004
Jan 2003
V2.1
Jan 2002
V1.4
V1.3 Sept 2001 Innovative
V1.2 June 2000
V1.1
Dec 1999
V1.0
Aug 1997

Nov 1995
June 1996 Mature
June 1995
Jan 1994

6
 IBM Software Group | WebSphere software

Global Financial Institutions rely on

WebSphere Transformation Extender

Fortune 1000 Coverage

9 of Top 10 Commercial Banks

6 of Top 15 Diversified Financials

4 of Top 12 Life and Health Insurance
Companies

8 of Top 14 Property and Casualty
Insurance

Significant Impact on Capital
Markets:

$3 Trillion of transactions flow through
WebSphere TX daily

40% of London securities transactions are
powered by WebSphere TX

WebSphere TX for STP:

100 international banks

70 investment managers

40 broker dealers

10 global custodians

Major Stock Exchanges

European Clearing Houses

IBM Software Group | WebSphere software

WebSphere TX is widely used in all industries

MANUFACTURING, HEALTHCARE /
FINANCIAL SERVICES
RETAIL & DISTRIBUTION INSURANCE

7
 IBM Software Group | WebSphere software

WebSphere TX: Universal Transformation & Reuse

Transform, validate, and enrich any document, message or complex data

Deliver trustworthy information for critical business initiatives

Meet regulatory compliance requirements

Codeless development; universal reuse and deployment

J2EE
Standalone App. Server
Process Server ESB Advanced ESB
System z (native)
“Make Advanced ESB
Windows, UNIX, Linux More Advanced”

Embedded Multi-In MAP
TRANSFORM
Multi-Out

Standalone

Event driven

IBM Software Group | WebSphere software

WebSphere TX: A Unified Transformation

Environment across the WebSphere portfolio

ONE Transformation For batch
Engine for the environments
WebSphere
WebSphere (as a standalone engine)
Enterprise!

Powerful transform TX
TX For WebSphere
capabilities without Message Broker
coding For WebSphere

Natively handles any Process Server
data type MAP
For WebSphere

Solves really hard TRANSFORM
transformation
ESB
problems For WebSphere

Complements Partner Gateway
Unifying
Unifying
existing portfolio of For 3rd Party
products
Enterprise
Enterprise Data
Data Integration Products
Transformation
Transformation (as an embeddable engine)

Delivers new sales
opportunities

8
 ®

IBM Software Group

Developing a WebSphere TX Solution

© 2004 IBM Corporation

IBM Software Group | WebSphere software

The Components of a WebSphere TX Solution

A WebSphere TX solution is composed of the following:

Adapter Type Tree Rules Type Tree Adapter

Source Mapping Destination

In other words…

An adapter is a technical connector which can be
“plugged” into a source or destination - without parsing!

A Type Tree is a graphical representation of meta-data

Rules define how a Source structure is transformed to
a Destination structure

Together these components form a WebSphere TX Map

9
 IBM Software Group | WebSphere software

The Components of a WebSphere TX Solution

However…many to many mapping is also possible:

Adapter Type Tree Type Tree Adapter

Adapter Type Tree Rules

Type Tree Adapter
Adapter Type Tree

Sources Mapping Destinations

A Map can perform many tasks :

Validation

Enrichment,
DATABASE UPDATE

Content Based Routing TRANSFORMED MESSAGE

etc. SOURCE FILE ON OUTPUT

CROSS-REFERENCE
TABLE

IBM Software Group | WebSphere software

Orchestrating Transformations

10
 IBM Software Group | WebSphere software

Developing a WebSphere TX Solution

We usually talk about three simple steps:

Describe the data...

Data that is handled by IBM WebSphere TX, are described in "Type Trees”
that define the data structure and semantics.

Transform the data...

Data is mapped between the source and destination using drag-and-drop.

Deploy the and execute the transformation

Multiple deployment models
 Batch
 Event Driven
 Tightly-coupled (through APIs)
 Loosely-coupled (RMI)

IBM Software Group | WebSphere software

WebSphere TX Design Environment

Describe the Data
Type Designer
• Properties
• Structure
Database Interface
• Validation Rules Designer

Define Transformation Rules

Interfaces Resources
• Structure • Communication
• Formats • Adapters Map Designer
• Semantics

Deploy and Manage

Message Launcher
• Flows
• Events Broker Batch / JCL
• Monitoring IMS CICS

11
 IBM Software Group | WebSphere software

Describe the Data...

IBM WebSphere TX Type Trees

The Type Trees defines the structure of the data to be handled in IBM WebSphere TX.

This XML structure is then represented as an IBM WebSphere TX Type Tree.

IBM Software Group | WebSphere software

WebSphere TX Design Environment

Describe the Data
Type Designer
• Properties
• Structure
Database Interface
• Validation Rules Designer

Define Transformation Rules

Interfaces Resources
• Structure • Communication
• Formats • Adapters Map Designer
• Semantics

Deploy and Manage

Message Launcher
• Flows
• Events Broker Batch / JCL
• Monitoring IMS CICS

12
 IBM Software Group | WebSphere software

Map Designer

Create maps to:

Validate data from data sources

Identify data targets

Specify data transformation logic (rules)

IBM Software Group | WebSphere software

Identifying Data
Sources and Targets

Identify
Identify type
type tree
tree and
and type
type
tree
tree components
components thatthat define
define
data
data target
target

Identify
Identify type
type of
of data
data and
and
location
location of
of data
data

13
 IBM Software Group | WebSphere software

WebSphere TX Adapters
Communication
Messaging ✔CICS *
✔BEA MessageQ & Tuxedo ✔COM
✔Candle Roma ✔CORBA
✔IBM WebSphere MQ * ✔E-MAIL *
(MQSeries) ✔FTP/FTPS *
✔JMS ✔HTTP/HTTPS
✔Microsoft MSMQ ✔JAVA Class
✔Oracle AQ ✔JCA Gateway
✔TIBCO Rendezvous ✔SOAP
✔SOCKET
✔VAN
Applications
Siebel WebSphere Utility
- BIM, EIM, OI COM TX ✔Archive (ZIP/TAR)
✔SAP R/3 & BW ✔Base 64
- ALE, BDC, DMI, DXOB Design Studio ✔Batch
- BW, EDI, BAPI ✔File *
✔PeopleSoft ✔GZIP/ZLIB
- Message Agent API, ✔JNDI
- Open Query, EDI ✔LDAP
✔QualityStage ✔MIME
✔Quoted-Printable
✔Shell Script
Database ✔Sink *
✔DB2 * ✔SNMP
✔Informix ✔XML *
✔ODBC *
✔OLE-DB
✔Oracle * Security
✔Microsoft SQL Server ✔S/MIME
✔Sybase
✔HP NonStop SQL/MX & ✔OpenPGP
SQL/MP ✔SSL Module

* Adapters supported on z/OS, varies by execution method

IBM Software Group | WebSphere software

Step 1 – Define the data structures

Pre-defined Importers for automatic generation • Pre-defined libraries (Packs)

XML Schema, DTDs – EDIFACT

COBOL Copybooks – SWIFT

Database (Tables, Vues, Procédures Stockées) – SEPA*

3rd Party Applications
– ACORD

SAP: BAPI, Idoc, DXOB, BDC – HIPAA

PeopleSoft 7: Business Document, Message – HL7
Agent

PeopleSoft 8: Application Messaging,
Component Interface

Siebel: COM Business Object, EAI, EIM

WSDL (Web Services)

EJB API

Java Class

Java Messaging Service
* En cours

Others: Tuxedo, Flat File, PL1, …

14
 IBM Software Group | WebSphere software

Built-in Validation (Restrictions)

Items have
extensive validation
options

• Enumerations
• Exclusions
(with substitution)
• Patterns
• Ranges

IBM Software Group | WebSphere software

Additional Built-in Validation

Component Rules allow cross-field validation

Incorporates semantic validation directly into the parsing process

15
 IBM Software Group | WebSphere software

Additional Built-in Validation

Component Rules support aggregate functions

Use component rules to specify conditions (business rules) that must
be met for a particular component to be valid

IBM Software Group | WebSphere software

Pre-defined

Step 2 - Creating transformation rules Mapping

Functions

• Drag and Drop

• No Procedural Coding!!
• Easily represent complex business rules

16
 IBM Software Group | WebSphere software

Calling external functions and programs

IBM Software Group | WebSphere software

Rejecting Errors

17
 IBM Software Group | WebSphere software

Step 3 – Deploy and execute the transformation map

Option “Build” creates a
compiled map

For the execution server

Automatically accounts for
“byte-order” and code page
differences

IBM Software Group | WebSphere software

Example – Developing a WebSphere TX Map

Multiple Execution Methods

API
Serveur WTX
1. Define the Data Structures (moteur)

Batch / JCL

Event-Driven / ESB
2. Define the transformation rules
WebSphere Message Broker

WebSphere ESB
3. Deploy and execute
WebSphere Process Server
WTX Adapters

WebSphere Partner Gateway

WebSphere Application Server

WebSphere DataStage, etc.

Z/OS
Deploy

XML Schema
000520****************************************************************
000530 01
000540 03
000550
TP-API-CB.
TP-AGAPI-CB. Importer
000560 05 TP-AGAPI-REQUEST PIC X(40).
000570
000580
88 TP-AGAPI-INITIALIZE-REQUEST
VALUE 'INITIALIZE-MAPPING'.
Map
000590 88 TP-AGAPI-PERFORM-MAPPING
000600
000610
000620
VALUE 'PERFORM-MAPPING'.
88 TP-AGAPI-FINISH-MAPPING
VALUE 'FINISH-MAPPING'.
Importer
000640 05 TP-AGAPI-VERSION PIC X(04).
000650 88 TP-AGAPI-VERSION-VALID VALUES ARE '0100'
000660 '0200'.
000670 88 TP-AGAPI-VERSION-0100 VALUE '0100'.
000680 88 TP-AGAPI-VERSION-0200 VALUE '0200'.

COBOL
Copybook
Design Studio
Windows

18
 IBM Software Group | WebSphere software

WebSphere TX General Architecture

Design Studio

Type Map Integration Flow Unit Test Engine

Designer Designer Designer Importaters Database
Interface
Designer

WebSphere TX Server

WTX Core Engine

WTX Adapters

WTX Compiled Maps (.mmc)

Server A

IBM Software Group

WTX for z/OS

© 2004 IBM Corporation

19
 IBM Software Group | WebSphere software

WTX for z

Transformation and execution on z/OS
Product Offering Execution Method

WebSphere Transformation Extender
Batch / JCL

with Command Server
CICS

IMS TM

IMS/DC
WebSphere Transformation Extender
Event-driven,
with Launcher USS Edition multi-threaded,
multi-process transformation
engine
WebSphere Transformation Extender
APIs & Interfaces for: Cobol,
(embedded edition) Java, C, EJB

For use with WAS, WPS,
Cobol programs, etc.
WebSphere Transformation Extender for
WTX Parser
Message Broker USS Edition
WTX Map Node

IBM Software Group | WebSphere software

WTX for Z/OS - Batch

« Pure » Batch execution mode invoked through JCL

// SET WTXLIB=L333703.WTX.REL8.XPL.LOAD <= WTX LOAD LIB

//DSTX EXEC PGM=DSTX,REGION=0M,

// PARM='SEPA2APP /IF1 INPTSEPA’'
//STEPLIB DD DISP=SHR,DSN=&WTXLIB

SMP/E installation

Supports the following sources/targets natively

Files (QSAM, VSAM)

DB2

FTP

MQ

EXIT function allows existing COBOL and C programs to be called from a

WTX Map

20
 IBM Software Group | WebSphere software

WTX for Z/OS – Batch - continued

Administration through standard z tools (ISPF, etc.)

No need to learn a new monitoring tool

No permanent active processes

WTX is only active when called by the JCL

When map transformation ends
WTX is no longer active

Integrates seamlessly into existing

production procedures

Does not require changes to customer
job and system administration processes

IBM Software Group | WebSphere software

WTX for Z/OS - CICS Z/OS

Transactional execution mode through a CICS transaction

Runs natively in CICS – inline and batch modes supported

d
CICS tee n

SMP/E installation an tio
r c
ua a
G ans
Tr

Supports the following sources/targets natively WTX
CICS

VSAM Files CICS
Appli

key-sequenced

entry-sequenced
WTX Map

relative-record data sets

DB2

MQ

Transient data queues

Temporary storage queues

Storage buffers (COMMAREA)

EXIT function allows existing COBOL and C programs to be

called from a WTX Map

21
 IBM Software Group | WebSphere software

WTX for Z/OS – CICS - continued

WTX z/OS CICS can be invoked:

From a clear terminal by using the DSTX transaction

i.e. DSTX MYMAP –IF1 INPUTF

From a user program by starting the DSTX transaction in the EXEC
CICS START command

From a trigger transient data queue by initiating the DSTX transaction

By calling the DSTXCICS program through an EXEC CICS LINK

Administration through standard z tools (CEBR, etc.)

Integrates seamlessly into existing production procedures

IBM Software Group | WebSphere software

WTX for Z/OS – IMS TM & IMS/DC

Allows WTX Maps to be programmatically executed through IMS Z/OS

Relies on WTX Platform APIs and high-level language programs

C/C++ and COBOL programs can invoke WTX through the IMS IMS
execution option

WTX Maps participate in the IMS transaction

Cobol/C
Application

EXIT function allows existing COBOL and C programs to be called
from a WTX Map WTX

22
 ®

IBM Software Group

WTX for WebSphere Message Broker

© 2004 IBM Corporation

IBM Software Group | WebSphere software

WebSphere Transformation Extender for WMB USS Edition

WebSphere TX is an additional Node on the Message Broker design palette

WebSphere Message Broker

“The most powerful
Any-to-Any
Industry
Compute Compute WebSphere Packs Broker on the Market”
XSLT Mapping TX (SWIFT, EDI,
ESQL Java transformation HIPAA)
transformation transformation Universal
transformation optional
Transformation

Embed WebSphere TX transformations within Message Broker Message Flows

Interoperates with Message Broker V5.x or 6.0

Fully compatible with Message Broker transaction control

Takes full advantage of Message Broker scalability

Full support for multiple sources and multiple targets within a single transaction
scope

Installs in minutes

23
 IBM Software Group | WebSphere software

WTX for WebSphere Message Broker Components

Parser

Transforms bit streams to Message Trees (MQInput)
and Message Trees to bit streams (MQOutput)

Transforms Message Trees to Message Trees (RCD)

Plug-in node

Transforms anything to anything

Icon installs into Message Flow palette

Easy configuration

IBM Software Group | WebSphere software

WTX for WebSphere Message Broker Plug-In Node

The Plugin node can be used in message flow like any

other node

Any to Any Transformations

EDI Transformations
SWIFT Transformations

24
 IBM Software Group | WebSphere software

Chief Value Differentiators of WebSphere TX to

alternative transformation methods

Powerful Transform capabilities with No Coding

Lowers cost to implement, maintain and re-use

High-Throughput, Complex Transformations

Cost per transaction very low

Natively Handles Any Data Type

Never resort to “flattening” or “re-structuring data”

Data integrity and meaning are never lost

In Process Data Validation

Greatly reduces development efforts and implementation costs

Solve Really Hard problems, in less time, with one

common design method

IBM Software Group | WebSphere software

GrazieItalian

Thai
Traditional Chinese

Russian Obrigado Merci

Brazilian Portuguese French

Gracias Spanish Arabic

Danke
German

Simplified Chinese

Japanese

25