To study ETL (Extract, transform, load) tools specially SQL Server Integration Services.

By: Shahzad Sarwar

To: Development Team
Date: 27th March 2010

Table of Content
1
2
3
4

Objective:...............................................................................................................................................3
Problem Definition:..............................................................................................................................3
Solution:.................................................................................................................................................3
What is ETL, Extract Transform and Load?.....................................................................................3

ETL - Extract from source...................................................................................3

ETL - Transform the data.........................................................................................3

ETL - Load into the data warehouse........................................................................4

4.1
Microsoft Technology Solution:..........................................................................4
4.1.1
SSIS Designer..............................................................................................6
4.1.2
Runtime engine............................................................................................6
4.1.3
Tasks and other executables.........................................................................6
4.1.4
Data Flow engine and Data Flow components............................................7
4.1.5
API or object model.....................................................................................7
4.1.6
Integration Services Service........................................................................7
4.1.7
SQL Server Import and Export Wizard.......................................................7
4.1.8
Other tools, wizards, and command prompt utilities...................................7
4.1.9
Integration Services Packages.....................................................................7
4.1.10
Command Prompt Utilities (Integration Services)......................................7
4.2
Typical Uses of Integration Services...................................................................7
4.2.1
Merging Data from Heterogeneous Data Stores..........................................8
4.2.2
Populating Data Warehouses and Data Marts..............................................8
4.2.3
Cleaning and Standardizing Data................................................................9
4.2.4
Building Business Intelligence into a Data Transformation Process...........9
4.2.5
Automating Administrative Functions and Data Loading.........................10

5
Conclusion:..........................................................................................................................................10
Reference:......................................................................................................................................................10

Objective:

To study ETL (Extract, transform, load) tools specially SQL Server Integration Services.

Problem Definition:

There are few databases which moved from location to location

during day time, later on at closing of the day or after some specific time interval, data from few tables of
these moving databases is needs to be pushed to central database tables after applying few business rule
like avoiding duplication of data.
Traditional solution to problem is to build a small software in .Net that pull data from moving databases
and apply business rules and push data to central database server.
This solution might work for small problem domain. But actually this is a bigger domain called ETL.

3
4

Solution:
What is ETL, Extract Transform and Load?

ETL is an abbreviation of the three words Extract, Transform and Load. It is an ETL process to extract
data, mostly from different types of system, transform it into a structure that's more appropriate for
reporting and analysis and finally load it into the database. The figure below displays these ETL steps.
ETL architecture and steps

But, today, ETL is much more than that. It also covers data profiling, data quality control, monitoring and
cleansing, real-time and on-demand data integration in a service oriented architecture (SOA), and metadata
management.
ETL - Extract from source
In this step we extract data from different internal and external sources, structured and/or unstructured.
Plain queries are sent to the source systems, using native connections, message queuing, ODBC or OLEDB middleware. The data will be put in a so-called Staging Area (SA), usually with the same structure as
the source. In some cases we want only the data that is new or has been changed, the queries will only
return the changes. Some ETL tools can do this automatically, providing a changed data capture (CDC)
mechanism.
ETL - Transform the data

Once the data is available in the Staging Area, it is all on one platform and one database. So we can easily
join and union tables, filter and sort the data using specific attributes, pivot to another structure and make
business calculations. In this step of the ETL process, we can check on data quality and cleans the data if
necessary. After having all the data prepared, we can choose to implement slowly changing dimensions. In
that case we want to keep track in our analysis and reports when attributes changes over time, for example
a customer moves from one region to another.
ETL - Load into the data warehouse
Finally, data is loaded into a data warehouse, usually into fact and dimension tables. From there the data
can be combined, aggregated and loaded into datamarts or cubes as is deemed necessary.
ETL Tools:
ETL tools are widely used for extracting, cleaning, transforming and loading data from different systems,
often into a data warehouse. Following is list of tools available for ETL activities.
No. List of ETL Tools

Version ETL Vendors

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19

11gR1
XI 3.0
8.1
4.2
8.5.1
7.2.2
7.6
10
3.1
6.5
8.12
7.1
5.2.2
8.2
2.5.2
4.2
9.1
3.0
4.9

Oracle Warehouse Builder (OWB)

Data Integrator & Data Services
IBM Information Server (Datastage)
SAS Data Integration Studio
PowerCenter
Elixir Repertoire
Data Migrator
SQL Server Integration Services
Talend Open Studio
DataFlow Manager
Data Integrator
Open Text Integration Center
Transformation Manager
Data Manager/Decision Stream
Clover ETL
ETL4ALL
DB2 Warehouse Edition
Pentaho Data Integration
Adeptia Integration Server

Oracle
SAP Business Objects
IBM
SAS Institute
Informatica
Elixir
Information Builders
Microsoft
Talend
Pitney Bowes Business Insight
Pervasive
Open Text
ETL Solutions Ltd.
IBM (Cognos)
Javlin
IKAN
IBM
Pentaho
Adeptia

http://www.etltool.com/etltoolsranking.htm

4.1 Microsoft Technology Solution:

Microsoft SQL Server Integration Services is a platform for building high performance data integration
solutions, including packages that provide extract, transform, and load (ETL) processing for data
warehousing.
Microsoft Integration Services is a platform for building enterprise-level data integration and data
transformations solutions. You use Integration Services to solve complex business problems by copying or
downloading files, sending e-mail messages in response to events, updating data warehouses, cleaning and
mining data, and managing SQL Server objects and data. The packages can work alone or in concert with
other packages to address complex business needs. Integration Services can extract and transform data from
a wide variety of sources such as XML data files, flat files, and relational data sources, and then load the
data into one or more destinations.

The SQL Server Import and Export Wizard offers the simplest method to create a Integration Services
package that copies data from a source to a destination.
Integration Services Architecture:

Of the components shown in the previous diagram, here are some important components to using
Integration Services succesfully:

4.1.1 SSIS Designer

SSIS Designer is a graphical tool that you can use to create and maintain Integration Services
packages. SSIS Designer is available in Business Intelligence Development Studio as part of an
Integration Services project.

4.1.2 Runtime engine

The Integration Services runtime saves the layout of packages, runs packages, and provides
support for logging, breakpoints, configuration, connections, and transactions.

4.1.3 Tasks and other executables

The Integration Services run-time executables are the package, containers, tasks, and event
handlers that Integration Services includes. Run-time executables also include custom tasks that
you develop.

4.1.4 Data Flow engine and Data Flow components

The Data Flow task encapsulates the data flow engine. The data flow engine provides the inmemory buffers that move data from source to destination, and calls the sources that extract data
from files and relational databases. The data flow engine also manages the transformations that
modify data, and the destinations that load data or make data available to other processes.
Integration Services data flow components are the sources, transformations, and destinations that
Integration Services includes.

4.1.5 API or object model

The Integration Services object model includes managed application programming interfaces
(API) for creating custom components for use in packages, or custom applications that create,
load, run, and manage packages. Developer can write custom applications or custom tasks or
transformations by using any common language runtime (CLR) compliant language.

4.1.6 Integration Services Service

The Integration Services service lets you use SQL Server Management Studio to monitor running
Integration Services packages and to manage the storage of packages.

4.1.7 SQL Server Import and Export Wizard

The SQL Server Import and Export Wizard can copy data to and from any data source for which a
managed .NET Framework data provider or a native OLE DB provider is available. This wizard
also offers the simplest method to create an Integration Services package that copies data from a
source to a destination.

4.1.8 Other tools, wizards, and command prompt utilities

Integration Services includes additional tools, wizards, and command prompt utilities for running
and managing Integration Services packages.

4.1.9 Integration Services Packages

A package is an organized collection of connections, control flow elements, data flow elements,
event handlers, variables, and configurations, that you assemble using either the graphical design
tools that SQL Server Integration Services provides, or build programmatically. You then save the
completed package to SQL Server, the SSIS Package Store, or the file system. The package is the
unit of work that is retrieved, executed, and saved.

4.1.10

Command Prompt Utilities (Integration Services)

Integration Services includes command prompt utilities for running and managing Integration
Services packages.
a. dtexec is used to run an existing package at the command prompt.
b. dtutil is used to manage existing packages at the command prompt.

4.2 Typical Uses of Integration Services

Integration Services provides a rich set of built-in tasks, containers, transformations, and data adapters that
support the development of business applications. Without writing a single line of code, you can create
SSIS solutions that solve complex business problems using ETL and business intelligence, manage SQL
Server databases, and copy SQL Server objects between instances of SQL Server.
The following scenarios describe typical uses of SSIS packages.

4.2.1 Merging Data from Heterogeneous Data Stores

Data is typically stored in many different data storage systems, and extracting data from all sources and
merging the data into a single, consistent dataset is challenging. This situation can occur for a number of
reasons. For example:

Many organizations archive information that is stored in legacy data storage systems. This data
may not be important to daily operations, but it may be valuable for trend analysis that requires data
collected over a long period of time.

Branches of an organization may use different data storage technologies to store the operational
data. The package may need to extract data from spreadsheets as well as relational databases before it
can merge the data.

Data may be stored in databases that use different schemas for the same data. The package may
need to change the data type of a column or combine data from multiple columns into one column
before it can merge the data.
Integration Services can connect to a wide variety of data sources, including multiple sources in a single
package. A package can connect to relational databases by using .NET and OLE DB providers, and to many
legacy databases by using ODBC drivers. It can also connect to flat files, Excel files, and Analysis Services
projects.
Integration Services includes source components that perform the work of extracting data from flat files,
Excel spreadsheets, XML documents, and tables and views in relational databases from the data source to
which the package connects.
Next, the data is typically transformed by using the transformations that Integration Services includes. After
the data is transformed to compatible formats, it can be merged physically into one dataset.
After the data is merged successfully and transformations are applied to data, the data is usually loaded into
one or more destinations. Integration Services includes destination for loading data into flat files, raw files,
and relational databases. The data can also be loaded into an in-memory recordset and accessed by other
package elements.

4.2.2 Populating Data Warehouses and Data Marts

The data in data warehouses and data marts is usually updated frequently, and the data loads are typically
very large.
Integration Services includes a task that bulk loads data directly from a flat file into SQL Server tables and
views, and a destination component that bulk loads data into a SQL Server database as the last step in a
data transformation process.
An SSIS package can be configured to be restartable. This means you can rerun the package from a
predetermined checkpoint, either a task or container in the package. The ability to restart a package can
save a lot of time, especially if the package processes data from a large number of sources.
You can use SSIS packages to load the dimension and fact tables in the database. If the source data for a
dimension table is stored in multiple data sources, the package can merge the data into one dataset and load
the dimension table in a single process, instead of using a separate process for each data source.
Updating data in data warehouses and data marts can be complex, because both types of data stores
typically include slowly changing dimensions that can be difficult to manage through a data transformation
process. The Slowly Changing Dimension Wizard automates support for slowly changing dimensions by
dynamically creating the SQL statements that insert and update records, update related records, and add
new columns to tables.
Additionally, tasks and transformations in Integration Services packages can process Analysis Services
cubes and dimensions. When the package updates tables in the database that a cube is built on, you can use
Integration Services tasks and transformations to automatically process the cube and to process dimensions
as well. Processing the cubes and dimensions automatically helps keep the data current for users in both
environments; users who access information in the cubes and dimensions, and users who access data in a
relational database.
Integration Services can also compute functions before the data is loaded into its destination. If your data
warehouses and data marts store aggregated information, the SSIS package can compute functions such as
SUM, AVERAGE, and COUNT. An SSIS transformation can also pivot relational data and transform it into
a less-normalized format that is more compatible with the table structure in the data warehouse.

4.2.3 Cleaning and Standardizing Data

Whether data is loaded into an online transaction processing (OLTP) or online analytic processing (OLAP)
database, an Excel spreadsheet, or a file, it needs to be cleaned and standardized before it is loaded. Data
may need to be updated for the following reasons:

Data is contributed from multiple branches of an organization, each using different conventions
and standards. Before the data can be used, it may need to be formatted differently. For example, you
may need to combine the first name and the last name into one column.

Data is rented or purchased. Before it can be used, the data may need to be standardized and
cleaned to meet business standards. For example, an organization wants to verify that all the records
use the same set of state abbreviations or the same set of product names.

Data is locale-specific. For example, the data may use varied date/time and numeric formats. If
data from different locales is merged, it must be converted to one locale before it is loaded to avoid
corruption of data.
Integration Services includes built-in transformations that you can add to packages to clean and standardize
data, change the case of data, convert data to a different type or format, or create new column values based
on expressions. For example, the package could concatenate first and last name columns into a single full
name column, and then change the characters to uppercase.
An Integration Services package can also clean data by replacing the values in columns with values from a
reference table, using either an exact lookup or fuzzy lookup to locate values in a reference table.
Frequently, a package applies the exact lookup first, and if the lookup fails, it applies the fuzzy lookup. For
example, the package first attempts to look up a product name in the reference table by using the primary
key value of the product. When this search fails to return the product name, the package attempts the search
again, this time using fuzzy matching on the product name.
Another transformation cleans data by grouping values in a dataset that are similar. This is useful for
identifying records that may be duplicates and therefore should not be inserted into your database without
further evaluation. For example, by comparing addresses in customer records you may identify a number of
duplicate customers.

4.2.4 Building Business Intelligence into a Data Transformation

Process
A data transformation process requires built-in logic to respond dynamically to the data it accesses and
processes.
The data may need to be summarized, converted, and distributed based on data values. The process may
even need to reject data, based on an assessment of column values.
To address this requirement, the logic in the SSIS package may need to perform the following types of
tasks:

Merging data from multiple data sources.

Evaluating data and applying data conversions.

Splitting a dataset into multiple datasets based on data values.

Applying different aggregations to different subsets of a dataset.

Loading subsets of the data into different or multiple destinations.

Integration Services provides containers, tasks, and transformations for building business intelligence into
SSIS packages.
Containers support the repetition of workflows by enumerating across files or objects and by evaluating
expressions. A package can evaluate data and repeat workflows based on results. For example, if the date is
in the current month, the package performs one set of tasks; if not, the package performs an alternative set
of tasks.
Tasks that use input parameters can also build business intelligence into packages. For example, the value
of an input parameter can filter the data that a task retrieves.
Transformations can evaluate expressions and then, based on the results, send rows in a dataset to different
destinations. After the data is divided, the package can apply different transformations to each subset of the
dataset. For example, an expression can evaluate a date column, add the sales data for the appropriate
period, and then store only the summary information.

It is also possible to send a data set to multiple destinations, and then apply different sets of transformation
to the same data. For example, one set of transformations can summarize the data, while another set of
transformations expands the data by looking up values in reference tables and adding data from other
sources.

4.2.5 Automating Administrative Functions and Data Loading

Administrators frequently want to automate administrative functions such as backing up and restoring
databases, copying SQL Server databases and the objects they contain, copying SQL Server objects, and
loading data. Integration Services packages can perform these functions.
Integration Services includes tasks that are specifically designed to copy SQL Server database objects such
as tables, views, and stored procedures; copy SQL Server objects such as databases, logins, and statistics;
and add, change, and delete SQL Server objects and data by using Transact-SQL statements.
Administration of an OLTP or OLAP database environment frequently includes the loading of data.
Integration Services includes several tasks that facilitate the bulk loading of data. You can use a task to load
data from text files directly into SQL Server tables and views, or you can use a destination component to
load data into SQL Server tables and views after applying transformations to the column data.
An Integration Services package can run other packages. A data transformation solution that includes many
administrative functions can be separated into multiple packages so that managing and reusing the
packages is easier.
If you need to perform the same administrative functions on different servers, you can use packages. A
package can use looping to enumerate across the servers and perform the same functions on multiple
computers. To support administration of SQL Server, Integration Services provides an enumerator that
iterates across SQL Server Management Objects (SMO) objects. For example, a package can use the SMO
enumerator to perform the same administrative functions on every job in the Jobs collection of a SQL
Server installation.
SSIS packages can also be scheduled using SQL Server Agent Jobs.

Conclusion:

Software Industry has may of ETL tools, but Comsoft being traditional Microsoft shop, should
prefer SSIS, SQL Server Integration services as ETL tool for general operations.
As we are using SQL server as backend database, so SSIS is already available in our development
environment. No Need to buy any extra tool.
This Document will provide a technology direction statement and introduction to ETL
implementation in Comsoft.
This document is just for knowledge sharing, no need to change our implementation for pushing
and pulling data via .Net as discussed with Sujjet in last voice call, as our problem domain is
very limited. But for future direction and bigger problems we might consider SQL Server
Integration services as ETL tools .

Reference:
ETL:
http://www.etltools.net
http://www.etltool.com
http://etl-tools.info/
SSIS:
http://www.microsoft.com/sqlserver/2008/en/us/integration.aspx