Community inTouch | SQL Server

2005

Integration Services for

Developers

Risman Adnan
Developer and Platform Evangelist
Microsoft Indonesia
rismana@microsoft.com
1

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

SQL Server
Business Intelligence
Integrate

Data acquisition from

source systems and
integration
Data transformation
and synthesis

Analyze

Data enrichment,
with business
logic, hierarchical
views
Data discovery via
data mining

Report

Data presentation
and distribution
Data access for
the masses

Data integration without SSIS

Alerts & escalation

Call centre data: semi structured

Text Mining

Staging

Staging
Legacy data: binary files

Hand
coding

Cleansing
&
ETL
Application database

ETL

Data mining

ETL

Staging

ETL

Warehouse

Reports
Mobile
data

Integration and warehousing require separate, staged, operations.

Preparation of data requires different, often incompatible, tools.
Reporting and escalation is a slow process, delaying smart responses.
Heavy data volumes make this scenario increasingly unworkable.

Data Integration with SSIS

Alerts & escalation

Call centre:
semi-structured data

Text mining
components

Data mining
components

Custom
source

Merges

Standard
sources

Data cleansing
components

Mobile
data

Warehouse

Legacy data: binary files

Application database

SQL Server Integration Services

Reports

Integration and warehousing are a seamless, manageable, operation.

Source, prepare and load data in a single, auditable process.

Now lets explore the SSIS platform

SSIS Logical Architecture

BI Development Studio
SQL Studio

SQL Agent
Subsystem

dtutil
dtexec

SSIS Runtime Client and SSIS Service

Connection
Manager
Variable and
Property
Expression
Configuration

Packages

Control Flow

Data Flow

Precedence
Constraint
Custom Tasks

Event Handler

Other Tasks

Package Explorer

Containers

Control Flow

SSIS Runtime Core Engine

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

SSIS Project and Package

Integration Services Project
Data Sources
Data Source Views
Packages
Control Flow
Data Flow
Event Handlers
Package Explorer

Dynamic Property
Configurations
Variable-based Properties

Development
environment basics

Putting It Together
Control Flow provides execution
environment
Data Flow is just one of many tasks
Copy data, just like you used to
Manipulate data like you never
thought possible!

10

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

11

Development with SSIS

A development environment
Design
Debug
Deploy

A development platform
Embeddable
Extensible
Custom workflow tasks
Custom Enumerators

Custom data flow components

Source, Destinations and Transformations

12

SSIS As A Source
ETL processes
typically encode
complex business
rules
Reuse is important
One version of the
truth
Updates in one place
Leverage advantages
of SSIS: scalability,
manageability, visual
building of complex
processes, etc.

13

Development environment
summary

SSIS Projects
Control flow and data flow distinguished
Control flow
Tasks
Loops, sequences and events
Variables and scoping
Precedence constraints

Connections
Data flow

Source and destination adapters

Transformations
Multiple sources with joins and unions
Multiple destinations with splits and multicast
14

The Visual Debugger

15

Visual debugging
summary
Debug control flow with breakpoints;

Debug control flow with breakpoints;

Add breakpoints to events
Variables in the locals window
Watches and the call stack

Debug data flow with data viewers

Grid data viewers
Data Visualizers
Chaining data viewers
16

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

17

Custom Workflow Tasks

Reusable units of work
Task Host enables your custom task
to be a full SSIS citizen with minimum
effort, inheriting
Logging
Debugging
Event Handling
Configuration
Checkpoint restart
18

Preparing to write a custom

task
1. Make sure your project has a fixed version attribute.
In C#, replace [assembly: AssemblyVersion("1.0.*")]
with [assembly: AssemblyVersion("1.0.0.0")]
in AssemblyInfo.cs file.
2. Add a strong name to your assembly in order to GAC it.
3. Create an SNK keyfile in your project directory: sn.exe -k
keyfile.snk
4. Add the following line to AssemblyInfo.cs
[assembly: AssemblyKeyFile("..\\..\\keyfile.snk")]
5. Add a reference to Microsoft.SqlServer.ManagedDTS.dll and to
Microsoft.SqlServer.DTSRuntimeWrap.dll.
6. Add using Microsoft.SqlServer.Dts.Runtime to the beginning
of your source code files.

19

Writing your Custom Task

7. Derive your class from Task and add DtsTask attribute:
[DtsTask (DisplayName = "My Custom Task")]
public class MyTask : Task
{ ... }
8. Write your task by overriding virtual functions defined in the
Task class:
Validate
May be called anytime
Should flag invalid settings or other potential failures

Execute
Takes parameters for connections, variables, events, logging, and transaction

20

Installing your Custom

Task
9. For SSIS to know about your component, copy the assembly
DLL to
C:\Program Files\Microsoft SQL Server\90\DTS\Tasks
10. For SSIS to be able to load your component, add the
assembly DLL to GAC
gacutil.exe /I <your_dll>
11. Restarting the BI development environment, right-click
Toolbox and choose Add/Remove Items.
12. Switch to the Control Flow Items tab, find your Task in the
list and check it.
13. Youre done! To debug your component, execute the
package using DTExec and configure Visual Studio debugger
to start DTExec when debugging the project.

21

Building a custom task

22

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

23

Custom Data Flow

Components

Reusable transformation components

Reusable data adapters
For unique, unusual or unsupported data
source needs

Again, the wrapper enables your

custom component to inherit rich
functionality
Data debugging
Highly optimized execution plans
Data error handling
24

Preparing to write a custom component

1. Make sure your project has fixed version attribute.
In C#, replace
[assembly: AssemblyVersion("1.0.*")]
with
[assembly: AssemblyVersion("1.0.0.0")]
in AssemblyInfo.cs file.
2. Add a strong name to your assembly in order to GAC it.
3. Create an SNK keyfile in your project directory: sn.exe -k
keyfile.snk
4. Add the following line to AssemblyInfo.cs
[assembly: AssemblyKeyFile("..\\..\\keyfile.snk")]
5. Add a reference to Microsoft.SqlServer.PipelineHost.dll and to
Microsoft.SqlServer.DTSPipelineWrap.dll to your project.
6. Add
using Microsoft.SqlServer.Dts.Pipeline;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;

25

Writing your custom component

7. Derive your class from PipelineComponent and add DtsTransform
attribute:
[DtsTransform(DisplayName = "My COOL Component")]
public class MyComponent : PipelineComponent
{ ... }
8. Write your transform by overriding virtual functions defined in
PipelineComponent class.
ProvideComponentProperties
Defines component meta data using ComponentMetaData member
Create custom component properties
Create input(s) and/or output(s)
Create custom input /output properties
Declare components connections

PreExecute
In the Pipeline buffer object, columns are accessed by index, not by
lineage ID, so capture the index here.
Given LineageID of column and Buffer Type of input, find the
column index
Anything you can move here from ProcessInput or PrimeOutput

ProcessInput
Modify data in place
Set values for new columns

26

Installing your custom

component
9. For SSIS to know about your component, copy the assembly DLL
to
"C:\Program Files\Microsoft SQL
Server\90\DTS\PipelineComponents"
10. For SSIS to be able to load your component, add the assembly
DLL to GAC
gacutil.exe /I <your_dll>
11. After restarting the Workbench, right-click Toolbox and choose
Add/Remove Items. Switch to Data Flow Items tab, find your
component in the list and check it.
12. Youre done! To debug your component, execute the package
using DTExec and configure Visual Studio debugger to start
DTExec when debugging the project.

27

Building a custom data

flow component

28

Agenda
Overview
Exploring BI Development Studio
Development with SSIS
Writing Custom Workflow Task
Writing Custom Data Flow
Component
BI Integration

29

BI Integration Summary
Design-time integration with Analysis Services
Reuse data sources and data source views
Access design-time UDM metadata from a
project
Access deployed UDM metadata from the server

Run-time integration with Analysis Services

Process UDM within the SSIS control flow

Load a UDM from the SSIS data flow
Load MOLAP without requiring a relational
For difficult sources flat files, or mainframe
via ODBC
For throwaway cubes which do not require
reprocessing

Use an SSIS Data Flow as a Report data source

30

Inherit Data quality

Data cleansing from MSR built into
SSIS
Fuzzy lookup for matching
Fuzzy grouping for de-duplication
Tuning for confidence and similarity

Integrate data mining directly into

your data flow
Smart escalation of issues before
populating the warehouse
Flexible, intuitive, data cleansing
31

Inherit International Support

SSIS support Moving database from
different collation/code page smoothly
ANSI to Unicode
Unicode to ANSI
And ANSI to ANSI (first to should convert to
Unicode)

SSIS also supports Character Map

transform
Uppercase<->Lowercase
Fullwidth <-> HalfWidth
HIRAGANA <-> KATAKANA
SIMPLIFIED_CHINESE <->TRADITIONAL_CHINESE
32

Resources
SQL Server 2005 Mailing List
sqlserver2005@netindonesia.net

www.sqljunkies.com/blogs
www.sqlservercentral.com
www.sqlis.com
MSDN Article: Lesson from Real
Project
http://msdn.microsoft.com/sql/default.aspx?pull=/library/enus/dnsql90/html/SQL05InSrREAL.asp

Or mail me

33

Resources
Download From www.sqlis.com
File Watcher Task
Row Number Transformation
Regular Expression Transformation
Checksum Transformation
Trash Destination Adapter

34