Telecom Solutions Lab

BAO & Big Data Overview

Applied to Real-time Campaign
GSE
Joel Viale
Telecom Solutions Lab Solution Architect

2011 IBM Corporation

Telecom Solutions Lab

Agenda
BAO & Big Data - Overview
Customer use-cases
Live Prototypes:
Streams for Real-time Campaign
Big Insights for Web Log Analysis

2011 IBM Corporation

Telecom Solutions Lab

The Business Analytics Journey: Transform data into actionable

insight!
Transactional
& Collaborative
Applications

Business Analytics
Applications

Analyze

Integrate

Big Data

Manage

Master
Data
Cubes

Streams

Data
Data
Warehouses

Content

Internal & External

Information Sources

Streaming
Information

Govern
Quality

Lifecycle

Security &
Privacy

2011 IBM Corporation

Telecom Solutions Lab

What is Big Data? How could it impact your organization?

Big Data technologies describe a new

generation of technologies and architectures,
designed to economically extract value from
very large volumes of a wide variety of data,
by enabling velocity capture, discovery and/or
analysis.
- Matt Eastwood, IDC
http://www.tweetdeck.com/twitter/matteastwood/~hHgsU

2011 IBM Corporation

Telecom Solutions Lab

The Big Data Challenge

Manage and benefit from massive and growing amounts of data
Handle varied data formats (structured, unstructured, semistructured) and increased data velocity

Exploit BIG Data in a timely and cost effective fashion

COLLECT

Collect

INTEGRATE
Integrate

MANAGE

Manage

ANALYZE
Analyze

2011 IBM Corporation

Telecom Solutions Lab

Bring Together a Large Volume and Variety of Data to Find New

Insights
Leverage Multi-channel customer
sentiment and experience
analysis to upsell in real time

Detect life-threatening
conditions at hospitals in time
to intervene
Predict weather patterns to plan
optimal wind turbine usage, and
optimize capital expenditure on
asset placement

Make risk decisions based on realtime transactional data

Identify criminals and threats from

disparate video, audio, and data
feeds
6

2011 IBM Corporation

Telecom Solutions Lab

The Solution IBMs Big Data Platform

Bring together any data source, at any velocity, to generate insight

Analyzing a variety of data at
enormous volumes

Insights on streaming data

Large volume structured data
analysis

IBM Big Data

Platform
Variety
Velocity
7

Volume

2011 IBM Corporation

Telecom Solutions Lab

2 ways of creating insights: Streaming and Storing Big Data

Real time analysis of data-in-motion
Structured or unstructured

Data

Queries

Results

Analytic operations on streaming data in real-time

b) streaming data

Managing and analyzing Internet-scale volumes

Structured or unstructured data
Based on Googles MapReduce technology
Inspired by Apache Hadoop; compatible with its ecosystem and distribution
Well-suited to batch-oriented, read-intensive applications
Integrated Text Analytics
8

2011 IBM Corporation

Telecom Solutions Lab

How to deal with Big Data: Split a big job into smaller pieces
Highly Massive Parallel Processing (MPP)
-Distributed File System (HDFS)
-Map/Reduce

9
9

2011 IBM Corporation

Telecom Solutions Lab

InfoSphere Streams enables highly scalable stream processing

InfoSphere Streams provides
a programming model for defining data flow graphs consisting of data sources
(inputs), operators, and sinks (outputs)
controls for fusing operators into processing elements (PEs)
infrastructure to support the composition of scalable stream processing
applications from these components
deployment and operation of these applications across distributed x86
processing nodes,
when scaled-up processing is required

10

2011 IBM Corporation

Telecom Solutions Lab

Big Data complements traditional warehouse & analytics

infrastructure
Moving Beyond the Traditional Warehouse
Traditional
Warehouse

Streams

Traditional /
Relational
Data Sources

Database & At-Rest Data Results

Warehouse Analytics

Non-Traditional /
Non-Relational
Data Sources
Ultra Low
Latency Results

In-Motion
Analytics

Non-Traditional/
Non-Relational
Data Sources

Interne
t Traditional/Relatio
Scale nal Data Sources
nal Data Sources

11

Internet Scale

InfoSphere
Big
Insights
Data Analytics,
Data Operations
& Model
Building

Results

2011 IBM Corporation

Telecom Solutions Lab

Agenda
BAO & Big Data - Overview
Customer use-cases
Live Prototypes:
Streams for Real-time Campaign
Big Insights for Web Log Analysis

12

2011 IBM Corporation

Telecom Solutions Lab

Examples of Streams use-cases

Stock market
Natural Systems
Wildfire management
Water management

Impact of weather on
securities prices
Analyze market data at
ultra-low latencies

Law Enforcement, Defense &

Cyber Security
Real-time multimodal surveillance
Situational awareness
Cyber security detection

Transportation
Fraud prevention

Intelligent traffic management

Detecting multi-party fraud

Real time fraud prevention

Manufacturing

e-Science

Process control for

microchip fabrication

Space weather prediction

Detection of transient events
Synchrotron atomic research

Health & Life Sciences

Neonatal ICU monitoring
Epidemic early warning system
Remote healthcare monitoring

13

Telephony

CDR processing
Social analysis
Churn prediction
Geomapping

Other

Smart Grid
Text Analysis
Whos Talking to Whom?
ERP for Commodities
FPGA Acceleration
2011 IBM Corporation

Telecom Solutions Lab

Streams in Telecommunications
Data in motion
Dropped Calls

CDRs

Outgoing International
Calls
Call Duration
Extra Call

Billing

Data Aggregated at
Single Customer
Level
st

Millions of
events per
second

Invoice Issued
5

Contract Expiration

la
the
la s t
ls in
l
o
a
the
m
C
o
n
i
r
d
rd P
lls
pe hou
rop
l Ca
as e
3D
n-b ationa
o
i
t
a
n
Loc g inter day
o in
g
t
Ou

CRM
Location
Account Mgt

100 SMSs sent in 1

hour on pre500$ top-up
paid

Entered new cell

No ga
me do
w
500 F
ailed S m nload in la
st 30
MSinDutes
eli
minute veries last
10
s

New Top-Up
5 minutes left on prepaid

Internet
Network

Ag

Microsecond
Latency

MDM
EDW
14

gre
gat
ed
Ag
Da
gre
ta
gat
ed L Reco
Ag
Da ev
gre
ta e rds a
gat
tS
edLe Rlec
e
o
Vove
ucl h rds a rvice
tC
er R
ech ell
arg
e

Data Aggregated at
Cell or Service
Level

Campaign Mgt
Real-time
Promo

Fraud Detection

Service
Assurance
Network
Monitoring
Real-time
Monitoring
(Voucher
Recharge &
Service Usage)
2011 IBM Corporation

Telecom Solutions Lab

CDR cleansing and pre-processing

The Pain Point:
A CSP has 6 Billions CDRs per day!
As much as 500,000 per second peak rate
Kept up, but re-processing meant waiting for
nights/weekends to catch up
The Solution:
InfoSphere Streams to clean and pre-process
CDRs
De-duplication of CDR against 15 days of data
(90B CDRs)
Offloading CDRs processing to Streams platform
increased the performance of their warehouse for
other analytics
Single platform for mediation and real time
analytics reduced IT complexity
2011 IBM Corporation

Telecom Solutions Lab

The Pain Point

Insight

ee
ittivv
iipp
r
r
cc
s
s
ee
pprr

Information

Data

ac
tiv
e

Business flexibility & responsiveness

Real-time marketing campaign

100M CDRs per day from SMS from

25M subscribers, only used to send
bills to customers
The Solution

InfoSphere Streams to create realtime marketing promotions and
monetize the CDRs

Business value
2011 IBM Corporation

Telecom Solutions Lab

Typical Telco Use-Cases with InfoSphere Streams

ETL-like for massive data to off-load traditional ETL and Warehouse

Mediations

Prediction and prevention of customer churn
In-motion analytics can help identify group leaders

Real Time context sensitive promotions

Location based promotions

Customer minutes usage based promotions
Customer smart phone browsing pattern based promotions
Network equipment utilization based promotions

Real Time & pro-active network monitoring

Real Time call traffic & revenue monitoring

SMS spam filtering

Fraud detection
17

2011 IBM Corporation

Telecom Solutions Lab

Typical Telco Use-Cases with InfoSphere Big Insights

CDR Use Cases:

User behavior analysis

Prediction on the basis of Customer churn

Service association analysis
Social Media Analytics: Get insights, improve campaign, influence opinions
Fraud Detection: Monitoring/Mining transaction logs to detect fraud activities
Recommendation Engines: Improving user experience and likelihood of purchase
Network Traffic Logging: Network optimization & fault detection
Advertising Optimization: in support of advertising based models
Server Logs: Fault detection / Performance related analysis
Email and Email Logs:

Consumer email analysis & decision system
18

2011 IBM Corporation

Telecom Solutions Lab

Big Insights at the core of Cognos Consumer Insight

Business Drivers
Competitive Analysis

Corporate
Corporate Reputation
Reputation

Campaign
Campaign Effectiveness
Effectiveness

Source Areas

Customer
CustomerCare
Care
Product Insight

Product Capabilities

FACEBOOK
FACEBOOK

COMPREHENSIVE
ANALYSIS

BLOGS
BLOGS

Keyword Search

Dimensional Navigation

Drill Through to Content

SENTIMENT

Dimensional Analysis

Filtering

Voice

DISCUSSION
DISCUSSION FORUMS
FORUMS

TWITTER
TWITTER

NEWSGROUPS
NEWSGROUPS

AFFINITY ANALYTICS

Relationship Tables

Relationship Matrix

Relationship Graph

EVOLVING TOPICS

Relevant Topics

Associated Themes

Ranking and Volume

MULTILINGUAL
MULTILINGUAL

19

2011 IBM Corporation

Telecom Solutions Lab

Agenda
BAO & Big Data - Overview
Customer use-cases
Live Prototypes:
Streams for Real-time Campaign
Big Insights for Web Log Analysis

20

2011 IBM Corporation

Telecom Solutions Lab

Real-time Campaign Solution Architecture

Advanced BI, Reporting & Real-time

Cognos Suite
Monitoring
Real-time Analytics of
InfoSphere Streams
Data-in-Motion
Enterprise Marketing
Management
Unica Suite

Advance
d
Predictiv
SPSS
e
Analytics

High-Performance
DataNetezza
Warehouse
Appliance
2011 IBM Corporation

Telecom Solutions Lab

Outbound campaign, enhanced with Real-time Monitoring and

advanced predictive analytics
Events
Product
Manager
Real-time Monitoring
Marketing
Manager

Detect situations and

patterns in service usage.
Determine the need of a
campaign

VP of
Marketin
g

Analytics

Collaboration

Identify
Target
Customers
and
Offering

Predictive Analytics
Segment Customers based
on Churn Prediction,
likelihood of campaign
response, LTCV, etcO

Real-time xDRs
Processing

Customer

Collect, correlate and

aggregate data records in
real-time
Network

22

Collect
and
aggregate
xDRs

Content

Monitoring

Monitor key KPIs

Sales/Usage increase
% of campaign response

Monitor &
Evaluate
Campaig
n

Execute
Campaig
n

Create
Campaig
n

Define
Marketing
Strategy

Rules

Approve
Campaig
n

Delivery
Channel

Presence & Rules based channel

selection
Send campaign message on appropriate
channel (including social networks),
based on customer preferences and
2011 IBM Corporation
availability/presence

Telecom Solutions Lab

Real-time campaign, based on network events

CDRs

Dropped Calls
Filtering
Dropped Calls
Aggregation
Per Subscriber

Real-time Event Detection

InfoSphere Streams correlate
multiple data records in realtime down to the customer
level.
Here: detects a number of
dropped calls in a certain
time

CDR
Feed

InfoSphere Streams
Real-time event detection

Event Based
Campaign
Processing

Manage
Campaign
Delivery

Event-based Campaign
The event-based campaign
processing is triggered: a
campaign is executed in Unica
Campaign.
23

Monitor
Campaign
Execution

Message Delivery
The campaign message is
delivered to the customer
on the appropriate channel

Evaluate
Campaign
Effectiveness

Campaign monitoring
Monitor campaign execution
and analyze effectiveness
2011 IBM Corporation

Telecom Solutions Lab

Agenda
BAO & Big Data - Overview
Customer use-cases
Live Prototypes:
Streams for Real-time Campaign
Big Insights for Web Log Analysis

24

2011 IBM Corporation

Telecom Solutions Lab

The Sample Outdoors Company

has many visitors a year to their
website and only a subset
resulted in purchases. They
would like to know what is
happening to the other visits?

25

2011 IBM Corporation

Telecom Solutions Lab

Every time the user browses a page, the click-stream data is logged

User goes to
web site

Search
for
item
Add
item
to cart
User
checks
out
26

2011 IBM Corporation

Telecom Solutions Lab

Click Stream Data Analysis for Customer Insights

27

Web logs capture click streams of user

Basic analysis of web logs can provide valuable insights to web site usage
and user behaviors
How many users going to my web site?
What pages and products are people looking at?

Additional sessionization and aggregation analysis help to detect valuable

click patterns and provide insights for:
Improving customer service
Ads promotion and sales improvement
Detection of incidents and errors

2011 IBM Corporation

Telecom Solutions Lab

Big Insights Web Log Analysis Solution Overview

Web Server
Logs

28

BigInsights

Analytics

2011 IBM Corporation

Telecom Solutions Lab

Solution Architecture for Web Log Analysis

Aggregated reports on user
shopping behavior

JAQL
Text Analytics (SystemT)

Big Insights (HDFS, )

O

Raw Logs

29

cluster

2011 IBM Corporation