Lecture - 7

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BIG DATA EXECUTIVE

Implementation, Issues and
TRAINING
Challenges

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

OVERVIEW

Produc
ts
CORE BUSINESS

ORGANIZATIO
N

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

Market
Deman
d

Service
s

Supply
Challenges

BUSINESS CHALLENGE

MARKETING
MIX
- PRODUCT
- PLACE
- PROMOTION
- PRICE

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

COMPLEX CONSUMER

I went to buy a
Dress

BU
I saw a
TBeautiful

pair of Shoe

SO

I bought
this

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ICT AS ENABLER

ORGANIZATIO
N

ICT
ENABLE
R

Decision Making Tools

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BIG
DAT

BIG DATA PATH

A
T
A
DE
L
M
B
Insight
R
A
Facilitate business
O
F
N
T
O
decision from
S
T
H
O
actionable insight
I
N
N
G
T
I
I
Prediction
A
C
S
Algorithmic output
R
A
T
IN Analysis
Analyzing information with
proprietary and models

Filtering
Data

Aggressive filtering and compression of

irrelevant data

Structured, semi-structured,
unstructured

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BIG DATA EVOLUTION

Most generally refers to the practice of
aggregating computing power in a way that
delivers much higher performance than one
could get out of a typical desktop computer or
workstation in order to solve large problems in
science, engineering or business
High

Performance
Computing
(HPC)

Grid
Computing

Cloud
Computing

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

Big
Data

BIG DATA EVOLUTION

Grid computing is a form of

distributed computing in which an
organization uses its existing
computers to handle its own longrunning computational tasks
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BIG DATA EVOLUTION

Cloud
computing
employs
visualization
of
computing
resources
to
run
numerous
standardized virtual servers on
the same physical machine

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BIG DATA EVOLUTION

Refers to huge data sets that are orders of

magnitude larger (volume); more diverse,
including structured, semi structured and
unstructured data (variety); and arriving
faster (velocity) than you or your
organization has had to deal with before
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

VIDEO
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

1
2

WHAT
BIG
IS
DATA
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DEFINITION

VOLUME

Large amounts of
data

VELOCITY

Need to be analyzed
quickly

VARIETY

Various data sources

Big Data is high-volume, high-velocity and

high-variety information assets that demand
cost-effective, innovative forms of information
processing that enable enhanced insight,
decision making, and process automation
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DEFINITION: Business Perspective

Jessica Kirkpatrick
Director of Data Science,
InstaEDU

Big data refers to

using complex
datasets to drive
focus, direction,
and decision
making within a
company or
organization.

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DEFINITION: Data Perspective

IBM

Big data is digital

data of unusual size
(beyond the
capacity of a
traditional
database) or
generated at
spectacular
velocity, such as the
data collected from

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DEFINITION: Technology Perspective

Microsoft

Big data is the term

increasingly used to
describe the process
of applying serious
computing powerthe
latest in machine
learning and artificial
intelligenceto
seriously massive and
often highly complex

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DEFINITION: Technology Perspective

John Arked
Founder and CTO,
Silicon Valley Data Science

Big Data refers to a

combination of an
approach to
informing decision
making with
analytical insight
derived from data,
and a set of enabling
technologies that
enable that insight to
be economically

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BUSINESS

BIG DATA
IMPLEMENTATION, ISSUES AND
CHALLENGES

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

DATA
APPLICATIO
N
TECHNOLO
GY

IMPLEMENTATI
ON
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION

According to a survey
conducted by Capgemini
Consulting on the status of
Big Data implementation,

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION
Status of Big Data Implementation
Source: Capgemini Consulting, Big Data Survey,
November 2014

Not implemented yet, no budget

has been allocated
Not implemented yet, but a budget
has been allocated and we have
identified focus areas

5%
19%

Proof of Concept: we are working

of Proof-of-Concept for selected
use-cases
Partial
Production:
predictive
insights are integrated into some
of our business operations
Full-scale Production: predictive
insights are extensively integrated
into business operations
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

29%
35%
13%
2
2

IMPLEMENTATI
Business
ON
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Business

Gather business requirements

before gathering data
Begin big data implementations by
first gathering, analyzing and
understanding
the
business
requirements; this is the first and
most essential step in the big data
analytics process
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Business

Implementing big data is a

business decision not IT
Analytics solutions are most
successful when approached from
a business perspective and not
from the IT/Engineering end

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATI
Data
ON
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Data

Evaluate data requirements

Whether a business is ready for
big data analytics or not, carrying
out a full evaluation of data
coming into a business and how it
can best be used to the businesss
advantage is advised
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Data

Associate big data with enterprise

data
Enterprises should establish new
capabilities and leverage their prior
investments
in
infrastructure,
platform, business intelligence and
data
warehouses,
rather
than
throwing them away

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATI
Application & Technology
ON
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Application &

Technology

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

IMPLEMENTATION: Hadoop Components

Oozie
(Workflow
Monitoring)

Chukw
(Monitoring)
a

Hive

Pig

(SQL)

(Data Flow)

Flume
(Monitoring)

Maho
(Machine
ut
Learning)

ZooKeepe
r
(Management)

Avro
(RPC,
Serialization)

Sqoo
(RDBMS
p

Data Access

Connector)

MapReduce

YARN

(Cluster Management)

(Cluster & Resource

Management)

HDFS

HBase

(Distributed File
System)

(Column DB Storage)

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

Data
Management

Data
Processing

Data
Storage

3
2

ISSUES
CHALLENG
&
ES
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

3
3

ISSUES AND CHALLENGES

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES

According
to
published by
about the main
to implementing

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

a
study
Accenture
challenges
big data,

ISSUES AND CHALLENGES

What are the main challenges to implementing big data?
Security

]
Lack of talent to implement big ]
Budget

data
Lack of talent to run big data
and
analytics on an ongoing basis
Integration with existing
systems
Procurement limitations on big data
vendors
Enterprise not ready for big
data

]
]
]
]

51
%
47
%
41
%
37
%
35
%
33
%
27
%

Source: Accenture Big Success with Big Data Survey, April

2014

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES &
CHALLENGES
Business

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

3
3

ISSUES AND CHALLENGES: Business

UNDERSTADING
BIG DATA

THE BIG DATA

TALENT GAP

MEETING THE NEED

FOR SPEED

GOVERNANCE

BUDGET

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

3
3

ISSUES AND CHALLENGES: Business

UNDERSTANDING THE NEED OF BIG DATA

Organization
or
companies
have
difficulty identifying the right data and
determine how to best use it. Building
data-related business cases often
means thinking outside of the box and
looking for revenue models that are
very different from the traditional
business
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Business

THE BIG DATA TALENT GAP

The talent gap is real. Consider these statistics:
According to analyst firm McKinsey & Company,

By 2018, the United States alone could face

a shortage of 140,000 to 190,000 people with
deep analytical skills as well as 1.5 million
managers and analysts with the know-how to
use the analysis of big data to make effective
decisions

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Business

MEETING THE NEED FOR SPEED

In todays hypercompetitive business

environment, companies not only have
to find and analyse the relevant data
they need, they must find it quickly.
The challenge is going through the
sheer volumes of data and accessing
the level of detail needed, all at a high
speed.
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Business

GOVERNANCE

How will you govern your organization's

information assets in support of your
enterprise information management
(EIM) goal? Are there new expectations
for
information
quality
and
management?

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Business

BUDGET

How can you deliver access to big data

in a rapid and cost-effective way to
support better decision-making?

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES &
CHALLENGES
Data

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Data

PREDICTIVE
ANALYTICS

DATA
BROKER

BEHAVIORAL
ANALYTICS

DATA
INTERPRETATION

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Data

PREDICTIVE ANALYTICS

How are you using data for predictive

and real-time analysis across various
business domains? How can you use
unstructured enterprise information
and data to drive a better client
experience?

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Data

DATA BROKER

By 2017, more than 30% of enterprise

access to broadly based big data will be
via intermediary data broker services,
serving context to business decisions

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Data

BEHAVIORAL ANALYTICS

How will you tap into complex data sets

to create new models to drive business
outcomes,
decrease
costs,
drive
innovation
or
improve
customer
satisfaction?

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Data

DATA INTERPRETATION

What new business analysis can be

drawn from your data? How will IT help
support insight discovery and new
information trends? You need to know
which data to integrate for new product
innovation.

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES &
CHALLENGES
Application

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Application

DISPLAYING
MEANINGFUL RESULTS

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

CONFINE WITH
ONE APPLICATION

ISSUES AND CHALLENGES: Application

DISPLAYING MEANINGFUL RESULTS

Plotting points on a graph for analysis

becomes difficult when dealing with
extremely large amounts of information
or a variety of categories of information

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Application

CONFINE WITH ONE APPLICATION

Data processing in place should not be

rigid to specific application, it must be
agile for speed and accuracy.

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES &
CHALLENGES
Technology

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

5
5

ISSUES AND CHALLENGES: Technology

TECHNOLOGY
EVOLVING
EXTREMELY FAST

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

TOOLS

55

ISSUES AND CHALLENGES: Technology

TECHNOLOGY EVOLVING EXTREMELY FAST

The technology landscape in the data

world is evolving extremely fast.
Leveraging data means working with a
strong and innovative technology
partner that can help create the right IT
architecture that can adapt to changes
in the landscape in an efficient manner.

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ISSUES AND CHALLENGES: Technology

TOOLS

How will you link these new sources of

diverse data? You need to plan the
impact on your data center. Have you
identified the processes, tools and
technologies you need to support big
data in your enterprise?

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

BENEFIT
BIGOF
DATA
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

5
5

BENEFIT

WAYS BIG DATA

WILL CHANGE
EVERY
BUSINESS

Data will become an

asset
to
every
business
Big data will enable
companies to collect
better
market
and
customer intelligence
It will improve internal
efficiency
and
operations
Allow companies to
improve the customer
experience and build
big data into their
product offering

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

connected everything

Bring Your Own

Internet
of Things
Device
Gadget
di
sp

la

da

ta

Cloud
Computing

d
co ata
ll e
cte
d

data
collected

Business
Dashboard
Intelligence

Big Data

ed
z
y
al
an data

connected everything
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

connected everything

e
vi c
r
se
e
us

connected
everything

SUCCESS
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

STATISTICS

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

SUCCESS IN BIG DATA

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ELK
STACK
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

ELK STACK

Elasticsearch

Logstash

Kibana

Elasticsearch is a schema-less database that

has powerful search capabilities and is easy to
scale horizontally
Logstashis a tool for log data intake,
processing, and output. (system logs, web
server logs, error logs, and app logs)

Kibana is a web-based data analysis and

dashboarding
tool
for
Elasticsearch.
It
leverages Elasticsearchs search capabilities to
visualize your big data in seconds

BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

THE END
Q&A
BIG DATA EXECUTIVE TRAINING: Implementation, Issues and

7
6