1

Leveraging NoSQL to Implement Real-time Data Architectures

Robert Greene, Senior Principal Product Manager, Oracle
Championing the Oracle NoSQL Database Initiative

Vineet Tyagi, VP Technology, Impetus Technologies

Heads the Innovation Labs

Recorded version available at http://bit.ly/1g6Eaj4

Program Agenda
The Evolving Real-time Data Caching
Architecture

Oracle NoSQL Database, features for

Real-time

An implementation: Real-time trading

with NoSQL

Recorded version available at http://bit.ly/1g6Eaj4

Data Caching Architectures

All data in memory

Move from process space to networked Need cache server and API Bottlenecks: data load, network

Recorded version available at http://bit.ly/1g6Eaj4

Data Caching Architectures

Partial data in memory

Bottlenecks in network and disk I/O

Algorithm dependencies Cache hit/miss (FIFO, LFU, LRU, MRU, ARC, etc.) Data composition, many seeks
5
Recorded version available at http://bit.ly/1g6Eaj4

Data Caching Architectures

All data in memory Distributed
Bottleneck Lock Manager Data Load (composition) Network Complexity APIs (App, Cache, Disk) Expensive to grow

Recorded version available at http://bit.ly/1g6Eaj4

Data Caching Architectures

All data in partitioned cache space
Bottleneck Data load,
Network

Complexity APIs (App,

Cache, Disk)

Expensive to grow

Recorded version available at http://bit.ly/1g6Eaj4

Data Caching Architectures

All data in NoSQL backed storage
Bottleneck Network Reduced
Complexity

API (App,
Cache)

Reduced Cost Disk instead of

memory
8
Recorded version available at http://bit.ly/1g6Eaj4

What is Oracle NoSQL Database?

Non-relational key-value database designed for cost effective simple queries of high volume, velocity & variety data. Provides high performance & availability data storage and retrieval of simple data using a scale-out of servers design.

Recorded version available at http://bit.ly/1g6Eaj4

Architecture

10

Recorded version available at http://bit.ly/1g6Eaj4

Oracle NoSQL Database

Scalable, Highly Available, Key-Value Database Application Features

Application

Application
Application NoSQL DB Driver

Flexible Key-Value Data Model ACID transactions Horizontally Scalable Highly Available Elastic Configuration Simple administration Intelligent Driver Commercial grade software and support

NoSQL DB Driver

Storage Nodes
Datacenter A

Storage Nodes
Datacenter B

Java SE 6 (JDK 1.6.0 u25)+; Solaris or Linux

11

Recorded version available at http://bit.ly/1g6Eaj4

Oracle NoSQL Database

Real-time enabling features

Simple key-value based

No data composition lag

Integrated Caching
Single layer in architecture

WAL transactions
No disk seek time on write

Optimized seek on read

SSD optimized

12

Smart topology driver

Single RPC to data Recorded version available at http://bit.ly/1g6Eaj4

Features Configurable CAP

Greater Flexibility Configurable Durability

13

per operation Configurable Consistency per operation ACID by default Transaction scope is single API call Records share same major key Multiple operations supported
Recorded version available at http://bit.ly/1g6Eaj4

Features Smart Topology Management

Automated Resource Planning Storage nodes have

Application Smart Topology Driver

14

indication of capacity System allocates replicas per storage node Intelligent Master/Replica load balancing Ensures distribution of replicas Efficient use of system resources Reduces operator-caused configuration errors

Recorded version available at http://bit.ly/1g6Eaj4

Features Elastic Cluster Expansion

Smart Topology Increase Data Capacity Add more storage
nodes New shards automatically created Increase Data Throughput More shards = better write throughput More replicas/shard = better read throughput Master Replica
Application NoSQL DB Driver

Master
Replica Replica
Shard-2
StorageNode StorageNode

Replica
Shard-1
StorageNode

15

Recorded version available at http://bit.ly/1g6Eaj4

YCSB Benchmark Commodity Servers

Whats the Big Deal 1.6 billion records 94K insert/sec 25K read/

update/sec Low latency Linear scalability

16

Recorded version available at http://bit.ly/1g6Eaj4

YCSB - Benchmarking with SSDs

Whats the Big Deal

Twitter sees ~500M tweets/day This is 750M a minute Capture twitter activity with 3 commodity servers 1.25M ops/sec 2 billion records 2 TB of data 95% read, 5% update Low latency, High Scalability
Recorded version available at http://bit.ly/1g6Eaj4

17

Global Data At Scale

Recorded version available at http://bit.ly/1g6Eaj4

Challenges in Real Time Analytics

Lack of understanding
Too many silos Operational decisions made manually, big mindset
change

Where to look for use cases ? Business decisions (rules) hard coded in
systems

Operational Planning, Real Time Service

Assurance and Product Optimization
19
Recorded version available at http://bit.ly/1g6Eaj4

Real Time Use Case

Real time and Offline Analysis over
Stock Tick data received from 10 Global Stock Exchanges

Real-time data ingestion at scale Real-time Analysis Support Offline Analysis

20

Recorded version available at http://bit.ly/1g6Eaj4

Scope and Solution

Stock tick data Analytics at Global
Market scale Around 10+ stock exchanges around the world
that provide stock tick data

Easy installation, configuration and

monitoring of Big data cluster

Real-time analytics at high volume Real-time dashboard, key stats, high

ingestion rates
21
Recorded version available at http://bit.ly/1g6Eaj4

Architecture

22

Technology
Oracle NoSQL Database Ankush (Impetus open source tool) StreamAnalytix (Impetus pre-built tech
stack)

23

Kafka Intellicus D3.js Tomcat

Recorded version available at http://bit.ly/1g6Eaj4

Deployment

24

Throughput
RF Storag Shards Replicatio Partition Memory Cache Throughput e n Nodes s Nodes 2 4 4 8 40 Default Default 220k ticks per sec 3 6 4 12 30 Default Default 200k ticks per sec

25

Recorded version available at http://bit.ly/1g6Eaj4

Functional View

26

Recorded version available at http://bit.ly/1g6Eaj4

World Indexes View

27

Recorded version available at http://bit.ly/1g6Eaj4

Regional Stock Exchange View

28

Recorded version available at http://bit.ly/1g6Eaj4

Functional View (Exchange View)

29

Recorded version available at http://bit.ly/1g6Eaj4

Functional View (Stock View)

30

Recorded version available at http://bit.ly/1g6Eaj4

Functional Views (Industry View)

31

Recorded version available at http://bit.ly/1g6Eaj4

Functional Views (Historical Report)

Intellicus view

32

Recorded version available at http://bit.ly/1g6Eaj4

Infrastructure View

33

Recorded version available at http://bit.ly/1g6Eaj4

Infrastructure Views - Ankush

34

Recorded version available at http://bit.ly/1g6Eaj4

Cluster Dashboard

35

Recorded version available at http://bit.ly/1g6Eaj4

Oracle NoSQL Database Monitoring

36

Recorded version available at http://bit.ly/1g6Eaj4

Cluster Metrics

37

Node Monitoring

38

Recorded version available at http://bit.ly/1g6Eaj4

Replica Node Monitoring

39

Recorded version available at http://bit.ly/1g6Eaj4

Operations Available

40

Recorded version available at http://bit.ly/1g6Eaj4

Contact Impetus to Accelerate Your Big Data Journey

For more info and/or a demo - bigdata@impetus.com Gartner BI and Analytics Summit 2014, Las Vegas Meet us and attend our session @impetustech

41

Recorded version available at http://bit.ly/1g6Eaj4

Join NoSQL Database Community

Oracle.com/BigData Twitter https://twitter.com/#!/OracleNoSQL LinkedIn http://www.linkedin.com/groups?gid=4147754 Oracles NoSQL DB blog https://blogs.oracle.com/nosql Oracle Technology Network

http://bit.ly/1f0d8wU
Developer Webcast Series http://bit.ly/1doV2jl

42

Recorded version available at http://bit.ly/1g6Eaj4

Q&A

43

Recorded version available at http://bit.ly/1g6Eaj4

Thank You

44