From Oracle to MongoDB

A real use case at Telefnica PDI

Pablo Enfedaque pev@tid.es 06.10.2012

Content
Introduction
01

Telefnica PDI. Who? Personalisation Server. Why? What?

The SQL version

02

Data model and architecture Integrations, problems and improvements

The NoSQL version

03

Data model and architecture Performance boost The bad

Conclusions

04

Conclusions Personal thoughts

01

Introduction Ttulo del captulo Mximo 3 lneas

01

Telefnica PDI. Who?

Telefnica

Fifth largest telecommunications company in the world Operations in Europe (7 countries), the United States and Latin America (15 countries)

Telefnica Digital
Web and mobile digital contents and services division

Product Development and Innovation unit

Formerly Telefnica R&D Product & service development, platforms development, research, technology strategy, user experience and deployment & operation Around 70 different on going projects at all time.

Telefnica PDI

01

Personalisation Server. What?

User proling system Machine learning Recommendations Customers prole storage

Telefnica PDI

01

Opt-in and prole module. Why?

Users data, prole and permissions, was scattered across different

storages

IPTV service Mobile service Music tickets service Location based offers

Gender Film and music preferences Permission to contact by SMS? Gender Address Music preferences Address Permission to contact by SMS? So you want to know my address AGAIN?!

Telefnica PDI

01

Opt-in and prole module. Why?

Users data, prole and permissions, was scattered across different

storages

IPTV service Mobile service Music tickets service Location based offers

Gender Film and music preferences Permission to contact by SMS? Gender Address Music preferences Address Permission to contact by SMS?

Telefnica PDI

01

Opt-in and prole module. Why?

Provide a module to become master

customers data storage

IPTV service Mobile service Music tickets service Location based offers

Gender Film and music preferences Permission to contact by SMS? Address

Telefnica PDI

01

Opt-in and prole module. What?

Features:
Flexible prole denition, classied in services Prole sharing options between different services Real time API Supplementary offline batch interface Authorization system High availability Inexpensive solution & hardware

Telefnica PDI

02

The SQL captulo Ttulo del solution Mximo 3 lneas

02

Data model
Services, users and their prole

Services dened a set of attributes (their prole), with default value and data type Users were registered in services Users dened values for some of the services attributes Each attribute value had an update date to avoid overwriting newer
changes through batch loads

Telefnica PDI

11

02

Data model
Services prole sharing matrix

Services could access attributes declared inside other services There were sharing rights for read or read and write The user had to be registered in both services

Telefnica PDI

12

02

Data model
Authorization system

Everything that could be accessed in the PS was a resource Roles dened access rights (read or read and write) of resources Auth users had roles Roles could include other roles

Telefnica PDI

13

02

Data model
Bonus features!

Multiple IDS:

Users prole could be accessed with different equivalent IDs depending on the service Each user ID was dened by an ID type (phone number, email, portal ID, hash) and the ID value

Telefnica PDI

14

02

High level logical architecture

Everything running on Red Hat EL 5.4 64 bits

Telefnica PDI 15

02

High level logical architecture

Everything running on Red Hat EL 5.4 64 bits

Telefnica PDI 16

02

Integration
Planned integration

PS replaces all customers prole and

permissions DBs

All systems access this data through

PS real time API

In special cases, some PS-consumers

could use the batch interface.

The same way new services could be

added quite easily

Telefnica PDI

17

02

Integration
Problems arise

Budget restrictions: adapt all services

to use the API was too expensive

Keep independent systems DBs and

synchronize PS through batch

Use DBs built-in massive extraction

feature to generate daily batch les

However in most cases those DBs

Provide full daily snapshots!

were not able to generate Delta (only changes) extractions

Telefnica PDI

18

02

First version performance

Ireland

1.8M customers, 180 prole attributes, 6 services Sizes

Tables + indexes size: 65Gb 30% of the size were indexes

Batch
Full DWH customers prole import: > 24 hours Delta extractions: 4 - 6 hours Loads and extractions performance proportional to data size

API:
Response time with average traffic: 110ms
Telefnica PDI 19

03

The SQL captulo Ttulo del solution Second 3 lneas Mximo version

03

Second version
High level logical architecture

New approach: batch processes access directly DB

Telefnica PDI 21

03

Second version
Batch processes

Batch processes had to

Validate authentication and authorization Verify user, service and attribute existence Check equivalent IDs Validate sharing matrix rights Validate values data type Check the update date of the existing values

Telefnica PDI

22

03

Second version
DB Batch processing

BAs ur D O

Telefnica PDI

23

03

Second version
New DB-based batch loading process

Preprocess incoming batch le in BE servers

Validate format, services and attributes existence and values data types Generate intermediate le with structure like target DB table

Load intermediate le (Oracles SQL*Loader) to a temporal table Switch DB to deferred writing, storing all incoming modications Merge temporal table and nal table, checking values update date Replace old users attributes values table with merge result Apply deferred writing operations
Telefnica PDI 24

03

Second version
New batch extraction process

Generate a temporal DB table with format similar to nal batch le.

Two loops over users attributes values table required:
Select format of the table; number and order of columns / attributes Fill the new table

Loop the whole temporal table for nal formatting (empty elds) From batch side loop across the whole table (SELECT * FROM ) Write each retrieved row as a line in the resulting le

Telefnica PDI

25

03

Second version performance

Ireland performance requirements

Batch time window: 3:30 hours

Full DWH load Two Delta loads Three Delta extractions

API:
Ireland requirement: < 500ms

Telefnica PDI

26

03

Second version performance

Ireland

1.8M customers, 180 prole attributes, 6 services Sizes Batch

Tables + indexes size: 65Gb 30% of the size were indexes Temporal tables size increases almost exponentially: 15Gb and above Intermediate le size: from 700Mb to 7Gb Full DWH customers prole import: 2:30 hours Delta extractions: 1:00 hour Loads performance worsened quickly (almost exp): 6:00 hours Extractions performance proportional to data size Concurrent batch processes may halt the DB

API:
Telefnica PDI

Response time with average traffic: 80ms Response time while loading was unpredictable: >300ms
27

04

The SQL captulo Ttulo del solution Third version Mximo 3 lneas

04

Third version
Speed up DB Batch processes

ain) g s (a A r DB Ou
Telefnica PDI 29

04

Third version
New (second) DB-based batch loading process

Minor preprocessing of incoming batch le in BE servers

Just validate format No intermediate le needed!

Load validated le (Oracles SQL*Loader) to a temporal table Loop the temporal table merging the values into nal table, checking
values update date and data types
Use several concurrent writing jobs

Store results on real table, no need to replace! No deferred writing!

Telefnica PDI 30

04

Third version
Enhancements to extraction process

Optimized loops to generate temporal output table.

Use several concurrent writing jobs We achieved a speed-up of between 1.5 and 2

Loop the whole temporal table for nal formatting (empty elds) Download and write lines directly inside Oracles sqlplus No SELECT * FROM query from Batch side!

Telefnica PDI

31

04

Third version performance

Ireland

1.8M customers, 180 prole attributes, 6 services Sizes

Tables + indexes size: 65Gb 30% of the size were indexes Temporal tables: 15Gb

Batch
Full DWH customers prole import: 1:10 hours (vs. 2:30 hours) Three Delta extractions: 2:15 hours (vs. 3:00 hours) Loads and extractions performance proportional to data size Concurrent batch processes not so harmful s DBA Our API: Response time with average traffic: 110ms Response time while loading: 400ms
Telefnica PDI 32

F**K YEAH

04

Third version performance

United Kingdom

25M customers, 150 prole attributes, 15 services Sizes

Tables + indexes size: 700Gb 40% of the size were indexes

Batch
Two Delta imports: < 2:00 hours Two Delta extractions: < 2:00 hours Loads and extractions performance proportional to data size

API:
Response time with average traffic: 90ms

s DBA Our

F**K YEAH

Telefnica PDI

33

04

Third version performance

Ireland DB size Full DWH load Three Delta exports Batch stability API response time API while loading United Kingdom DB size Two Delta loads Three Delta exports API response time 3rd version 65Gb + 15Gb (temp) 1:10 hours 2:15 hours 110ms 400ms 3rd version 700Gb < 2:00 hours < 2:00 hours 90ms
34

2nd version 65Gb + > 15Gb 2:30 hours 3:00 hours 110ms Unpredictable

Stable, linear Unstable, exponential

s DBA Our

F**K YEAH

Telefnica PDI

04

Third version performance

DB stats

20 database tables API: several queries withup to 35 joins and even some unions Authorization: 5 joins to validate auth users access Batch:
Load: 1700 lines of PL/SQL Extraction: 1200 of PL/SQL

Telefnica PDI

35

04

Mission completed?

Telefnica PDI

36

04

Third version performance

Mexico

20M customers, 200 prole attributes, 10 services Mexico time window: 4:00 hours
Full DWH load! Additional Delta feeds loads At least two Delta extractions

BAs D Our

Telefnica PDI

37

05

The NoSQL solution Ttulo del captulo Mximo 3 lneas

05

MongoDB Data Model

Services and their prole + sharing matrix
{ _id : 7, service_name : "root", id_type : 1, default_values: false, attrib_id = service_id * 10000 + num attribs + 1 owned_attribs : [ { attrib_id : 70005, attrib_nane : marketing.consent", attrib_data_type : 1, attrib_def_value : "no", attrib_status : 1 }, ... attrib_id = service_id * 10000 + num attribs + 1 ], shared_attribs : [ {attrib_id : 20144, sharing_mode : 0}, ... ] }

Telefnica PDI

39

05

MongoDB Data Model

Users and their prole + multiple IDs
{ Equivalent ID document: _id : "011234" services_list : _id = id type + user ID { [ _id : 05abcd" { ue : "011234" service_id : 1, } reg_date : {"$date" : 1318040693000} }, _id = id type + user ID ... ], user_values : attrib_id = service_id * 10000 + num attribs + 1 [ { attrib_id : 10140, attrib_value : "Open", update_date : {"$date" : 1317110161000} }, ... ]

}
Telefnica PDI 40

05

MongoDB Data Model

Authorization system
AUTH USERS COLLECTION: { _id: "admin" auth_pswd: XXX", auth_roles: ['PS_ADMIN_ROLE, ], auth_uris: [ {uri_path: "/**", method: 'R'}, {uri_path: "/stats/**", method: 'RW'}, {uri_path: "/kpis/**", method: IMPORT'}, ... ] } ROLES COLLECTION: { _id: 'PS_ADMIN_ROLE', roles_resources: [ { resource_id: "admin.**, method: 'R' }, { resource_id: "stats.**, method: 'IMPORT' }, ... ] } RESOURCES COLLECTION: { _id: "admin.**", role_uri: "/**" }

Replicate uris (from resources) and methods (from roles)

Telefnica PDI

41

05

MongoDB Data Model

DB stats

Only 5 collections API: typically 2 accesses (services and users collections) Authorization: access only 1 collection to grant access Batch: all processing done outside DB

Telefnica PDI

42

05

NoSQL version
High level logical architecture

Everything running on Red Hat EL 6.2 64 bits

Telefnica PDI 43

05

NoSQL version performance

Ireland (at PDI lab)

1.8M customers, 180 prole attributes, 6 services Sizes

Collections + indexes size: 20Gb (vs. 65Gb) < 5% of the size are indexes (vs. 30%)

Batch
Full DWH customers prole import: 0:12 hours (vs. 1:10 hours) Three Delta extractions: 0:40 hours (vs. 2:15 hours) Loads and extractions performance proportional to data size Concurrent batch processes without performance affection

API:
Response time with average traffic: < 10ms (vs. 110ms) Response time while loading: the same High load (600 TPS) response time while loading: 300ms
Telefnica PDI 44

05

NoSQL version performance

United Kingdom (at PDI lab)

25M customers, 150 prole attributes, 15 services Sizes

Collections + indexes size: 210Gb (vs. 700Gb) < 5% of the size were indexes

Batch
Two Delta imports: < 0:40 hours (vs. 2:00 hours) Loads and extractions performance proportional to data size

Telefnica PDI

45

05

NoSQL version performance

Mexico

20M customers, 200 prole attributes, 15 services Sizes

Collections + indexes size: 320Gb Indexes size: 1.2Gb

Batch
Initial Full import (20M, 40 attributes): 2:00 hours Small Full import (20M, 6 attributes): 0:40 hours

API:
Response time with average traffic: < 10ms (vs. 90ms) Response time while loading: the same High load (500 TPS) response time while loading: 270ms

Telefnica PDI

46

04

NoSQL version performance

Ireland DB size Full DWH load Three Delta exports API while loading API 600TPS + loading United Kingdom DB size Two Delta loads Mexico DB size Initial Full load (40 attr) Daily Full load (6 attr) API while loading NoSQL version 20Gb 0:12 hours 0:40 hours < 10ms 300ms NoSQL version 210Gb < 0:40hours NoSQL version 320Gb 2:00 hours 0:40 hours < 10ms 270ms 47
BAs D Our

SQL version 80Gb 1:10 hours 2:15 hours 400ms Timeout / failure SQL version 700Gb < 2:00 hours

API 500TPS Telefnica PDI

+ loading

05

Mission completed?

Telefnica PDI

48

05

The bad

Batch load process was too fast

To keep secondary nodes synched we needed oplog of 16 or 24Gb We had to disable journaling for the rst migrations

Labels of documents elds take up disc space

Reduced them to just 2 chars: attribute_id -> ai

Respect the unwritten law of at least 70% of size in RAM Take care with compound indexes, order matters
You can save one index or you can have problems Put most important key (never nullable) the rst one

DBAs whining and complaining about NoSQL

Telefnica PDI 49

If we had enough RAM for all data, Oracle would outperform MongoDB

05

The ugly

Second migration once the PS is already running

Full import adding 30 new attributes values: 10:00 hours Full import adding 150 new attributes values: 40:00 hours

Increase considerably documents size (i.e. adding lots of new values

to the users) makes MongoDB rearrange the documents, performing around 5 times slower
Thats a problem when you are updating 10k documents per second

Solutions?
Avoid this situation at all cost. Run away! Normalize users values; move to a new individual collection Prealloc the size with a faux eld
You could waste space!

Load in new collection, merge and swap, like we did in Oracle

Telefnica PDI 50

06

Conclusions Ttulo del captulo Ttulo del captulo Mximo lneas Mximo 3 3 lneas

06

Conclusions & personal thoughts

Awesome performance boost

But not all use cases t in a MongoDB / NoSQL solution!

New technology, different limitations Fear of the unknown

SSDs performance? Long term performance and stability?

Python + MongoDB + pymongo = fast development

I mean, really fast

MongoDB Monitoring Service (MMS) 10gen people were very helpful

Telefnica PDI 52

06

Questions?

Telefnica PDI

53

0X

SQL Physical architecture

Scale horizontally adding more BE or DB servers or disks in the SAN Virtualized or physical servers depending on the deployment
Telefnica PDI 55

0X

MongoDB Physical architecture

MongoDB arbiters running on BE servers Scale horizontally adding more BE servers or disks in the SAN Sharding may already be congured to scale adding more replica sets
Telefnica PDI 56