BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come) Draft

1
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come) Draft

2
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come) Draft

3
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

4-Node Hadoop Cluster

SAP HANA

4
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Query 1 runs against SAP HANA and takes 1 second to run Query2 runs against Hadoop/Hive table and takes 66 seconds to run (which is normal for Hadoop)

When executing above Query2 under the covers in Hadoop a map-reduce job get executed to get the data from Hive table.

5
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Hadoop Data: Year 2012

6
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

SAP HANA Data: Year 2011

7
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Merged HANA and Hadoop Data

8
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Details steps for connecting from SAP to Hadoop

Step 1 - In Hadoop, created Hive table 2012_weblog by loading apache web log file which I send you earlier
Local web log [root@n1 test]# ls -ltr 2012_weblog* -rw-r--r-- 1 root root 12170 Sep 10 11:05 2012_weblog.txt [root@n1 test]#

Copied to Hadoop fs
[root@n1 test]# hadoop fs -ls /tmp/test/2012_weblog* Found 1 items -rw-r--r-- 3 root hdfs 12170 2012-09-10 11:10 /tmp/test/2012_weblog.txt

and created table by parsing the file.

[root@n1 test]# hive WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-0.8.1-cdh4.0.1.jar!/hive-log4j.properties Hive history file=/tmp/root/hive_job_log_root_201209121539_740097618.txt hive> show tables; OK 2012_weblog Time taken: 3.648 seconds hive> select * from 2012_weblog; OK 127.0.0.1 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 301 28698 NULLMozilla 78.29.15.137 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 304 85930 http://localhost/phpmyadmin/ Mozilla 58.218.199.250 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 200 77138 http://localhost/phpmyadmin/ Mozilla 127.0.0.1 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 304 16940 http://localhost/phpmyadmin/navigation.php?lang=en-utf-8&convcharset=utf8&collation_connection=utf8_general_ci&token=697ef7c7f1c23069ba45d8d397722c04&phpMyAdmin=1nb4g50nb43hbftn krrl88dnk5hs3q5o Zafari 66.249.71.204 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 304 506 http://localhost/phpmyadmin/navigation.php?lang=en-utf-8&convcharset=utf8&collation_connection=utf8_general_ci&token=697ef7c7f1c23069ba45d8d397722c04&phpMyAdmin=1nb4g50nb43hbftn krrl88dnk5hs3q5o IE 220.181.108.180 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 304 955 http://localhost/phpmyadmin/navigation.php?lang=en-utf-8&convcharset=utf8&collation_connection=utf8_general_ci&token=697ef7c7f1c23069ba45d8d397722c04&phpMyAdmin=1nb4g50nb43hbftn krrl88dnk5hs3q5o IE 95.108.150.235 NULL NULL 2012 Q1 28/Feb/2012:10:08:59 -0600 GET NULL HTTP/1.1 304 1012 http://localhost/phpmyadmin/navigation.php?lang=en-utf-8&convcharset=utf8&collation_connection=utf8_general_ci&token=697ef7c7f1c23069ba45d8d397722c04&phpMyAdmin=1nb4g50nb43hbftn krrl88dnk5hs3q5o IE

9
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Step 2 In SAP Information Design Tool, created 2 connections 1) To Hadoop (hive.cns) 2) To HANA (hana.cnx) see below

Steps for Hadoop: 1) Create New Relational Connection (hive.cnx) where I specified my Hadoop nodename server ie. n1.example.com:10000

10
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

2) Create Data Foundation hive 2012 weblog.dfx 3) Create Business Layer Hadoop WebLogDM.blx 4) Publish the connection hive.cnx to CMC (Central Management Control server) 5) Publish Business Layer Hadoop WebLogDM.blx to CMC 6) Once connection and Business Layer is published to CMC, they can be then accessed in Webi.

Steps for HANA: 1) Create New Relational Connection (hana.cnx) where I specified my HANA database ie. hanasrv:30015

11
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

2) Remaining steps same as Hadoop

12
Community Contributors: Minesh Amin and Jothi Periasamy

BIG Data Implementation Community Services

A SAP HANA and HADOOP Rapid Deployment Open Source Solutions (Draft More to Come)

Note: As a community service , Our open source rapid deployment solutions group currently focuses on developing an end-to-end Customer Behavioral and Fraud Analysis using SAP CRM , SAP SD and SAP FICO data along with social media information We will keep informed about our progress on this open source rapid deployment solutions, and also our SAP HANA book project.

SAP Community Contributor Jothi Periasamy

13
Community Contributors: Minesh Amin and Jothi Periasamy