Prof. Dr.

Stefan Edlich

Hadoop bersicht

Nach Google BigTable modelliert Anschub von Powerset / Yahoo Map/Reduce Abfragen

http://labs.google.com/papers/bigtable.html

HBase

Zeilen und Spalten (asso. Array) Vesionierung Zeile mit Schlssel identifiziert Trennzeichen :

Datenmodell

Nicht trivial POSIX UNIX Java 6 $ tar -xzf hbase-x.y.z.tar.gz $ cd ./hbase-x.z.z $ bin/start-hbase.sh Initial Single Node Cluster

Installation

Kommandozeilenoptionen

Test der Hbase Instanz

Java API JRuby Shell

$ bin/hbase shell HBase Shell; enter help<RETURN> for list of supported commands. Version: 0.20.3, r902334, Mon Jan 25 13:13:28 PST 2010 hbase(main):001:0> hbase(main):002:0> create Table, {NAME => F1, VERSIONS => 2} hbase(main):002:0> list Table 1 row(s) in 0.1030 seconds

CRUD via Shell

import org.apache.hadoop.hbase.*; import org.apache.hadoop.hbase.client.*; public class FirstHBaseClient { public static void main(String[] args) throws IOException { HTableDescriptor table = new HTableDescriptor(Table); HColumnDescriptor family = new HColumnDescriptor(F1); family.setMaxVersions(2); table.addFamily(family); HBaseConfiguration config = new HBaseConfiguration(); HBaseAdmin admin = new HBaseAdmin(config); admin.createTable(table); HTableDescriptor tables = admin.listTables(); for (int i=0; i<tables.length; i++) System.out.println( tables[i].getNameAsString() ); } }

via java

import import import import

java.io.IOException; org.apache.hadoop.hbase.*; org.apache.hadoop.hbase.client.*; org.apache.hadoop.hbase.util.Bytes;

public class FirstHBaseClient { public static void main(String[] args) throws IOException { HBaseConfiguration config = new HBaseConfiguration(); HTable table = new HTable(config, "Table"); Put p = new Put(Bytes.toBytes("FirstRowKey")); p.add(Bytes.toBytes("F1"), Bytes.toBytes("FirstColumn"), Bytes.toBytes("First Value")); table.put(p); } }

Insert = Put

import import import import

java.io.IOException; org.apache.hadoop.hbase.*; org.apache.hadoop.hbase.client.*; org.apache.hadoop.hbase.util.Bytes;

public class FirstHBaseClient { public static void main(String[] args) throws IOException { HBaseConfiguration config = new HBaseConfiguration(); HTable table = new HTable(config, "Table"); Get get = new Get(Bytes.toBytes("FirstRowKey")); Result result = table.get(get); byte[] value = result.getValue( Bytes.toBytes("F1"), Bytes.toBytes("FirstColumn")); System.out.println(Bytes.toString(value)); } }

Read mit Java

$ cp contrib/stargate/hbase-[version]-stargate.jar lib/ $ cp contrib/stargate/lib/* lib/ $ bin/hbase-daemon.sh start org.apache.hadoop.hbase.stargate.Main p <port> $ curl http://localhost:8080/Table/schema // TEST { NAME=> Table, IS_META => false, IS_ROOT => false, COLUMNS => [ { NAME => F1, BLOCKSIZE => 65536, BLOOMFILTER => false, BLOCKCACHE => true, COMPRESSION => NONE, VERSIONS => 2, TTL => 2147483647, IN_MEMORY => false } ] }

REST => Stargate Servlet

1-Master / N-Slaves (Region Server) regionservers / hbase-site.xml Replikation via HDFS

+= neue IP

Skalierung

Vorteile
Skalierung Map/Reduce Community APIs

Nachteile
Aufsetzen / Optimieren / Warten (Cloudera?) Map/Reduce keine DB-Replikation