首先,解決talend連接hbase的問題:
公司使用的機器是HDP2.2的機器,上面配置好Hbase服務,在集群的/etc/hbase/conf/hbase-site.xml下,有如下配置:
<property> <name>zookeeper.znode.parent</name> <value>/hbase-unsecure</value> </property>
這個配置是決定, Hbase master在zookeeper中,存儲數據的根節點,如果不特別設置,默認位置是:/hbase (這個坑了我好久呀。。。,經過漫長的研究,終於頓悟,着實不易呀)。
在設置hbase databae時,配置如下:
測試zk的聯通性:
[zookeeper@hdp1 ~]$ zookeeper-client -server 172.28.3.61:2181 [zk: 172.28.3.61:2181(CONNECTED) 15] ls / [hiveserver2, controller_epoch, controller, brokers, zookeeper, hbase-unsecure, hadoop-ha, rmstore, admin, consumers, config] [zk: 172.28.3.61:2181(CONNECTED) 16] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, region-in-transition, online-snapshot, master, running, recovering-regions, draining, namespace, hbaseid, table]
最后的狀態是連接成功:
連接成功后,retrive schema:
在hbase中,表格信息如下:
hbase(main):047:0> scan 'zhangchao' ROW COLUMN+CELL example column=cc2:name, timestamp=1436838509057, value=justin example2 column=cc1:age, timestamp=1436838869051, value=38 example3 column=cc1:sex, timestamp=1436839082133, value=male example4 column=cc1:age, timestamp=1436854903976, value=334 4 row(s) in 0.0270 seconds hbase(main):048:0> describe 'zhangchao' DESCRIPTION ENABLED 'zhangchao', {NAME => 'cc1', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_M true EMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BL OCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NA ME => 'cc2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false' , KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => ' FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'tru e', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 1 row(s) in 0.0770 seconds
mysql中表的結構如下:
建立Talend模型,將hbase中的數據導入到mysql中:
執行后,mysql中的表中有了hbase的信息:
這樣hbase到mysql的數據通道就算打通,問題的關鍵其實是,如何設計mysql里的行數據,讓其轉換為hbase中的類數據,因為hbase是列式存儲,按照表->行->列簇:列->值的形式來存儲。