一、zookeeper集成
使用自帶zookeeper可參考hbase官網 https://hbase.apache.org/book.html#quickstart
二、spark/hadoop集成
需要將jar放到hadoop的lib下
或者在spark提交命令中添加 --driver-class-path /opt/hadoop-2.7.7/lib/*:/opt/hbase-2.1.2/conf/參數
三、此時會出現各種奇葩問題，找不到類啊等等
一般都是jar包沖突問題，將沖突的jar包處理即可
netty
jackson
四、spark 執行時報java.lang.ClassNotFoundException: org.apache.htrace.core.HTraceConfiguration
將/hadoop-2.7.7/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar放在在spark的jars中
推測如果是hadoop時報錯，需要將/hadoop-2.7.7/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar 放到hadoop/lib中
五、org.apache.hadoop.hbase.TableNotFoundException: hbase:test
hbase:test去掉hbase:
六、Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not se（轉）
當從SparkSql得到的dataFrame，映射成RDD之后向hbase中直接保存數據的時候報錯：
Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
采用的是saveAsNewApiHadoopDataSet
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import SparkContext._
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.hbase.mapreduce.TableOutputFormat
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.client.Result
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.client.Put
import org.apache.hadoop.mapred.JobConf
import org.apache.hadoop.conf.Configuration
但是更換為saveAsHadoopDataset就可以使用，
// var conf = new Configuration()
// var tableName = "test_t1"
// val jobConf = new JobConf(conf,this.getClass)
// jobConf.set("hbase.zookeeper.quorum","10.172.10.169,10.172.10.168,10.172.10.170")
// jobConf.setOutputKeyClass(classOf[ImmutableBytesWritable])
//// jobConf.setOutputValueClass(classOf[Put])
// jobConf.setOutputFormat(classOf[org.apache.hadoop.hbase.mapred.TableOutputFormat])
// jobConf.set(TableOutputFormat.OUTPUT_TABLE,"test_t1")
// rdd1.map(
// x => {
// var put = new Put(Bytes.toBytes(x._1))
// put.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("c1"), Bytes.toBytes(x._2))
// (new ImmutableBytesWritable,put)
// }
// ).saveAsHadoopDataset(jobConf)
不知道在哪里出現了錯誤，對比發現應該是 沒有使用sc.hadoopConfiguration,而是使用的JobConf 作為參數，新API不能用舊的configuration 。
// sc.hadoopConfiguration.set("hbase.zookeeper.quorum","10.172.10.169,10.172.10.168,10.172.10.170")
//// sc.hadoopConfiguration.set("zookeeper.znode.parent","/hbase")
// sc.hadoopConfiguration.set(TableOutputFormat.OUTPUT_TABLE,"test_t1")
// var job = new Job(sc.hadoopConfiguration)
// job.setOutputKeyClass(classOf[ImmutableBytesWritable])
// job.setOutputValueClass(classOf[Result])
// job.setOutputFormatClass(classOf[TableOutputFormat[ImmutableBytesWritable]])
//
// rdd1.map(
// x => {
// var put = new Put(Bytes.toBytes(x._1))
// put.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("c1"), Bytes.toBytes(x._2))
// (new ImmutableBytesWritable,put)
// }
// ).saveAsNewAPIHadoopDataset(job.getConfiguration)
七、Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
將/hadoop-2.7.7/lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar放在在spark的jars中
推測如果是hadoop時報錯，需要將/hadoop-2.7.7/lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar 放到hadoop/lib中