使用spark訪問elasticsearch的數據

使用spark訪問elasticsearch的數據，前提是spark能訪問hive，hive能訪問es
http://blog.csdn.net/ggz631047367/article/details/50445832

1.配置

cp elasticsearch-hadoop-2.1.1/dist/elasticsearch-hadoop-2.1.1.jar /usr/local/spark-1.5.2/lib/

2.修改配置文件/usr/local/spark/conf/hive-site.xml,添加

<property>
  <name>hive.aux.jars.path</name>
  <value>file:///usr/local/spark-1.5.2/lib/elasticsearch-hadoop-2.1.1.jar</value>
  <description>A comma separated list (with no spaces) of the jar files</description>
</property>

3.配置/usr/local/hive-1.2.1/lib/mysql-connector-java-5.1.31-bin.jar,在spark-env.sh添加

export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/local/hive-1.2.1/lib/mysql-connector-java-5.1.31-bin.jar

4.啟動

sbin/start-thriftserver.sh   --hiveconf hive.server2.thrift.port=10000  --hiveconf hive.server2.thrift.bind.host=hadoop-master     --master spark://10.8.2.100:7077  --jars /usr/local/spark-1.5.2/lib/elasticsearch-hadoop-2.1.1.jar

或者

bin/spark-sql –master spark://10.8.2.100:7077 –jars /usr/local/spark-1.5.2/lib/elasticsearch-hadoop-2.1.1.jar

5.測試

beeline -u jdbc:hive2://10.8.2.100:10000 (稍等一下執行此命令，報錯有可能是服務還沒起來)

select count(dtime) from achi_ex limit 10;

注意：–driver-class-path和–jars的區別，使用sbin/start-thriftserver.sh -h查看
如果1中拷貝到所有slave中spark/lib中，后面不用—jars指定jar包

這種方式比直接訪問hive中的數據慢一倍左右