Spark高可用集群搭建

 

---

Spark高可用集群搭建

node1    node2     node3

 

1.node1修改spark-env.sh，注釋掉hadoop(就不用開啟Hadoop集群了),添加如下語句

1. export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181 -Dspark.deploy.zookeeper.dir=/spark20170302"

 

2.同步到其他spark節點node2,node3

 

3. node2中 spark-env.sh 修改master為node2

 

4.啟動spark集群--Spark run on Standalone

 node2中啟動master,這樣master就成高可用了

 sbin下./start-master.sh

 

5.測試Spark高可用

1. ./bin/spark-submit --class org.apache.spark.examples.SparkPi--master spark://node1:7077,node2:7077 --driver-memory 512m --deploy-mode cluster --supervise --executor-memory 512M --total-executor-cores 1 ./lib/spark-examples-1.6.0-hadoop2.6.0.jar 100000

kill -9 node1中master的進程，看是否還能繼續執行