Spark2.4.3 部署

scala下載地址：https://www.scala-lang.org/download/

spark下載地址：https://spark.apache.org/downloads.html

安裝步驟

安裝scala

tar -zxvf scala-2.13.0.tgz
scp -r scala-2.13.0 node102:/root
scp -r scala-2.13.0 node103:/root

配置環境變量 vi /etc/profile

#scala
export SCALA_HOME=/root/scala-2.13.0
export PATH=$SCALA_NAME/bin:$PATH

使環境變量生效

source /etc/profile

安裝spark

解壓

配置環境變量

#spark
export SPARK_HOME=/root/spark-2.4.3
export PATH=$PATH:$SPARK_HOME/bin

spark-env.sh文件配置

export JAVA_HOME=/root/jdk1.8.0_211
export SCALA_HOME=/root/scala-2.13.0

export HADOOP_HOME=/root/hadoop-3.1.2
export HADOOP_CONF_DIR=/root/hadoop-3.1.2/etc/hadoop
export SPARK_WORKER_MEMORY=500m
export SPARK_WORKER_CORES=1
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop1:2181,hadoop2:2181,hadoop3:2181,hadoop4:2181 -Dspark.deploy.zookeeper.dir=/spark"

slaves文件配置

node101
node102
node103

spark目錄文件同步到其他節點

scp -r spark-2.4.3/ node102:/root
scp -r spark-2.4.3/ node103:/root

啟動spark

cd /spark-2.4.3/sbin
start-all.sh

web頁面查看spark節點情況（端口號8080）