Linux安裝Spark

1、下載

http://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz

2、解壓

tar -xvf spark-2.4.1-bin-hadoop2.7.tgz

3、配置環境變量SPARK_HOME

• vim /etc/profile 或者 vim ~/.bashrc

#Spark Environment
export SPARK_HOME=$USER_HOME/software/spark-2.4.1-bin-hadoop2.7

• source /etc/profile 或者 source ~/.bashrc

4、修改配置文件

• slave配置文件復制

cp $SPARK_HOME/conf/slaves.template $SPARK_HOME/conf/slaves

• vim $SPARK_HOME/conf/slaves

修改主機名，暫時默認為localhost，不修改

• spark-env配置文件修改

cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh

• vim $SPARK_HOME/conf/spark-env.sh

export USER_HOME=/home/hadoop
export JAVA_HOME=$USER_HOME/software/jdk1.8.0_201 
export SCALA_HOME=$USER_HOME/software/scala-2.12.8 
export SPARK_WORKER_MEMORY=1G 
export HADOOP_HOME=$USER_HOME/software/hadoop-2.7.3 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export SPARK_MASTER_IP=hadoop

5、Spark啟停

$SPARK_HOME/sbin/start-all.sh
$SPARK_HOME/sbin/stop-all.sh

啟動后可以通過訪問如下網址查看SPARK的控制台

http://hadoop:8080/

6、進入spark shell控制台

$SPARK_HOME/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.1
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201)

啟動成功后，可訪問http://hadoop:4040/jobs/

7、Spark測試

• Hadoop命令

hadoop fs -mkdir -p /usr/hadoop	創建/usr/hadoop文件夾
hadoop fs -ls -R /			列舉出根目錄/下面的文件和文件夾
hadoop fs -mkdir -p /usr/data/input	創建/usr/hadoop/input文件夾
hadoop fs -put /home/hadoop/aaa.txt /usr/data/input	上傳本地文件/home/hadoop/aaa.txt到/usr/data/input目錄
hadoop fs -ls -R /

• Spark Shell測試

var test_file = sc.textFile("/usr/data/input/aaa.txt")
test_file.count()
scala> var test_file = sc.textFile("/usr/data/input/aaa.txt")
test_file: org.apache.spark.rdd.RDD[String] = /usr/data/input/aaa.txt MapPartitionsRDD[1] at textFile at <console>:24

scala> test_file.count()
[Stage 0:> (0 + 1) / 1    res0: Long = 5

8、Web界面查看

http://hadoop:8100/	默認端口為50070
http://localhost:8100	默認端口為50070
http://localhost:8088/cluster
http://hadoop:8080/
http://hadoop:4040/jobs/

9、命令測試

$HADOOP_HOME/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /out