Spark—local模式環境搭建

Spark——local模式環境搭建

一、Spark運行模式介紹

1、本地模式（loca模式）：spark單機運行，一般用戶測試和開發使用

2、Standalone模式：構建一個主從結構(Master+Slave)的spark集群,spark運行在集群中。

3、Spark on yarn 模式：Spark客戶端直接連接Yarn,不用構建Spark集群

4、Spark on Mesos 模式：Spark客戶端直接連接Mesos.不需要額外構建Spark集群 

 

二、local模式

1、將編譯好的spark包解壓到指定目錄，我這里是使用spark源碼編譯的，編譯過程看上篇博客

tar -zxvf ./spark-2.1.0-bin-2.6.0-cdh5.7.0.tgz -C /home/hadoop/app/

2、配置環境變量

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0
export MAVEN_HOME=/home/hadoop/app/apache-maven-3.5.4
export SCALA_HOME=/home/hadoop/app/scala-2.11.8
export SPARK_HOME=/home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0
export PATH=$SPARK_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:
$PATH

3、啟動local模式

spark-shell --master local[2]

 

啟動過程如下：

[hadoop@hadoop01 ~]$ spark-shell --master local[2]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/10/09 19:49:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
18/10/09 19:50:13 WARN ObjectStore: Version information not found in metastore. 
hive.metastore.schema.verification is not enabled so recording 
the schema version 1.2.0
18/10/09 19:50:13 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
18/10/09 19:50:15 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.44.183:4040
Spark context available as 'sc' (master = local[2], app id = local-1539085800463).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
 
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
 
scala>

 

啟動后可以通過UI界面查看詳情：http://192.168.44.183:4040