Kafka：ZK+Kafka+Spark Streaming集群環境搭建（二）安裝hadoop2.9.0

如何搭建配置centos虛擬機請參考《Kafka：ZK+Kafka+Spark Streaming集群環境搭建（一）VMW安裝四台CentOS，並實現本機與它們能交互，虛擬機內部實現可以上網。》

如何配置hadoop2.9.0 HA 請參考《Kafka：ZK+Kafka+Spark Streaming集群環境搭建（十）安裝hadoop2.9.0搭建HA》

安裝hadoop的服務器：

192.168.0.120      master
192.168.0.121      slave1
192.168.0.122      slave2
192.168.0.123      slave3

公共運行及環境准備工作：

修改主機名

將搭建1個master，3個slave的hadoop集群方案：首先修改主機名vi /etc/hostname，在master上修改為master，其中一個slave上修改為slave1，其他依次推。

192.168.0.120

192.168.0.121

192.168.0.122

192.168.0.123

關閉防火牆

依次在master,slave1,slave2,slave3上執行以下命令：

systemctl stop firewalld.service 
systemctl disable firewalld.service
firewall-cmd --state

配置hosts

在每台主機上修改host文件：vi /etc/hosts

192.168.0.120      master
192.168.0.121      slave1
192.168.0.122      slave2
192.168.0.123      slave3

以master節點配置后的結果為：

配置之后ping一下用戶名看是否生效

ping master
ping slave1
ping slave2
ping slave3

ssh配置

安裝Openssh server

備注：默認centos上都是已經安裝好了ssh，因此不需要安裝。

1）確認centos是否安裝ssh命令：

 yum install openssh-server -y

2）在所有機器上都生成私鑰和公鑰

ssh-keygen -t rsa   #一路回車

依次在master,slave1,slave2,slave3上執行：

3）需要讓機器間都能相互訪問，就把每個機子上的id_rsa.pub發給master節點，傳輸公鑰可以用scp來傳輸。

slave1執行：

scp ~/.ssh/id_rsa.pub spark@master:~/.ssh/id_rsa.pub.slave1

slave2執行：

scp ~/.ssh/id_rsa.pub spark@master:~/.ssh/id_rsa.pub.slave2

slave3執行：

scp ~/.ssh/id_rsa.pub spark@master:~/.ssh/id_rsa.pub.slave3

以slave1上執行過程截圖：

執行后master顯示結果：

4）在master上，將所有公鑰加到用於認證的公鑰文件authorized_keys中

cat ~/.ssh/id_rsa.pub* >> ~/.ssh/authorized_keys

注意：上邊是id_rsa.pub*，后邊帶了一個“*”。

5）將master上生成的公鑰文件authorized_keys分發給每台slave

scp ~/.ssh/authorized_keys spark@slave1:~/.ssh/
scp ~/.ssh/authorized_keys spark@slave2:~/.ssh/
scp ~/.ssh/authorized_keys spark@slave3:~/.ssh/

執行過程截圖：

6）在每台機子上驗證SSH無密碼通信

ssh master
ssh slave1
ssh slave2
ssh slave3

如果登陸測試不成功，則可能需要修改文件authorized_keys的權限（權限的設置非常重要，因為不安全的設置安全設置,會讓你不能使用RSA功能 ）

chmod 600 ~/.ssh/authorized_keys

安裝java8

1）查看是否系統已經安裝了jdk

[spark@localhost ~]$ rpm -qa | grep Java
[spark@localhost ~]$ 

我這里發現未安裝過jdk，如果你發現有可以參考《使用CentOS7卸載自帶jdk安裝自己的JDK1.8》進行卸載。

2）從官網下載最新版 Java

Spark官方說明 Java 只要是6以上的版本都可以，我下的是jdk-8u171-linux-x64.tar.gz

3）上傳jdk到centos下的/opt目錄

安裝“在線導入安裝包”插件：  

yum -y install lrzsz

備注：目的為了上傳文件使用。

安裝插件完成之后輸入  rz  命令然后按回車，就會彈出一個窗口，然后你就在這個窗口找到你下載好的jdk，

備注：使用  rz  命令的好處就是你在哪里輸入rz導入的安裝包他就在哪里，不會跑到根目錄下

關於該插件的使用和安裝，請參考《https://blog.csdn.net/hui_2016/article/details/69941850》

4）執行解壓

 tar -zxvf jdk-8u171-linux-x64.tar.gz 

5）配置jdk環境變量

修改環境變量vi /etc/profile，添加下列內容，注意將home路徑替換成你的：

export WORK_SPACE=/opt/
export JAVA_HOME=$WORK_SPACE/jdk1.8.0_171
export JRE_HOME=/opt/jdk1.8.0_171/jre
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

然后使環境變量生效，並驗證 Java 是否安裝成功

$ source /etc/profile   #生效環境變量
$ java -version         #如果打印出如下版本信息，則說明安裝成功
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
[root@localhost opt]# 

依次把master,slave1,slave2,slave3安裝jdk8。

安裝scala

Spark官方要求 Scala 版本為 2.10.x，注意不要下錯版本，我這里下了 2.11.0，官方下載地址。

依據參考：http://spark.apache.org/downloads.html

同樣解壓到/opt下：

 tar -zxvf scala-2.11.0.tgz 

再次修改環境變量 vi /etc/profile，添加以下內容：

export SCALA_HOME=$WORK_SPACE/scala-2.11.0
export PATH=$PATH:$SCALA_HOME/bin

同樣的方法使環境變量生效，並驗證 scala 是否安裝成功

$ source /etc/profile   #生效環境變量
$ scala -version        #如果打印出如下版本信息，則說明安裝成功
Scala code runner version 2.11.0 -- Copyright 2002-2013, LAMP/EPFL

 

安裝hadoop2.9.0 YARN

下載解壓

從官網下載 hadoop2.9.0 版本。

同樣在/opt/中解壓

tar -zxvf hadoop-2.9.0.tar.gz

配置 Hadoop

cd /opt/hadoop-2.9.0/etc/hadoop進入hadoop配置目錄，需要配置有以下7個文件：hadoop-env.sh，yarn-env.sh，slaves，core-site.xml，hdfs-site.xml，maprd-site.xml，yarn-site.xml

1）在hadoop-env.sh中配置JAVA_HOME

# The java implementation to use.
export JAVA_HOME=/opt/jdk1.8.0_171

2）在yarn-env.sh中配置JAVA_HOME

# The java implementation to use.
export JAVA_HOME=/opt/jdk1.8.0_171

3）在slaves中配置slave節點的ip或者host

#localhost
slave1
slave2
slave3

4）修改core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000/</value>
    </property>
    <property>
         <name>hadoop.tmp.dir</name>
         <value>file:/opt/hadoop-2.9.0/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
</configuration>

備注：不需要在/opt/hadoop-2.9.0下建立tmp文件夾，同時分配權限777

5）修改hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:9001</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/hadoop-2.9.0/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/hadoop-2.9.0/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

備注：不需要在/opt/hadoop-2.9.0下新建dfs文件夾，再在dfs下新建name,data文件夾，並給他們分配777權限。

1）dfs.namenode.name.dir----HDFS namenode數據鏡像目錄

2）dfs.datanode.data.dir---HDFS datanode數據鏡像存儲路徑，可以配置多個不同的分區和磁盤中，使用,號分隔

3）還可以配置：dfs.namenode.http-address---HDFS Web查看主機和端口號

可以參考下邊這個hdfs-site.xml配置項 

vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/data/hadoop/hdfs/name</value>
                <!-- HDFS namenode數據鏡像目錄 -->
                <description> </description>
        </property>

        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/data/hadoop/hdfs/data</value>
                <!-- HDFS datanode數據鏡像存儲路徑，可以配置多個不同的分區和磁盤中，使用,號分隔 -->
                <description> </description>
        </property>

        <property>
                <name>dfs.namenode.http-address</name>
                <value>apollo.hadoop.com:50070</value>
                <!-- HDFS Web查看主機和端口號 -->
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>artemis.hadoop.com:50090</value>
                <!-- 輔控HDFS Web查看主機和端口 -->
        </property>

        <property>
                <name>dfs.webhdfs.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
                <!-- HDFS數據保存份數，通常是3 -->
        </property>

        <property>
                <name>dfs.datanode.du.reserved</name>
                <value>1073741824</value>
                <!-- datanode寫磁盤會預留1G空間給其它程序使用，而非寫滿，單位 bytes -->
        </property>

        <property>
                <name>dfs.block.size</name>
                <value>134217728</value>
                <!-- HDFS數據塊大小，當前設置為128M/Blocka -->
        </property>

        <property>
                <name>dfs.permissions.enabled</name>
                <value>false</value>
                <!-- HDFS關閉文件權限 -->
        </property>

</configuration>

來自於《https://blog.csdn.net/jssg_tzw/article/details/70314184》

6）修改mapred-site.xml

需要從mapred-site.xml.template拷貝mapred-site.xml:

[root@localhost hadoop]# scp mapred-site.xml.template mapred-site.xml
[root@localhost hadoop]# vi mapred-site.xml

配置為：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobtracker.http.address</name>
        <value>master:50030</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>http://master:9001</value>
    </property>
</configuration>

7）修改yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
       <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8035</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>512修改為 2048，否則會拋出內存不足問題導致nodemanager啟動不了</value>
    </property>
</configuration>

將配置好的hadoop-2.9.0文件夾分發給所有slaves

scp -r /opt/hadoop-2.9.0 spark@slave1:/opt/
scp -r /opt/hadoop-2.9.0 spark@slave2:/opt/
scp -r /opt/hadoop-2.9.0 spark@slave3:/opt/

 注意：此時默認master,slave1,slave2,slave3上是沒有/opt/hadoop-2.9.0，因此直接拷貝可能會出現無權限操作 。

            解決方案，分別在master,slave1,slave2,slave3的/opt下創建hadoop-2.9.0，並分配777權限。

[root@localhost opt]# mkdir hadoop-2.9.0
[root@localhost opt]# chmod 777 hadoop-2.9.0
[root@localhost opt]# 

之后，再次操作拷貝就有權限操作了。

啟動 Hadoop

在 master 上執行以下操作，就可以啟動 hadoop 了。

cd /opt/hadoop-2.9.0     　　　　#進入hadoop目錄
bin/hadoop namenode -format     #格式化namenode
sbin/start-all.sh               #啟動dfs，yarn

執行format成功的提示信息如下：

[spark@master hadoop-2.9.0]$ bin/hadoop namenode -format     #格式化namenode
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

18/06/30 08:28:40 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.0.120
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.9.0
STARTUP_MSG:   classpath = /opt/hadoop-2.9.0/etc/hadoop:/opt/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/hadoop-2.9.0。。。
TARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 756ebc8394e473ac25feac05fa493f6d612e6c50; compiled by 'arsuresh' on 2017-11-13T23:15Z
STARTUP_MSG:   java = 1.8.0_171
************************************************************/
18/06/30 08:28:40 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/06/30 08:28:40 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-5003db37-c0ea-4bac-a789-bd88cae43202
18/06/30 08:28:40 INFO namenode.FSEditLog: Edit logging is async:true
18/06/30 08:28:40 INFO namenode.FSNamesystem: KeyProvider: null
18/06/30 08:28:40 INFO namenode.FSNamesystem: fsLock is fair: true
18/06/30 08:28:40 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
18/06/30 08:28:40 INFO namenode.FSNamesystem: fsOwner             = spark (auth:SIMPLE)
18/06/30 08:28:40 INFO namenode.FSNamesystem: supergroup          = supergroup
18/06/30 08:28:40 INFO namenode.FSNamesystem: isPermissionEnabled = true
18/06/30 08:28:40 INFO namenode.FSNamesystem: HA Enabled: false
18/06/30 08:28:40 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
18/06/30 08:28:40 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
18/06/30 08:28:40 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
18/06/30 08:28:40 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
18/06/30 08:28:40 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jun 30 08:28:40
18/06/30 08:28:40 INFO util.GSet: Computing capacity for map BlocksMap
18/06/30 08:28:40 INFO util.GSet: VM type       = 64-bit
18/06/30 08:28:40 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
18/06/30 08:28:40 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/06/30 08:28:41 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
18/06/30 08:28:41 WARN conf.Configuration: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
18/06/30 08:28:41 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
18/06/30 08:28:41 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
18/06/30 08:28:41 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
18/06/30 08:28:41 INFO blockmanagement.BlockManager: defaultReplication         = 3
18/06/30 08:28:41 INFO blockmanagement.BlockManager: maxReplication             = 512
18/06/30 08:28:41 INFO blockmanagement.BlockManager: minReplication             = 1
18/06/30 08:28:41 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
18/06/30 08:28:41 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
18/06/30 08:28:41 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
18/06/30 08:28:41 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
18/06/30 08:28:41 INFO namenode.FSNamesystem: Append Enabled: true
18/06/30 08:28:41 INFO util.GSet: Computing capacity for map INodeMap
18/06/30 08:28:41 INFO util.GSet: VM type       = 64-bit
18/06/30 08:28:41 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
18/06/30 08:28:41 INFO util.GSet: capacity      = 2^20 = 1048576 entries
18/06/30 08:28:41 INFO namenode.FSDirectory: ACLs enabled? false
18/06/30 08:28:41 INFO namenode.FSDirectory: XAttrs enabled? true
18/06/30 08:28:41 INFO namenode.NameNode: Caching file names occurring more than 10 times
18/06/30 08:28:41 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: falseskipCaptureAccessTimeOnlyChange: false
18/06/30 08:28:41 INFO util.GSet: Computing capacity for map cachedBlocks
18/06/30 08:28:41 INFO util.GSet: VM type       = 64-bit
18/06/30 08:28:41 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
18/06/30 08:28:41 INFO util.GSet: capacity      = 2^18 = 262144 entries
18/06/30 08:28:41 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/06/30 08:28:41 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/06/30 08:28:41 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/06/30 08:28:41 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/06/30 08:28:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/06/30 08:28:41 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/06/30 08:28:41 INFO util.GSet: VM type       = 64-bit
18/06/30 08:28:41 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
18/06/30 08:28:41 INFO util.GSet: capacity      = 2^15 = 32768 entries
18/06/30 08:28:41 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1045060460-192.168.0.120-1530318521716
18/06/30 08:28:41 INFO common.Storage: Storage directory /opt/hadoop-2.9.0/dfs/name has been successfully formatted.
18/06/30 08:28:41 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop-2.9.0/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/06/30 08:28:41 INFO namenode.FSImageFormatProtobuf: Image file /opt/hadoop-2.9.0/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 322 bytes saved in 0 seconds.
18/06/30 08:28:41 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/30 08:28:41 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.0.120
************************************************************/

備注：如果出現上邊紅色部分信息系則說明格式化成功。

啟動執行過程：

[spark@master hadoop-2.9.0]$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop-2.9.0/logs/hadoop-spark-namenode-master.out
slave3: starting datanode, logging to /opt/hadoop-2.9.0/logs/hadoop-spark-datanode-slave3.out
slave2: starting datanode, logging to /opt/hadoop-2.9.0/logs/hadoop-spark-datanode-slave2.out
slave1: starting datanode, logging to /opt/hadoop-2.9.0/logs/hadoop-spark-datanode-slave1.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /opt/hadoop-2.9.0/logs/hadoop-spark-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-resourcemanager-master.out
slave2: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave2.out
slave1: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave1.out
slave3: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave3.out
[spark@master hadoop-2.9.0]$ jps
1361 NameNode
1553 SecondaryNameNode
1700 ResourceManager
1958 Jps

[spark@master hadoop-2.9.0]$ sbin/start-yarn.sh #啟動yarn
starting yarn daemons
resourcemanager running as process 1700. Stop it first.
slave3: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave3.out
slave2: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave2.out
slave1: starting nodemanager, logging to /opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave1.out
[spark@master hadoop-2.9.0]$
[spark@master hadoop-2.9.0]$ jps
1361 NameNode
1553 SecondaryNameNode
1700 ResourceManager
2346 Jps

啟動過程發現問題：

1）啟動問題：slave1,slave2,slave3都是只啟動了DataNode，而DataManager並沒有啟動：解決方案參考《Kafka：ZK+Kafka+Spark Streaming集群環境搭建（七）針對hadoop2.9.0啟動DataManager失敗問題》

2）啟動問題：啟動之后發現slave上正常啟動了DataNode,DataManager，但是過了幾秒后發現DataNode被關閉：解決方案參考《Kafka：ZK+Kafka+Spark Streaming集群環境搭建（五）針對hadoop2.9.0啟動之后發現slave上正常啟動了DataNode,DataManager，但是過了幾秒后發現DataNode被關閉》

3）啟動問題：執行start-all.sh出現異常：failed to launch: nice -n 0 /bin/spark-class org.apache.spark.deploy.worker.Worker：解決方案參考《Kafka：ZK+Kafka+Spark Streaming集群環境搭建（四）針對hadoop2.9.0啟動執行start-all.sh出現異常：failed to launch: nice -n 0 /bin/spark-class org.apache.spark.deploy.worker.Worker》

正常啟動hadoop后，master正常啟動包含的進程有：

[spark@master hadoop2.9.0]$ jps
5938 ResourceManager
5780 SecondaryNameNode
6775 Jps
5579 NameNode
[spark@master hadoop2.9.0]$ 

正常啟動hadoop后，slaves上正常啟動包含的進程有：

[spark@slave1 hadoop-2.9.0]$ jps
3202 NodeManager
3784 Jps
3086 DataNode
[spark@slave1 hadoop-2.9.0]$

另外啟動master的JobHistoryServer進程方式：

[spark@master hadoop-2.9.0]$ /opt/hadoop-2.9.0/sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/hadoop-2.9.0/logs/mapred-spark-historyserver-master.out
[spark@master hadoop-2.9.0]$ jps
2945 SecondaryNameNode
2216 QuorumPeerMain
3386 JobHistoryServer
3099 ResourceManager
3454 Jps

驗證是否安裝成功訪問地址：http://192.168.0.120:8088/，是否可以訪問。

訪問：http://192.168.0.120:50070/  備注：該端口50070配置項是可以設置在hdfs-site.xml  

 

 

參考：《spark+hadoop (yarn模式)》