文章目錄
1 運行環境
1軟件環境
三個節點
OS：64位RHEL5及以上或者64位CentOS6.0及以上
JVM：預裝64位JDK 1.8及以上版本
2 安裝准備
2.1准備虛擬機
准備三個節點的虛擬機
2.2 修改主機名
在各個節點執行以下操作來修改主機名，使集群下的主機有格式一個統一的主機名，以便后續的操作和維護。
修改主機名
vi /etc/sysconfig/network
192.168.xx.210 ha01
（其它倆台分別修改自己的ha02 ha03）
修改host映射：
vi /etc/hosts
192.168.xx.210 ha01
192.168.xx.220 ha02
192.168.xx.230 ha03
2.3 關閉防火牆
service iptables stop
chkconfig iptables off
2.4 配置時間同步
2.5 配置ssh免秘登錄
2.6 安裝jdk
3 安裝其他組件
3.1 安裝zookeeper和hadoop
3.2 安裝高可用hadoop
hadoop部分的配置分為兩部分hdfs和yarn。
3.2.1 HDFS
修改配置文件
修改core-site.xml（如果文件不存在，但是core-site.xml.template文件存在，則先修改文件名，執行mv core-site.xml.template core-site.xml）
vi /usr /local/hadoop-2.7.3/etc/hadoop/core-site.xml
修改為以下內容：
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://beh</value>
<final>false</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoopdata</value>
<final>false</final>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>ha01:2181,ha02:2181,ha03:2181</value>
<final>false</final>
</property>
</configuration>
修改hdfs-site.xml
vi /usr/local/hadoop-2.7.3/etc/hadoop/hdfs-site.xml
修改為以下內容：
<configuration>
<property>
<name>dfs.nameservices</name>
<value>beh</value>
<final>false</final>
</property>
<property>
<name>dfs.ha.namenodes.beh</name>
<value>nn1,nn2</value>
<final>false</final>
</property>
<property>
<name>dfs.namenode.rpc-address.beh.nn1</name>
<value>ha01:9000</value>
<final>false</final>
</property>
<property>
<name>dfs.namenode.http-address.beh.nn1</name>
<value>ha01:50070</value>
<final>false</final>
</property>
<property>
<name>dfs.namenode.rpc-address.beh.nn2</name>
<value>ha02:9000</value>
<final>false</final>
</property>
<property>
<name>dfs.namenode.http-address.beh.nn2</name>
<value>ha02:50070</value>
<final>false</final>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://ha01:8485;ha02:8485;ha03:8485/beh</value>
<final>false</final>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.beh</name>
<value>true</value>
<final>false</final>
</property>
<property>
<name>dfs.client.failover.proxy.provider.beh</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
<final>false</final>
</property>
<property>
<name>dfs.journalnode.edits .dir</name>
<value>/usr/local/metadata/journal</value>
<final>false</final>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence
shell(/bin/true)
</value>
<final>false</final>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/usr/local/.ssh/id_rsa</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<final>false</final>
</property>
<configuration>
修改slaves
vi /usr/local/hadoop-2.7.3/etc/hadoop/slaves
修改為以下內容：
ha02
ha03
3.2.2 YARN
修改mapred-site.xml
vi /usr/local/hadoop2.7.3/etc/hadoop/mapred-site.xml
修改為以下內容：
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>ha02:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>ha03:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/usr/local/metadata/hadoop-yarn/staging</value>
</property>
</configuration>
修改yarn-site.xml
vi /usr/local/hadoop2.7.3/etc/hadoop/yarn-site.xml
修改為以下內容：
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/usr/local/metadata/yarn</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/usr/local/logs/yarn/userlogs</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>hdfs://beh/var/log/hadoop-yarn/apps</value>
</property>
<!-- Resource Manager Configs -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>beh</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!--RM1 RM2 is different-->
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>ha01:2181,ha02:2181,ha03:2181</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<!-- RM1 configs -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>ha01:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>ha01:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>ha01:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>ha01:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>ha01:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>ha01:23141</value>
</property>
<!-- RM2 configs -->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>ha02:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>ha02:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>ha02:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>ha02:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>ha02:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>ha02:23141</value>
</property>
<!-- Node Manager Configs -->
<property>
<name>mapreduce.shuffle.port</name>
<value>23080</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>ha01:2181,ha02:2181,ha03:2181</value>
</property>
</configuration>
修改環境變量
vim /usr/local/hadoop-2.7.3/etc/hadoop/hadoop-env.sh
vim /usr/local/hadoop-2.7.3/etc/hadoop/yarn-env.sh
修改為以下內容：
export JAVA_HOME=/usr/local/jdk1.8.0_102
3.2.3 分發配置文件
scp -r /usr/local/hadoop2.7.3 ha02:/usr/local
scp -r /usr/local/hadoop2.7.3 ha03:/usr/local
注：將以上配置復制到所有節點
3.2.4 啟動HDFS
啟動journalnode（進程名：JournalNode）
sbin/hadoop-daemon.sh start journalnode
格式化zookeeper，在ha01上執行
hdfs zkfc -formatZK
對ha01節點進行格式化和啟動啟動namenode（進程名：NameNode）：
hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
對ha02節點進行格式化和啟動
hdfs namenode -bootstrapStandby
sbin/hadoop-daemon.sh start namenode
在ha01和ha02上啟動zkfc服務（zkfc服務進程名：DFSZKFailoverController）：此時ha01和ha02就會有一個節點變為active狀態
sbin/hadoop-daemon.sh start zkfc
啟動datanode（進程名：DataNode）：在ha01上執行
sbin/hadoop-daemons.sh start datanode
3.2.5 驗證是否成功
打開瀏覽器，訪問 hadoop1:50070 以及 hadoop2:50070，你將會看到兩個namenode一個是active而另一個是standby。
然后kill掉其中active的namenode進程，另一個standby的naemnode將會自動轉換為active狀態
hadoop01:50070或hadoop01的ip:50070
hadoop02:50070或hadoop02的ip:50070
