文章目錄
前言
在Mac上搭建Hadoop環境(1) — 虛擬機的安裝及SSH免密設置中，我們已經配置好了節點之間基本的網絡環境，接下來，只要在master(MBP)上下載安裝hadoop即可。
下載hadoop
可以前往Apache Hadoop官網，選擇你想要的版本進行下載。 這里，我選擇下載的是hadoop-2.7.7的binary版本。
安裝hadoop
在下載完hadoop-2.7.7.tar.gz后，將其解壓到你想安裝的目錄即可。 我將其解壓到 /opt 文件夾下。
sudo tar -C /opt -xvf ~/Downloads/hadoop-2.7.7.tar.gz;
然后對目錄重命名並創建相應的子目錄
sudo mv /opt/hadoop-2.7.7 /opt/hadoop;
mkdir /opt/hadoop/dfs;
mkdir /opt/hadoop/dfs/name;
mkdir /opt/hadoop/dfs/data;
mkdir /opt/hadoop/tmp;
配置hadoop
hadoop-env.sh
編輯 /opt/hadoop/etc/hadoop/hadoop-env.sh，設置JAVA_HOME值，如下
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home
yarn-env.sh
編輯 /opt/hadoop/etc/hadoop/yarn-env.sh，設置JAVA_HOME值，如下
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home
slaves
編輯 /opt/hadoop/etc/hadoop/slaves，加入之前設置的slave節點的hostname：
slave1
slave2
更新*.xml配置文件
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.lestat.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.lestat.groups</name>
<value>*</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
將hadoop復制到slave1和slave1上
scp -r /opt/hadoop parallels@slave1:~/
scp -r /opt/hadoop parallels@slave2:~/
然后分別在slave節點執行
sudo mv ~/hadoop /opt/hadoop;
sudo chown -R parallels:parallels /opt/hadoop
修改slave上的hadoop-env.sh和yarn-env.sh中的JAVA_HOME值
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64
添加hadoop環境變量
export PATH=$PATH:/opt/hadoop/bin:/opt/hadoop/sbin
啟動hadoop
在啟動之前，先執行hadoop namenode -format 進行格式化，
然后執行 start-all.sh來啟動hadoop集群，以下是我的啟動日志
Lestats-MBP:~ lestat$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/03/24 09:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 45208. Stop it first.
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave2.out
Starting secondary namenodes [master]
master: secondarynamenode running as process 45331. Stop it first.
19/03/24 09:57:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 45435. Stop it first.
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave2.out
然后可以執行master上jps來查看啟動的JVM。
Lestats-MBP:~ lestat$ jps
45331 SecondaryNameNode
45846 Jps
45208 NameNode
45435 ResourceManager
執行slave上jps來查看啟動的JVM。
如果jps命令找不到， 可以運行sudo yum install java-1.8.0-openjdk-devel進行安裝
[parallels@slave1 ~]$ jps
28832 Jps
26454 DataNode
26590 NodeManager
[parallels@slave2 ~]$ jps
26034 DataNode
26180 NodeManager
27931 Jps
這樣hadoop偽分布式集群的安裝就告一段落了。
測試
例如我們執行一些hadoop fs的命令：
lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -mkdir /input
19/03/24 10:09:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - lestat supergroup 0 2019-03-24 10:09 /input