1.主機名和IP配置
我們按照【三節點大數據環境安裝教程1】已經完成虛擬機的克隆，但是我們克隆出來的三台虛擬機的配置是一樣的需要做簡單的修改.
1.1 啟動三台虛擬機
1.啟動第一台虛擬機
2.啟動第二台虛擬機
3.啟動第三台虛擬機
1.2 配置三台虛擬機主機名
- 首先使用root用戶名和root密碼分別登錄三台虛擬機
- 分別在三台虛擬機上執行命令:hostnamectl set-hostname nodeXXX(虛擬機名)
第一台機器上設置主機名node1
第二台機器上設置主機名node2
第三台機器上設置主機名node3
然后在三台機器上分別執行命令:logout
發現主機名已經修改成node1了，相同的操作大家在其他兩台機器上執行下看看效果.
1.3 ip配置
三節點ip規划如下：
| 節點名稱 | ip |
|---|---|
| node1 | 192.168.200.11 |
| node2 | 192.168.200.12 |
| node3 | 192.168.200.13 |
如下圖，將node1上的ip修改為192.168.200.11，修改完后使用命令:systemctl restart network重啟網卡
按照上面步驟一次修改node2的ip為:192.168.200.12,修改完后使用systemctl restart nework命令重啟網卡，node3的ip修改方法一樣，修改為192.168.200.13,修改完后重啟網卡.
2.在xshell中創建三台虛擬機連接會話
3.root用戶的免密登錄配置
3.1 連接三台虛擬機
按住Ctrl一次選擇三台虛擬機的會話連接，點擊連接，這時會一次性打開三台虛擬機的連接會話
會出現三次安全警告，連續點擊三次接受並保存即可.
3.2 生成公鑰和私鑰
使用此命令：ssh-keygen -t rsa 分別在三台機器中都執行一遍,這里只在node1上做演示，其他兩台機器也需要執行此命令。
[root@node1 ~]# ssh-keygen -t rsa #<--回車
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): #<--回車
#會在root用戶的家目錄下生成.ssh目錄，此目錄中會保存生成的公鑰和私鑰
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): #<--回車
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:gpDw08iG9Tq+sGZ48TXirWTY17ajXhIea3drjy+pU3g root@node1
The key's randomart image is:
+---[RSA 2048]----+
|. . |
| * = |
|. O o |
| . + . |
| o . + S. |
| ..+..o*. E |
|o o+++*.=o.. |
|.=.+oo.=oo+o |
|+.. .oo.o=o+o |
+----[SHA256]-----+
You have new mail in /var/spool/mail/root
[root@node1 ~]#
3.3 配置hosts文件
#hosts文件中配置三台機器ip和主機名的映射關系,其他兩台機器按照相同的方式操作.
[root@node1 ~]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.11 node1
192.168.200.12 node2
192.168.200.13 node3
node1的配置:
node2的配置:
node3的配置:
3.4 拷貝公鑰文件
- 將node1的公鑰拷貝到node2,node3上
- 將node2的公鑰拷貝到node1,node3上
- 將node3的公鑰拷貝到node1,node2上
以下以node1為例執行秘鑰復制命令：ssh-copy-id -i 主機名
#復制到node2上
[root@node1 ~]# ssh-copy-id -i node2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node2 (192.168.200.12)' can't be established.
ECDSA key fingerprint is SHA256:rJzUyoggUP/Zn9v5rvqKpWppnG9xZ4gBZuXqHWxPy5k.
ECDSA key fingerprint is MD5:f3:37:16:c4:bb:00:3e:59:ec:b3:37:23:1b:24:88:e6.
Are you sure you want to continue connecting (yes/no)? yes #詢問是否要連接輸入yes回車
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node2's password: #輸入root用戶的密碼root后回車
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node2'"
and check to make sure that only the key(s) you wanted were added.
#復制到node3上
[root@node1 ~]# ssh-copy-id -i node3
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node3 (192.168.200.13)' can't be established.
ECDSA key fingerprint is SHA256:rJzUyoggUP/Zn9v5rvqKpWppnG9xZ4gBZuXqHWxPy5k.
ECDSA key fingerprint is MD5:f3:37:16:c4:bb:00:3e:59:ec:b3:37:23:1b:24:88:e6.
Are you sure you want to continue connecting (yes/no)? yes #詢問是否要連接輸入yes回車
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node3's password: #輸入root用戶的密碼root后回車
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node3'"
and check to make sure that only the key(s) you wanted were added.
[root@node1 ~]#
3.4 驗證免密登錄配置
此操作只在node1上操作，其他機器上大家在驗證。
#使用ssh 命令登錄node2
[root@node1 ~]# ssh node2
Last login: Sun Jun 30 13:56:53 2019 from node1
#登錄成功后這里的命令提示符已經變為[root@node2 ~]#說明登錄成功
[root@node2 ~]# logout #退出node2繼續 驗證登錄node3
Connection to node2 closed.
#登錄node3
[root@node1 ~]# ssh node3
Last login: Sun Jun 30 13:56:55 2019 from node1
#登錄成功
[root@node3 ~]# logout
Connection to node3 closed.
You have new mail in /var/spool/mail/root
[root@node1 ~]#
3.5 添加本地認證公鑰到認證文件中
#進入到root用戶的家目錄下
[root@node1 ~]# cd ~
[root@node1 ~]# cd .ssh/
#講生成的公鑰添加到認證文件中
[root@node1 .ssh]# cat id_rsa.pub >> authorized_keys
[root@node1 .ssh]#
4.安裝hadoop
4.1 創建hadoop用戶組和hadoop用戶
創建hadoop用戶組和hadoop用戶需要在三台機器上分別操作
#1.創建用戶組hadoop
[root@node1 ~]# groupadd hadoop
#2.創建用戶hadoop並添加到hadoop用戶組中
[root@node1 ~]# useradd -g hadoop hadoop
#3.使用id命令查看hadoop用戶組和hadoop用戶創建是否成功
[root@node1 ~]# id hadoop
#用戶uid 用戶組id gid 用戶組名
uid=1000(hadoop) gid=1000(hadoop) groups=1000(hadoop)
#設置hadoop用戶密碼為hadoop
[root@node1 ~]# passwd hadoop
Changing password for user hadoop.
New password: #輸入hadoop后回車
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: #再次輸入hadoop后回車
passwd: all authentication tokens updated successfully.
[root@node1 ~]# chown -R hadoop:hadoop /home/hadoop/
[root@node1 ~]# chmod -R 755 /home/hadoop/
#把root用戶的環境變量文件復制並覆蓋hadoop用戶下的.bash_profile
[root@node1 ~]# cp .bash_profile /home/hadoop/
對hadoop用戶做免密碼登錄配置參考root用戶的方式
[hadoop@node1 ~] su - hadoop
[hadoop@node1 ~] source .bash_profile
#使用su - hadoop切換到hadoop用戶下執行如下操作
[hadoop@node1 ~]# ssh-keygen -t rsa #<--回車
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): #<--回車
#會在root用戶的家目錄下生成.ssh目錄，此目錄中會保存生成的公鑰和私鑰
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): #<--回車
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:gpDw08iG9Tq+sGZ48TXirWTY17ajXhIea3drjy+pU3g root@node1
The key's randomart image is:
+---[RSA 2048]----+
|. . |
| * = |
|. O o |
| . + . |
| o . + S. |
| ..+..o*. E |
|o o+++*.=o.. |
|.=.+oo.=oo+o |
|+.. .oo.o=o+o |
+----[SHA256]-----+
You have new mail in /var/spool/mail/root
[hadoop@node1 ~]#
#修改.ssh目錄權限
[hadoop@node1 ~]$ chmod -R 755 .ssh/
[hadoop@node1 ~]$ cd .ssh/
[hadoop@node1 .ssh]$ chmod 644 *
[hadoop@node1 .ssh]$ chmod 600 id_rsa
[hadoop@node1 .ssh]$ chmod 600 id_rsa.pub
[hadoop@node1 .ssh]$ ssh-copy-id -i node3 # 將hadoop用戶下得公鑰配置到其他機器
[hadoop@node1 .ssh]$ cat id_rsa.pub >> authorized_keys #將自己的公鑰也配置進去
4.2 配置hadoop
在一台機器上配置好后復制到其他機器上即可,這樣保證三台機器的hadoop配置是一致的.
1.上傳hadoop安裝包,進行解壓
#1.創建hadoop安裝目錄
[root@node1 ~]# mkdir -p /opt/bigdata
#2.解壓hadoop-3.1.2.tar.gz
[root@node1 ~]# tar -xzvf hadoop-3.1.2.tar.gz -C /opt/bigdata/
2.配置hadoop環境變量
1.配置環境變量
[root@node1 ~]# vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
JAVA_HOME=/usr/java/jdk1.8.0_211-amd64
HADOOP_HOME=/opt/bigdata/hadoop-3.1.2
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_HOME
export HADOOP_HOME
export PATH
~
:wq!
2.驗證環境變量
#1.使環境變量生效
[root@node1 ~]# source .bash_profile
#2.顯示hadoop的版本信息
[root@node1 ~]# hadoop version
#3.顯示出hadoop版本信息表示安裝和環境變量成功.
Hadoop 3.1.2
Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Compiled by sunilg on 2019-01-29T01:39Z
Compiled with protoc 2.5.0
From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9
This command was run using /opt/bigdata/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar
[root@node1 ~]#
hadoop用戶下也需要按照root用戶配置環境變量的方式操作一下
3.配置hadoop-env.sh
這個文件只需要配置JAVA_HOME的值即可,在文件中找到export JAVA_HOME字眼的位置，刪除最前面的#
export JAVA_HOME=/usr/java/jdk1.8.0_211-amd64
[root@node1 ~]# cd /opt/bigdata/hadoop-3.1.2/etc/hadoop/
You have new mail in /var/spool/mail/root
[root@node1 hadoop]# pwd
/opt/bigdata/hadoop-3.1.2/etc/hadoop
[root@node1 hadoop]# vi hadoop-env.sh
4.配置core-site.xml
切換到cd /opt/bigdata/hadoop-3.1.2/etc/hadoop/目錄下
[root@node1 ~]# cd /opt/bigdata/hadoop-3.1.2/etc/hadoop/
<configuration>
<!-- 指定hdfs的namenode主機的hostname -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:9000</value>
</property>
<!-- io操作流的配置 -->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<!--hadoop集群臨時數據存儲目錄-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/bigdata/hadoop-3.1.2/tmpdata</value>
</property>
</configuration>
5.配置hdfs-site.xml
配置/opt/bigdata/hadoop-3.1.2/etc/hadoop/目錄下的hdfs-site.xml
<configuration>
<!--namenode元數據存儲目錄-->
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/bigdata/hadoop-3.1.2/hadoop/hdfs/name/</value>
</property>
<!--指定block塊的的大小-->
<property>
<name>dfs.blocksize</name>
<value>268435456</value>
</property>
<!-- -->
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<!--工作節點的數據塊存儲目錄 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/bigdata/hadoop-3.1.2/hadoop/hdfs/data/</value>
</property>
<!--block的副本數-->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
6.配置mapred-site.xml
配置/opt/bigdata/hadoop-3.1.2/etc/hadoop/目錄下的mapred-site.xml
<!--指定運行mapreduce的環境是yarn -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
7.配置yarn-site.xml
配置/opt/bigdata/hadoop-3.1.2/etc/hadoop/目錄下的yarn-site.xml
<configuration>
<!--指定resourcemanager的位置-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>node1:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>node1:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>node1:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>node1:18141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>node1:18088</value>
</property>
</configuration>
8.編輯works
此文件用於配置集群有多少個數據節點,我們把node2，node3作為數據節點,node1作為集群管理節點.
配置/opt/bigdata/hadoop-3.1.2/etc/hadoop/目錄下的works
[root@node1 hadoop]# vi workers
#將localhost這一行刪除掉
node2
node3
~
4.3 遠程復制hadoop到集群機器
#1.進入到root用戶家目錄下
[root@node1 hadoop]# cd ~
#2.使用scp遠程拷貝命令將root用戶的環境變量配置文件復制到node2
[root@node1 ~]# scp .bash_profile root@node2:~
.bash_profile 100% 338 566.5KB/s 00:00
#3.使用scp遠程拷貝命令將root用戶的環境變量配置文件復制到node3
[root@node1 ~]# scp .bash_profile root@node3:~
.bash_profile 100% 338 212.6KB/s 00:00
[root@node1 ~]#
#4.進入到hadoop的share目錄下
[root@node1 ~]# cd /opt/bigdata/hadoop-3.1.2/share/
You have new mail in /var/spool/mail/root
[root@node1 share]# ll
total 0
drwxr-xr-x 3 1001 1002 20 Jan 29 12:05 doc
drwxr-xr-x 8 1001 1002 88 Jan 29 11:36 hadoop
#5.刪除doc目錄,這個目錄存放的是用戶手冊，比較大，等會兒下面進行遠程復制的時候時間比較長，刪除后節約復制時間
[root@node1 share]# rm -rf doc/
[root@node1 share]# cd ~
You have new mail in /var/spool/mail/root
[root@node1 ~]# scp -r /opt root@node2:/
[root@node1 ~]# scp -r /opt root@node3:/
4.4 使集群所有機器環境變量生效
在node2，node3的root用戶家目錄下使環境變量生效
node2節點如下操作:
[root@node2 hadoop-3.1.2]# cd ~
[root@node2 ~]# source .bash_profile
[root@node2 ~]# hadoop version
Hadoop 3.1.2
Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Compiled by sunilg on 2019-01-29T01:39Z
Compiled with protoc 2.5.0
From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9
This command was run using /opt/bigdata/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar
[root@node2 ~]#
node3節點如下操作:
[root@node3 bin]# cd ~
[root@node3 ~]# source .bash_profile
[root@node3 ~]# hadoop version
Hadoop 3.1.2
Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Compiled by sunilg on 2019-01-29T01:39Z
Compiled with protoc 2.5.0
From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9
This command was run using /opt/bigdata/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar
[root@node3 ~]#
5.修改hadoop安裝目錄的權限
node2，node3也需要進行如下操作
#1.修改目錄所屬用戶和組為hadoop:hadoop
[root@node1 ~]# chown -R hadoop:hadoop /opt/
You have new mail in /var/spool/mail/root
You have new mail in /var/spool/mail/root
#2.修改目錄所屬用戶和組的權限值為755
[root@node1 ~]# chmod -R 755 /opt/
[root@node1 ~]# chmod -R g+w /opt/
[root@node1 ~]# chmod -R o+w /opt/
[root@node1 ~]#
6.格式化hadoop
#切換
[root@node1 ~]# su - hadoop
[hadoop@node1 hadoop]$ hdfs namenode -format
2019-06-30 16:11:35,914 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1/192.168.200.11
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.2
#此處省略部分日志
2019-06-30 16:11:36,636 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.200.11
************************************************************/
[hadoop@node1 hadoop]$
7.啟動集群
[hadoop@node1 ~]$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [node1]
Starting datanodes
Starting secondary namenodes [node1]
Starting resourcemanager
Starting nodemanagers
#使用jps顯示java進程
[hadoop@node1 ~]$ jps
40852 ResourceManager
40294 NameNode
40615 SecondaryNameNode
41164 Jps
[hadoop@node1 ~]$
在瀏覽器地址欄中輸入:http://192.168.200.11:9870查看namenode的web界面.
8.運行mapreduce程序
#1.使用hdfs dfs -ls / 命令瀏覽hdfs文件系統，集群剛開始搭建好，由於沒有任何目錄所以什么都不顯示.
[hadoop@node1 ~]$ hdfs dfs -ls /
#2.創建測試目錄
[hadoop@node1 ~]$ hdfs dfs -mkdir /test
#3.在此使用hdfs dfs -ls 發現我們剛才創建的test目錄
[hadoop@node1 ~]$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2019-06-30 17:23 /test
#4.使用touch命令在linux本地目錄創建一個words文件
[hadoop@node1 ~]$ touch words
#5.文件中輸入如下內容
[hadoop@node1 ~]$ vi words
i love you
are you ok
#6.將創建的本地words文件上傳到hdfs的test目錄下
[hadoop@node1 ~]$ hdfs dfs -put words /test
#7.查看上傳的文件是否成功
[hadoop@node1 ~]$ hdfs dfs -ls -r /test
Found 1 items
-rw-r--r-- 3 hadoop supergroup 23 2019-06-30 17:28 /test/words
#/test/words 是hdfs上的文件存儲路徑 /test/output是mapreduce程序的輸出路徑，這個輸出路徑是不能已經存在的路徑，mapreduce程序運行的過程中會自動創建輸出路徑，數據路徑存在的話會報錯，這里需要同學注意下.
[hadoop@node1 ~]$ hadoop jar /opt/bigdata/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /test/words /test/output
2019-06-30 17:32:23,685 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.200.11:18040
2019-06-30 17:32:24,060 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1561886252942_0001
2019-06-30 17:32:24,215 INFO input.FileInputFormat: Total input files to process : 1
2019-06-30 17:32:24,291 INFO mapreduce.JobSubmitter: number of splits:1
2019-06-30 17:32:24,394 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1561886252942_0001
2019-06-30 17:32:24,395 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-06-30 17:32:24,592 INFO conf.Configuration: resource-types.xml not found
2019-06-30 17:32:24,593 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-06-30 17:32:24,806 INFO impl.YarnClientImpl: Submitted application application_1561886252942_0001
2019-06-30 17:32:24,861 INFO mapreduce.Job: The url to track the job: http://node1:18088/proxy/application_1561886252942_0001/
2019-06-30 17:32:24,862 INFO mapreduce.Job: Running job: job_1561886252942_0001
2019-06-30 17:32:33,025 INFO mapreduce.Job: Job job_1561886252942_0001 running in uber mode : false
2019-06-30 17:32:33,030 INFO mapreduce.Job: map 0% reduce 0%
2019-06-30 17:32:39,174 INFO mapreduce.Job: map 100% reduce 0%
2019-06-30 17:32:43,229 INFO mapreduce.Job: map 100% reduce 100%
2019-06-30 17:32:43,266 INFO mapreduce.Job: Job job_1561886252942_0001 completed successfully
2019-06-30 17:32:43,369 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=54
FILE: Number of bytes written=432335
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=116
HDFS: Number of bytes written=28
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3346
Total time spent by all reduces in occupied slots (ms)=2290
Total time spent by all map tasks (ms)=3346
Total time spent by all reduce tasks (ms)=2290
Total vcore-milliseconds taken by all map tasks=3346
Total vcore-milliseconds taken by all reduce tasks=2290
Total megabyte-milliseconds taken by all map tasks=3426304
Total megabyte-milliseconds taken by all reduce tasks=2344960
Map-Reduce Framework
Map input records=2
Map output records=6
Map output bytes=46
Map output materialized bytes=54
Input split bytes=93
Combine input records=6
Combine output records=5
Reduce input groups=5
Reduce shuffle bytes=54
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=472
CPU time spent (ms)=1380
Physical memory (bytes) snapshot=492576768
Virtual memory (bytes) snapshot=5577179136
Total committed heap usage (bytes)=407371776
Peak Map Physical memory (bytes)=294256640
Peak Map Virtual memory (bytes)=2788634624
Peak Reduce Physical memory (bytes)=198320128
Peak Reduce Virtual memory (bytes)=2788544512
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=23
File Output Format Counters
Bytes Written=28
[hadoop@node1 ~]$ hdfs dfs -ls -r /test/output
Found 2 items
-rw-r--r-- 3 hadoop supergroup 28 2019-06-30 17:32 /test/output/part-r-00000
-rw-r--r-- 3 hadoop supergroup 0 2019-06-30 17:32 /test/output/_SUCCESS
[hadoop@node1 ~]$ hdfs dfs -text /test/output/part-r-00000
are 1
i 1
love 1
ok 1
you 2
[hadoop@node1 ~]$
至此三節點的hadoop集群環境搭建完成，謝謝大家能夠有耐心的看完!
