服務器准備
啟動hadoop最小集群的典型配置是3台服務器, 一台作為Master, NameNode, 兩台作為Slave, DataNode. 操作系統使用的Ubuntu18.04 Server, 安裝過程就省略了, 使用的是LVM文件系統, XFS文件格式, 為了避免浪費空間, 除了划分1G給/boot以外, 其他都划為/
服務器規划
192.168.1.148 vm148 -- 作為master, NameNode, ResourceManager
192.168.1.149 vm149 -- 作為slave, DataNode, NodeManager
192.168.1.150 vm150 -- 作為slave, DataNode, NodeManager
注意: 這里是第一個坑, 主機名里面不能帶下划線 _ , 會導致DataServer創建socket失敗無法啟動.
安裝后的升級
sudo apt update sudo apt upgrade
添加普通用戶
用於運行hadoop的受限用戶, 我習慣用tomcat作為用戶名, 這里使用adduser而不是useradd, 因為后者不帶參數時, 有時候不會創建home目錄
sudo adduser tomcat # 按提示輸入
.如果是虛機, 這時候就可以以當前狀態創建模板了.
設置hostname和hosts
# view current hostname sudo hostnamectl status # set sudo hostnamectl set-hostname vm148 # add entries to hosts sudo vi /etc/hosts # add following lines 192.168.1.148 vm148 192.168.1.149 vm149 192.168.1.150 vm150
.依次將服務器設置為vm148, vm149, vm150. 重啟后登入檢查是否生效, 互相ping看看是否生效
對tomcat用戶互相添加免密登錄
# 生成id_rsa和id_rsa.pub ssh-keygen cd .ssh/ # 創建 authorized_keys mv id_rsa.pub authorized_keys # 注意權限必須是600 chmod 600 authorized_keys # 將本服務器的私鑰改名為id_rsa_mine mv id_rsa id_rsa_mine # 修改config vi config # 添加如下內容 Host vm149 IdentityFile ~/.ssh/id_rsa_mine User tomcat Host vm150 IdentityFile ~/.ssh/id_rsa_mine User tomcat Host vm148 IdentityFile ~/.ssh/id_rsa_mine User tomcat # 如果是master機器, 還需要添加如下, 用於啟動Secondary name server Host 0.0.0.0 IdentityFile ~/.ssh/id_rsa_mine User tomcat
將各個服務器的authorized_keys的內容互相合並, 最后各服務器的authorized_keys文件都是一樣的.
在以上工作完成后, 從各個機器嘗試ssh tomcat@[主機名], 確保登錄沒問題, 也避免在啟動服務時提示發現新key是否接受
防火牆ufw
如果是初次設置, 建議關閉, 確保不會因為防火牆而導致服務啟動失敗, 可以等服務配置完成后, 再根據實際的端口, 打開並配置ufw
sudo ufw disable
配置JDK
將jdk解壓縮至/opt/jdk, 並創建latest軟鏈, 完成后結構如下
$ ll /opt/jdk/ total 0 drwxr-xr-x 7 root root 245 Oct 6 13:58 jdk1.8.0_192/ lrwxrwxrwx 1 root root 12 Jan 18 05:49 latest -> jdk1.8.0_192/
需要將jps軟鏈到/usr/bin
cd /usr/bin sudo ln -s /opt/jdk/latest/bin/jps jps
配置Hadoop
將hadoop解壓縮至 /opt/hadoop, 並創建latest 軟鏈, 完成后目錄結構如下
$ ll /opt/hadoop/ total 0 drwxr-xr-x 9 root root 149 Nov 13 15:15 hadoop-2.9.2/ lrwxrwxrwx 1 root root 12 Jan 18 10:26 latest -> hadoop-2.9.2/
修改配置文件 etc/hadoop/hadoop-env.sh
需要修改的變量有兩處
# The java implementation to use. export JAVA_HOME=/opt/jdk/latest # Where log files are stored. $HADOOP_HOME/logs by default. export HADOOP_LOG_DIR=/home/tomcat/run/hadoop/logs
修改配置文件 etc/hadoop/yarn-env.sh
需要修改的變量有兩處
# some Java parameters export JAVA_HOME=/opt/jdk/latest # default log directory & file export YARN_LOG_DIR=/home/tomcat/run/yarn/logs
修改配置文件/etc/hadoop/slaves
將內容修改為兩個slave的主機名
vm149 vm150
修改配置文件/etc/hadoop/core-site.xml
添加以下內容. 配置明細需要參考 share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/tomcat/run/hadoop</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://vm148:9000</value> </property> </configuration>
修改配置文件/etc/hadoop/hdfs-site.xml
添加以下內容
<configuration> <property> <name>dfs.replication</name> <value>2</value> </property> </configuration>
修改配置文件/etc/hadoop/mapred-site.xml
添加以下內容
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
修改配置文件/etc/hadoop/yarn-site.xml
添加以下內容. 配置明細需要參考 share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
<configuration> <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>vm148</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
將配置好的hadoop, 按當前的目錄結構, 復制到另外兩個服務器中
啟動Hadoop
第一次啟動前, 需要format nameserver, 在master服務器上執行
/opt/hadoop/latest/bin/hdfs namenode -format
然后啟動hdfs服務
/opt/hadoop/latest/sbin/start-dfs.sh
然后啟動yarn服務
/opt/hadoop/latest/sbin/start-yarn.sh
每一步, 都需要用jps命令查看服務是否正常啟動, 對於master服務器, 正常啟動后應該顯示如下進程
tomcat@vm148:/opt$ jps 3173 SecondaryNameNode 3495 ResourceManager 4583 Jps 2906 NameNode
slave服務器
tomcat@vm149:~/run$ jps 3074 NodeManager 2691 DataNode 3591 Jps
.
WEB訪問
服務啟動后, 可以通過 http://vm148:50070/ 訪問web界面
服務端口
master端
21, FTP for ? 8030, YARN resourcemanager scheduler 8031, YARN resourcemanager tracker 8032, YARN resourcemanager 8033, YARN resourcemanager admin 8088, YARN resourcemanager webapp 8090, YARN resourcemanager webapp https 9000, HDFS 50070, WEB UI 50090,
slave, data node端
50075
運行WordCount Example
首先編譯java, 生成class, 生成jar. 因為JAVA_HOME已經在hadoop里配置過, 而PATH在此環境不需要, 只需要配置一個tools.jar的classpath就可以了
export HADOOP_CLASSPATH=/opt/jdk/latest/lib/tools.jar /opt/hadoop/latest/bin/hadoop com.sun.tools.javac.Main WordCount.java /opt/jdk/latest/bin/jar cf wc.jar WordCount*.class
然后將兩個輸入文件上載到hdfs.
/opt/hadoop/latest/bin/hadoop fs -put file01 /workspace/input/ /opt/hadoop/latest/bin/hadoop fs -ls /workspace/input /opt/hadoop/latest/bin/hadoop fs -put file02 /workspace/input/ /opt/hadoop/latest/bin/hadoop fs -cat /workspace/input/file01 /opt/hadoop/latest/bin/hadoop fs -cat /workspace/input/file02
一開始我在這里遇到了個坑: 我把文件放到/tmp/下面去了, 把/tmp作為輸入目錄, 結果在運行中yarn會把staging信息存在 /tmp/hadoop-yarn/staging 文件中, 然后任務就拋異常了. 教訓就是: 任務文件不要放到/tmp下
執行任務
/opt/hadoop/latest/bin/hadoop jar wc.jar WordCount /workspace/input /workspace/output
這里最后一個路徑是輸出路徑, 這個路徑在運行任務前不能存在, 否則也會報錯
最后的執行結果
tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop jar wc.jar WordCount /workspace/input /workspace/output 19/01/30 08:24:55 INFO client.RMProxy: Connecting to ResourceManager at vm148/192.168.1.148:8032 19/01/30 08:24:55 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/01/30 08:24:56 INFO input.FileInputFormat: Total input files to process : 2 19/01/30 08:24:56 INFO mapreduce.JobSubmitter: number of splits:2 19/01/30 08:24:56 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/01/30 08:24:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547812325179_0004 19/01/30 08:24:56 INFO impl.YarnClientImpl: Submitted application application_1547812325179_0004 19/01/30 08:24:56 INFO mapreduce.Job: The url to track the job: http://vm148:8088/proxy/application_1547812325179_0004/ 19/01/30 08:24:56 INFO mapreduce.Job: Running job: job_1547812325179_0004 19/01/30 08:25:03 INFO mapreduce.Job: Job job_1547812325179_0004 running in uber mode : false 19/01/30 08:25:03 INFO mapreduce.Job: map 0% reduce 0% 19/01/30 08:25:10 INFO mapreduce.Job: map 100% reduce 0% 19/01/30 08:25:18 INFO mapreduce.Job: map 100% reduce 100% 19/01/30 08:25:18 INFO mapreduce.Job: Job job_1547812325179_0004 completed successfully 19/01/30 08:25:18 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=97 FILE: Number of bytes written=594622 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=266 HDFS: Number of bytes written=38 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=10309 Total time spent by all reduces in occupied slots (ms)=3850 Total time spent by all map tasks (ms)=10309 Total time spent by all reduce tasks (ms)=3850 Total vcore-milliseconds taken by all map tasks=10309 Total vcore-milliseconds taken by all reduce tasks=3850 Total megabyte-milliseconds taken by all map tasks=10556416 Total megabyte-milliseconds taken by all reduce tasks=3942400 Map-Reduce Framework Map input records=2 Map output records=10 Map output bytes=96 Map output materialized bytes=103 Input split bytes=210 Combine input records=10 Combine output records=8 Reduce input groups=5 Reduce shuffle bytes=103 Reduce input records=8 Reduce output records=5 Spilled Records=16 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=379 CPU time spent (ms)=2090 Physical memory (bytes) snapshot=778280960 Virtual memory (bytes) snapshot=5914849280 Total committed heap usage (bytes)=507510784 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=56 File Output Format Counters Bytes Written=38 tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop fs -ls /workspace/output Found 2 items -rw-r--r-- 2 tomcat supergroup 0 2019-01-30 08:25 /workspace/output/_SUCCESS -rw-r--r-- 2 tomcat supergroup 38 2019-01-30 08:25 /workspace/output/part-r-00000 tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop fs -cat /workspace/output/part-r-00000 Day 2 Good 2 Hadoop 2 Hello 2 World 2
一個簡單的Map Reduce 例子
輸入的內容格式是這樣的, 每一行是一個日志記錄, 記錄了用戶, IP和時間戳, 需要統計每個 (用戶+IP) 出現的次數
1571 76 738 legnd 166.111.8.133 870876781 1572 121 697 kuoc 202.116.65.16 870909489 1573 121 697 kuoc 202.116.65.16 870910644 1574 121 739 maerick 870926284
代碼 pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.rockbb</groupId> <artifactId>hdtask</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <name>HD Task</name> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.4.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.4.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-core</artifactId> <version>2.4.1</version> </dependency> </dependencies> <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.3</version> <configuration> <source>1.8</source> <target>1.8</target> <encoding>UTF-8</encoding> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-resources-plugin</artifactId> <configuration> <encoding>UTF-8</encoding> </configuration> </plugin> </plugins> </pluginManagement> </build> </project>
代碼 DataBean.java
package com.rockbb.hdtask; import org.apache.hadoop.io.Writable; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; public class DataBean implements Writable { private String nameIp; private long count; public DataBean() { } public DataBean(String nameIp, long count) { this.nameIp = nameIp; this.count = count; } public String getNameIp() { return nameIp; } public void setNameIp(String nameIp) { this.nameIp = nameIp; } public long getCount() { return count; } public void setCount(long count) { this.count = count; } /** * Important: this will be use for the final output. */ @Override public String toString() { return this.nameIp + "\t" + this.count; } @Override public void write(DataOutput dataOutput) throws IOException { dataOutput.writeUTF(nameIp); dataOutput.writeLong(count); } @Override public void readFields(DataInput dataInput) throws IOException { this.nameIp = dataInput.readUTF(); this.count = dataInput.readLong(); } }
代碼 IpCount.java
package com.rockbb.hdtask; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.IOException; public class IpCount { public static class IpMapper extends Mapper<LongWritable, Text, Text, DataBean> { @Override public void map(LongWritable keyIn, Text valueIn, Context context) throws IOException, InterruptedException { String line = valueIn.toString(); String[] fields = line.split("\t"); String keyOut = fields[3] + '-' + fields[4]; long valueOut = 1; DataBean bean = new DataBean(keyOut, valueOut); context.write(new Text(keyOut), bean); } } public static class IpReducer extends Reducer<Text, DataBean, Text, DataBean> { @Override public void reduce(Text keyIn, Iterable<DataBean> valuesIn, Context context) throws IOException, InterruptedException { long total = 0; for (DataBean bean : valuesIn) { total += bean.getCount(); } DataBean bean = new DataBean("", total); context.write(keyIn, bean); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf); job.setJarByClass(IpCount.class); job.setMapperClass(IpMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(DataBean.class); FileInputFormat.addInputPath(job, new Path(args[0])); job.setReducerClass(IpReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(DataBean.class); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } }
運行命令
/opt/hadoop/latest/bin/hadoop jar hdtask.jar com.rockbb.hdtask.IpCount /workspace/input/ /workspace/output3
.數據文件有2.3GB, 因為默認的block大小為128MB, 所以提交后產生了19個Map任務和一個Reduce任務, 任務的命令行輸出
19/01/31 10:08:01 INFO client.RMProxy: Connecting to ResourceManager at vm148/192.168.31.148:8032 19/01/31 10:08:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/01/31 10:08:02 INFO input.FileInputFormat: Total input files to process : 1 19/01/31 10:08:02 INFO mapreduce.JobSubmitter: number of splits:19 19/01/31 10:08:02 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/01/31 10:08:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547812325179_0008 19/01/31 10:08:03 INFO impl.YarnClientImpl: Submitted application application_1547812325179_0008 19/01/31 10:08:03 INFO mapreduce.Job: The url to track the job: http://vm148:8088/proxy/application_1547812325179_0008/ 19/01/31 10:08:03 INFO mapreduce.Job: Running job: job_1547812325179_0008 19/01/31 10:08:13 INFO mapreduce.Job: Job job_1547812325179_0008 running in uber mode : false 19/01/31 10:08:13 INFO mapreduce.Job: map 0% reduce 0% 19/01/31 10:08:41 INFO mapreduce.Job: map 11% reduce 0% 19/01/31 10:08:45 INFO mapreduce.Job: map 21% reduce 0% 19/01/31 10:08:47 INFO mapreduce.Job: map 23% reduce 0% 19/01/31 10:08:51 INFO mapreduce.Job: map 28% reduce 0% 19/01/31 10:08:53 INFO mapreduce.Job: map 30% reduce 0% 19/01/31 10:08:57 INFO mapreduce.Job: map 31% reduce 0% 19/01/31 10:08:59 INFO mapreduce.Job: map 38% reduce 0% 19/01/31 10:09:09 INFO mapreduce.Job: map 39% reduce 0% 19/01/31 10:09:10 INFO mapreduce.Job: map 40% reduce 0% 19/01/31 10:09:11 INFO mapreduce.Job: map 44% reduce 0% 19/01/31 10:09:14 INFO mapreduce.Job: map 46% reduce 0% 19/01/31 10:09:16 INFO mapreduce.Job: map 48% reduce 0% 19/01/31 10:09:17 INFO mapreduce.Job: map 49% reduce 0% 19/01/31 10:09:22 INFO mapreduce.Job: map 55% reduce 0% 19/01/31 10:09:24 INFO mapreduce.Job: map 56% reduce 0% 19/01/31 10:09:28 INFO mapreduce.Job: map 61% reduce 0% 19/01/31 10:09:40 INFO mapreduce.Job: map 64% reduce 0% 19/01/31 10:09:42 INFO mapreduce.Job: map 64% reduce 7% 19/01/31 10:09:46 INFO mapreduce.Job: map 66% reduce 7% 19/01/31 10:09:48 INFO mapreduce.Job: map 68% reduce 9% 19/01/31 10:09:52 INFO mapreduce.Job: map 71% reduce 9% 19/01/31 10:09:54 INFO mapreduce.Job: map 71% reduce 12% 19/01/31 10:09:58 INFO mapreduce.Job: map 73% reduce 12% 19/01/31 10:09:59 INFO mapreduce.Job: map 74% reduce 12% 19/01/31 10:10:01 INFO mapreduce.Job: map 75% reduce 12% 19/01/31 10:10:04 INFO mapreduce.Job: map 80% reduce 12% 19/01/31 10:10:06 INFO mapreduce.Job: map 81% reduce 12% 19/01/31 10:10:10 INFO mapreduce.Job: map 85% reduce 12% 19/01/31 10:10:12 INFO mapreduce.Job: map 86% reduce 12% 19/01/31 10:10:13 INFO mapreduce.Job: map 87% reduce 12% 19/01/31 10:10:15 INFO mapreduce.Job: map 88% reduce 12% 19/01/31 10:10:18 INFO mapreduce.Job: map 88% reduce 16% 19/01/31 10:10:22 INFO mapreduce.Job: map 90% reduce 16% 19/01/31 10:10:23 INFO mapreduce.Job: map 91% reduce 16% 19/01/31 10:10:24 INFO mapreduce.Job: map 91% reduce 18% 19/01/31 10:10:25 INFO mapreduce.Job: map 92% reduce 18% 19/01/31 10:10:29 INFO mapreduce.Job: map 93% reduce 18% 19/01/31 10:10:31 INFO mapreduce.Job: map 93% reduce 21% 19/01/31 10:10:32 INFO mapreduce.Job: map 94% reduce 21% 19/01/31 10:10:34 INFO mapreduce.Job: map 96% reduce 21% 19/01/31 10:10:35 INFO mapreduce.Job: map 97% reduce 21% 19/01/31 10:10:37 INFO mapreduce.Job: map 98% reduce 23% 19/01/31 10:10:38 INFO mapreduce.Job: map 99% reduce 23% 19/01/31 10:10:41 INFO mapreduce.Job: map 100% reduce 23% 19/01/31 10:10:43 INFO mapreduce.Job: map 100% reduce 30% 19/01/31 10:10:49 INFO mapreduce.Job: map 100% reduce 33% 19/01/31 10:11:25 INFO mapreduce.Job: map 100% reduce 67% 19/01/31 10:11:31 INFO mapreduce.Job: map 100% reduce 70% 19/01/31 10:11:37 INFO mapreduce.Job: map 100% reduce 74% 19/01/31 10:11:43 INFO mapreduce.Job: map 100% reduce 78% 19/01/31 10:11:49 INFO mapreduce.Job: map 100% reduce 83% 19/01/31 10:11:55 INFO mapreduce.Job: map 100% reduce 86% 19/01/31 10:12:01 INFO mapreduce.Job: map 100% reduce 89% 19/01/31 10:12:07 INFO mapreduce.Job: map 100% reduce 93% 19/01/31 10:12:13 INFO mapreduce.Job: map 100% reduce 97% 19/01/31 10:12:18 INFO mapreduce.Job: map 100% reduce 100% 19/01/31 10:12:19 INFO mapreduce.Job: Job job_1547812325179_0008 completed successfully 19/01/31 10:12:19 INFO mapreduce.Job: Counters: 50 File System Counters FILE: Number of bytes read=6635434217 FILE: Number of bytes written=9269615741 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2551940756 HDFS: Number of bytes written=134288980 HDFS: Number of read operations=60 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Killed map tasks=3 Launched map tasks=22 Launched reduce tasks=1 Data-local map tasks=22 Total time spent by all maps in occupied slots (ms)=1737403 Total time spent by all reduces in occupied slots (ms)=178563 Total time spent by all map tasks (ms)=1737403 Total time spent by all reduce tasks (ms)=178563 Total vcore-milliseconds taken by all map tasks=1737403 Total vcore-milliseconds taken by all reduce tasks=178563 Total megabyte-milliseconds taken by all map tasks=1779100672 Total megabyte-milliseconds taken by all reduce tasks=182848512 Map-Reduce Framework Map input records=49458230 Map output records=49458230 Map output bytes=2531297616 Map output materialized bytes=2630214190 Input split bytes=2052 Combine input records=0 Combine output records=0 Reduce input groups=5453085 Reduce shuffle bytes=2630214190 Reduce input records=49458230 Reduce output records=5453085 Spilled Records=174185483 Shuffled Maps =19 Failed Shuffles=0 Merged Map outputs=19 GC time elapsed (ms)=9585 CPU time spent (ms)=389790 Physical memory (bytes) snapshot=5763260416 Virtual memory (bytes) snapshot=39333715968 Total committed heap usage (bytes)=4077912064 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=2551938704 File Output Format Counters Bytes Written=134288980