Mapreduce 測試自帶實例 wordcount

2.7.3版本的hadoop：

jar程序所在目錄：$HADOOP_HOME/shar/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar

 

1.本地創建測試文本：

mkdir /home/hadoop/data  //創建data文件夾
touch testinput.txt  //創建測試文本
vim testinput.txt  //修改文本

//文本添加一行
this is a test log

cat testinput.txt  //檢查txt文本

 

2.hdfs

hadoop fs -ls /  //查看hdfs上的目錄
hadoop fs -mkdir /input  //創建input目錄
hadoop fs -rm -r /output  //如果有output目錄，刪除
hadoop fs -put /home/hadoop/data/testiinput.txt /input  //把測試文本上傳到input目錄上

hadoop jar /home/hadoop/softwares/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input /output  
//執行程序,wordcount為程序的主類名， /input  輸入目錄  /output 輸出目錄（輸出目錄不能存在）


hadoop fs -ls /output  //完成后查看輸出目錄
hadoop fs -cat /output/part-r-00000   //查看輸出結果

 

運行過程：

18/04/16 19:43:15 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.30.129:8032
18/04/16 19:43:18 INFO input.FileInputFormat: Total input paths to process : 1
18/04/16 19:43:18 INFO mapreduce.JobSubmitter: number of splits:1
18/04/16 19:43:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523884458275_0001
18/04/16 19:43:19 INFO impl.YarnClientImpl: Submitted application application_1523884458275_0001
18/04/16 19:43:19 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1523884458275_0001/
18/04/16 19:43:19 INFO mapreduce.Job: Running job: job_1523884458275_0001
18/04/16 19:43:41 INFO mapreduce.Job: Job job_1523884458275_0001 running in uber mode : false
18/04/16 19:43:41 INFO mapreduce.Job:  map 0% reduce 0%
18/04/16 19:43:51 INFO mapreduce.Job:  map 100% reduce 0%
18/04/16 19:44:03 INFO mapreduce.Job:  map 100% reduce 100%
18/04/16 19:44:05 INFO mapreduce.Job: Job job_1523884458275_0001 completed successfully
18/04/16 19:44:05 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=6
        FILE: Number of bytes written=237375
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=100
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=8459
        Total time spent by all reduces in occupied slots (ms)=8037
        Total time spent by all map tasks (ms)=8459
        Total time spent by all reduce tasks (ms)=8037
        Total vcore-milliseconds taken by all map tasks=8459
        Total vcore-milliseconds taken by all reduce tasks=8037
        Total megabyte-milliseconds taken by all map tasks=8662016
        Total megabyte-milliseconds taken by all reduce tasks=8229888
    Map-Reduce Framework
        Map input records=0
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=6
        Input split bytes=100
        Combine input records=0
        Combine output records=0
        Reduce input groups=0
        Reduce shuffle bytes=6
        Reduce input records=0
        Reduce output records=0
        Spilled Records=0
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=264
        CPU time spent (ms)=1460
        Physical memory (bytes) snapshot=287879168
        Virtual memory (bytes) snapshot=3887841280
        Total committed heap usage (bytes)=139845632
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=0

 

結果：

a    1
is    1
log    1
test    1
this    1