hadoop跑第一個實例過程

第一次跑hadoop實例，中間經過了不少彎路，特此記錄下來：

第一步：建立一個maven過程，pom.xml文件：(打包為jar包)

    <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.7.0</version>
</dependency>

第二步：創建一個WordCount(從官網上copy)：

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

第三步：打jar包：

mvn clean install

第四步：將jar包放入hadoop集群中的master機器上。

第五步：設置hdfs文件輸入目錄

　　在hadoop-2.6.0/etc/hadoop目錄下core-site配置：

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000/</value>
    </property>
    <property>
         <name>hadoop.tmp.dir</name>
         <value>file:/home/localadmin/filedata</value>
    </property>
</configuration>

　　上面可以看到hdfs的根目錄，或者使用命令查看：

 bin/hadoop fs -ls /

設置輸入目錄

在/home/localadmin創建filedata/infile目錄，並創建文件file01，file02

bin/hadoop fs -put /home/localadmin/filedata/infile/
bin/hadoop fs -put /home/localadmin/filedata/infile/file01
bin/hadoop fs -put /home/localadmin/filedata/infile/file02

檢查文件情況命令：

# bin/hadoop fs -ls /home/localadmin/filedata/input
Found 2 items
-rw-r--r-- 3 root supergroup 22 2015-12-25 13:56 /home/localadmin/filedata/input/file01
-rw-r--r-- 3 root supergroup 28 2015-12-25 13:56 /home/localadmin/filedata/input/file02

注意：不要設置輸出目錄：

hadoop 由於進行的是耗費資源的計算，生產的結果默認是不能被覆蓋的，
因此中間結果輸出目錄一定不能存在，否則出現這個錯誤。

第六步：執行命令：

hadoop jar wc.jar com.nonobank.hadoop.WordCount ../filedata/input/ ../filedata/output/

 

參考文獻：

【1】http://blog.sina.com.cn/s/blog_757dbe670101gnj9.html

【2】https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0

 【3】http://blog.itpub.net/26230597/viewspace-1370205/