MapReduce 編程(單次計算)出現的問題


問題1:

Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set

原因:

將FileOutputFormat.setOutputPath(job, new Path(args[1]));寫成了FileInputFormat

 

問題2:

[demo1@master hadoop-2.7.3]$ hadoop jar wordCount.jar org.demo1.hadoop.wordcount.WordMain /test.txt /word
/test.txt
/word
20/02/25 20:00:57 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.122.1:8032
20/02/25 20:00:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
20/02/25 20:00:59 INFO input.FileInputFormat: Total input paths to process : 1
20/02/25 20:01:00 INFO mapreduce.JobSubmitter: number of splits:1
20/02/25 20:01:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582682268936_0001
20/02/25 20:01:01 INFO impl.YarnClientImpl: Submitted application application_1582682268936_0001
20/02/25 20:01:01 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1582682268936_0001/
20/02/25 20:01:01 INFO mapreduce.Job: Running job: job_1582682268936_0001
20/02/25 20:01:12 INFO mapreduce.Job: Job job_1582682268936_0001 running in uber mode : false
20/02/25 20:01:12 INFO mapreduce.Job: map 0% reduce 0%
20/02/25 20:01:20 INFO mapreduce.Job: Task Id : attempt_1582682268936_0001_m_000000_0, Status : FAILED
Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was :class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:414)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
at java.lang.Class.asSubclass(Class.java:3404)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:887)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1004)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
... 9 more

20/02/25 20:01:25 INFO mapreduce.Job: Task Id : attempt_1582682268936_0001_m_000000_1, Status : FAILED
Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was :class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:414)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
at java.lang.Class.asSubclass(Class.java:3404)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:887)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1004)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
... 9 more

20/02/25 20:01:30 INFO mapreduce.Job: Task Id : attempt_1582682268936_0001_m_000000_2, Status : FAILED

 

原因:

應該導入import org.apache.hadoop.io.Text;

  但是導入了import com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider.Text;


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM