Mac下報錯'WARN security.UserGroupInformation: PriviledgedActionException as：用戶名...No such file or directory'的一種解決方法

Mac下使用IDEA遠程連接Hadoop調試MapReduce程序，參考網上博客，總會出現如題報錯，下面是我在mac下的一種解決辦法，可以參考。

前期准備

如果想遠程調試，需要做一些准備工作，簡單羅列下。

（1）在本地准備一份了hadoop(有博主直接從集群中copy一份)，設置環境變量。

# hadoop路徑為具體路徑
export HADOOP_HOME=/Users/yangchaolin/hadoop2.6.0/hadoop-2.6.0-cdh5.14.0

（2）IDEA工程下，將本地hadoop中share文件下的資源jar包都引入到項目中。

（3） 准備MapReduce程序，並創建一個application，這個application使用的工作目錄就使用本地hadoop。

map端程序

package com.kaikeba.mapreduce;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * mapreduce's map
 */
public class WordCountMap extends Mapper<LongWritable, Text,Text, IntWritable> {
    //most application should override map method
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //split
        String readLine = value.toString();
        String[] words = readLine.split(" ");
        //words output to disk
        for(String word:words){
            context.write(new Text(word),new IntWritable(1));
        }
    }
}

reduce端程序

package com.kaikeba.mapreduce;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

/**
 * mapreduce's reduce
 */
public class WordCountReduce extends Reducer<Text, IntWritable,Text,IntWritable> {
    //should override reduce method
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        //count words count
        int count=0;
        for(IntWritable i:values){
            count+=i.get();
        }
        //output key value to hdfs
        context.write(key,new IntWritable(count));
    }
}

main函數

package com.kaikeba.mapreduce;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;

/**
 * mapreduce's main method, 'WordCountMain' is mapreduce's job name
 */
public class WordCountMain {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        //check args,args is input path and output path
        if(args==null||args.length!=2){
            System.out.println("please input path");
            System.exit(0);//exit main method
        }
        //create mapreduce job
        Configuration conf=new Configuration();
        //if run mapreduce in cluster,set conf and mapred-site should set yarn
        //conf.set("mapreduce.job.jar","/home/hadoop/IdeaProject/hadoop/target/hadoop-1.0-SNAPSHOT.jar");
        //conf.set("mapreduce.app-submission.cross-platform","true");
        //conf.set("mapreduce.framework.name","yarn");

        /**
         * Creates a new Job with no particular Cluster and a given jobName.
         * A Cluster will be created from the conf parameter only when it's needed
         */
        Job job=Job.getInstance(conf,WordCountMain.class.getSimpleName());//get class simplename
        //set mapreduce job
        job.setJarByClass(WordCountMain.class);//Set the Jar by finding where a given class came from
        //set input/output format
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        //set input/output path
        FileInputFormat.setInputPaths(job,new Path(args[0]));
        FileOutputFormat.setOutputPath(job,new Path(args[1]));
        //set map/reduce class
        job.setMapperClass(WordCountMap.class);
        job.setReducerClass(WordCountReduce.class);
        //add combine class
        job.setCombinerClass(WordCountReduce.class);
        //set map/reduce output key-value type
        //map
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        //reduce
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        //set job number ---> 4 reduce task
        job.setNumReduceTasks(4);
        //job commit
        try{
            job.waitForCompletion(true);
        }catch(Exception e){
            e.printStackTrace();
        }
    }
}

其他資源文件內容省略，因為本文報錯跟資源配置文件沒有關系。

程序使用maven打成jar包，使用hadoop jar 類的全路徑名 輸入路徑 輸出路徑的命令在linux下執行是沒有問題，具體過程省略。

報錯解決

接下來使用IDEA遠程調試hadoop，配置好上文創建的application后，發現執行會報如下錯誤，說找不到路徑或目錄。 

19/10/14 00:53:02 WARN security.UserGroupInformation: PriviledgedActionException as:yangchaolin (auth:SIMPLE) cause:ExitCodeException exitCode=1: chmod: /kkb/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/tempDatas/mapred/staging/yangchaolin90307621/.staging/job_local90307621_0001: No such file or directory

Exception in thread "main" ExitCodeException exitCode=1: chmod: /kkb/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/tempDatas/mapred/staging/yangchaolin90307621/.staging/job_local90307621_0001: No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
    at org.apache.hadoop.util.Shell.run(Shell.java:507)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:882)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:865)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:720)
    at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:498)
    at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:479)
    at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:495)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:616)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:814)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:774)
    at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:178)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:991)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:582)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:612)
    at com.kaikeba.mapreduce.WordCountMain.main(WordCountMain.java:60)

Process finished with exit code 1

（1）嘗試將集群中hadoop安裝目錄權限修改為777，繼續測試，依然報錯。

（2）根據報錯說找不到目錄，參考使用window本的童鞋，發現他們會在磁盤下自動創建一個目錄（這個目錄是在資源配置文件中指定的），查看我的mac發現並沒有這個目錄，嘗試在mac的根目錄下創建主目錄，並賦予777權限，再次執行竟然通過了！！！懷疑跟mac下用戶的權限有關系，因為window下一般用戶都是管理員，而mac不是。

# 進入根目錄
youngchaolinMac:/ yangchaolin$ cd /
# 創建主目錄
youngchaolinMac:/ yangchaolin$ sudo mkdir kkb
Password:
youngchaolinMac:/ yangchaolin$ ls -l
total 45...
drwxr-xr-x   2 root         wheel    68 10 23 22:52 kkb
# 賦予777權限
youngchaolinMac:/ yangchaolin$ sudo chmod -R 777 kkb
youngchaolinMac:/ yangchaolin$ ls -l
total 45...
drwxrwxrwx   2 root         wheel    68 10 23 22:52 kkb

IDEA執行通過，完成MapReduce任務。

集群中查看計算結果，ok。

 網上查了很多博客，有這個報警的各種解決方法，但是很多是window下。如果是mac，可參考本文，這可能是這個報錯的一種解決方法。

 

參考博文：

（1）https://www.cnblogs.com/yjmyzz/p/how-to-remote-debug-hadoop-with-eclipse-and-intellij-idea.html