IDEA遠程連接Hadoop

Win

1.Hadoop配置

下載並配置到本地環境

HADOOP_HOME D:\Tools\hadoop-2.7.7\hadoop-2.7.7
HADOOP_PREFIX   D:\Tools\hadoop-2.7.7\hadoop-2.7.7
PATH += %HADOOP_HOME%\bin

2.Maven pom.xml

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.7.7</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.7.7</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.7.7</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    <version>2.7.7</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-common</artifactId>
    <version>2.7.7</version>
</dependency>

3.core-site.xml

hadoop集群里的core-site.xml文件拿下來，放到resource目錄下（記在etc/hosts里的名字記得換成ip地址）

4.替換

windows下開發的可執行文件肥腸重要，就是\(HADOOP_HOME\)/bin下的windows可執行文件或者dll，下載下來的覆蓋掉bin下的目錄下即可

(hadoop.dll winutils.exe winutils.pdb......balabala)(版本版本！！！)

5.碼代碼

String filePath = "hdfs://172.18.145.167//user//zwj//wordCountTest.txt";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(filePath),conf);
InputStream in = null;
try
{
    in = fs.open(new Path(filePath));
    IOUtils.copyBytes(in,System.out,4096,false);
}finally {
    IOUtils.closeStream(in);
}

完全ojbk

問題

1.org.apache.hadoop.hdfs.BlckMissingException

子節點datanode崩了，去log里找問題

2.Exception in thread "main" java.lang.UnsatisfiedLinkError:...nativeIO$Wi.....

win下可執行文件版本不對

3.hadoop Permission denied : USER=ZWJ,access=WRITE......

windows系統的用戶名和ubuntu上的用戶名不一致，炸

hadoop fs -chmod 777 filePath（filePath下文件所有者、群組用戶、其他用戶均可讀可寫可執行OTZ）