HDFS 讀取、寫入、遍歷文件夾獲取文件全路徑、append

版權聲明：本文為博主原創文章，未經博主同意不得轉載。安金龍 的博客。 https://blog.csdn.net/smile0198/article/details/37573081

1、從HDFS中讀取數據

Configuration conf = getConf();
  Path path = new Path(pathstr); 
  FileSystem fs = FileSystem.get(conf);
   FSDataInputStream fsin= fs.open(path ); 
   BufferedReader br =null;
   String line ;
   try{
    br = new BufferedReader(new InputStreamReader(fsin));
       while ((line = br.readLine()) != null) {
         System.out.println(line);
        } 
   }finally{
    br.close();
   }

2、寫HDFS

  Configuration conf = getConf();
  Path path = new Path(mid_sort); 
  FileSystem fs = FileSystem.get(conf); 
  FSDataOutputStream out = fs.create(resultpath);
  out.write(sb.toString().getBytes());
  out.close();

3、遍歷文件夾 獲取文件 全路徑

/**
  * 得到一個文件夾(不包含子文件夾)下的全部名字匹配上pattern的文件名稱
  * @param fs
  * @param folderPath
  * @param pattern 用於匹配文件名稱的正則
  * @return
  * @throws IOException
  */
 public static List<Path> getFilesUnderFolder(FileSystem fs, Path folderPath, String pattern) throws IOException {
  List<Path> paths = new ArrayList<Path>();
  if (fs.exists(folderPath)) {
   FileStatus[] fileStatus = fs.listStatus(folderPath);
   for (int i = 0; i < fileStatus.length; i++) {
    FileStatus fileStatu = fileStatus[i];
    if (!fileStatu.isDir()) {//僅僅要文件
     Path oneFilePath = fileStatu.getPath();
     if (pattern == null) {
      paths.add(oneFilePath);
     } else {
      if (oneFilePath.getName().contains(pattern)) {
       paths.add(oneFilePath);
      }
     }  
    }
   }
  }
  return paths;
 }

4、追加數據 append

  public static boolean appendRTData(String hdfsFile, String appendFile) {
    boolean flag = false;

    Configuration conf = new Configuration();
    FileSystem fs = null;
    try {
      fs = FileSystem.get(URI.create(hdfsFile), conf);
      InputStream in = new BufferedInputStream(new FileInputStream(appendFile));
      OutputStream out = fs.append(new Path(hdfsFile));
      IOUtils.copyBytes(in, out, 4096, true);
    } catch (IOException e) {
      e.printStackTrace();
    }

    return flag;
  }

***********************************************************************************************************************************************

***********************************************************************************************************************************************

異常信息

1、Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: ns6

原因是沒有載入hdfs的配置信息，須要加入以下的代碼：

conf.addResource(new Path("/xxxx/hdfs-site.xml"));//path是配置文件地址

假設配置了環境變量能夠在不同的機器上使用:

conf.addResource(new Path(System.getenv("HADOOP_CONF") + "/hdfs-site.xml"));