項目實戰 從 0 到 1 學習之Flink （20）Flink讀取hdfs文件

接一下以一個示例配置來介紹一下如何以Flink連接HDFS

1. 依賴HDFS

pom.xml 添加依賴

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-hadoop-compatibility_2.11</artifactId>
        <version>${flink.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>

2. 配置 HDFS

將hdfs-site.xml和core-site.xml放入到src/main/resources目錄下面

3. 讀取HDFS上面文件

  final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
        DataSource<String> text = env.readTextFile("hdfs://flinkhadoop:9000/user/wuhulala/input/core-site.xml");

---

TIP

1. 請關閉HDFS 權限,不關閉需要把認證copy到resources目錄下

 <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>