structed streaming基礎---跳過的坑

1.spark structed streaming 報如圖下面的錯誤

在pom.xml加這個配置即可

<spark.version>2.3.0</spark.version>
 <scala.version>2.11</scala.version>
 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
  <version>${spark.version}</version>
 </dependency>

或在maven官網搜:spark－sql－kafka  也可以

第二個問題是運行structed streaming就無緣無故報空指針異常，看下報錯日志

18/09/20 18:59:46 ERROR StreamMetadata: Error writing stream metadata StreamMetadata(ea90df83-a9ab-4e23-b03f-90bc3f8581ca) to file:/C:/Users/iwen/AppData/Local/Temp/temporary-c0117371-5f2e-419b-b45c-1dfd7609c644/metadata
java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
......
Exception in thread "main" java.lang.NullPointerException

解決方法

下載這個，放到這個位置(https://github.com/srccodes/hadoop-common-2.2.0-bin)

代碼最上面加一句這個即可

System.setProperty("hadoop.home.dir", "C:\\hadoop-common-2.2.0")