WaterDrop 系列報錯


1.使用WaterDrop 從kafka中消費數據,寫入到ClickHouse

1.1 環境

SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
clickhouse-1.1.54236-4.el7.x86_64
waterdrop-1.4.2

waterdrop 配置文件

spark {
  spark.streaming.batchDuration = 5
  spark.app.name = "Waterdrop"
  spark.executor.instances = 2
  spark.executor.cores = 1
  spark.executor.memory = "1g"
}

input {
  kafkaStream {
    topics = "waterdrop"
    consumer.bootstrap.servers = "cdh01:9092,cdh02:9092,cdh03:9092"
    consumer.group.id = "waterdrop_group"
    }
}

filter {
  split {
    fields = ["FlightDate", "Year"]
    delimiter = ","
  }
}

output {
  clickhouse {
    host = "cdh03:8123"
    clickhouse.socket_timeout = 50000
    database = "ontime"
    table = "ontime_test"
    fields = ["FlightDate", "Year"]
    username = "default"
    password = "default"
    bulk_size = 20000
  }
}

啟動waterdrop 報錯:

Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
 at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1023)
 at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:59)
 at ru.yandex.clickhouse.

原因:

  1. clickhouse 依賴於 clickhouse-jdbc2.0,clickhouse-jdbc2.0 又依賴於 httpclient 4.5.2
    參考地址:https://www.mvnjar.com/ru.yandex.clickhouse/clickhouse-jdbc/0.2/detail.html

  2. SPARK2-2.3.0.cloudera4-1.cdh5.13.3 沒有 httpclient 4.5.2,只有 httpcore

解決方法:

  1. 下載httpclient 4.5.2

  2. httpclient 放入 Spark 的安裝目錄;如:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/jars


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM