Streamset 安裝與實踐

注意：要先注冊才能下載。

官網地址：

https://accounts.streamsets.com/install/instruction/data-collector/linux/common-tarball

1、官方步驟

1、安裝jdk8

2、系統文件數設置必須要大於32768

3、注冊之后下載是自動的

4、解壓：tar xvzf streamsets-datacollector-common-3.22.3.tgz

5、cd streamsets-datacollector-3.22.3

6、啟動：bin/streamsets dc

7、http://ip:18360

8、要用注冊的賬戶登錄

 

 2、創建自己的PipLine

 

3、添加組件

 

 

4、Mysql組件配置

 

 

 

 

 

 5、Kafka組件配置

 

 

 

 

 

 FAQ

 

1、mysql只支持mysql5.7以上

 

2、com.streamsets.pipeline.api.base.OnRecordErrorException: TEXT_GENERATOR_01 - Record 'mysql-bin.000001:3800' is missing field /text
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.writeOneMessagePerRecord(KafkaTarget.java:255)
    at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.write(KafkaTarget.java:84)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:303)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
    at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)

這個錯誤就是kafka設置為Text導致，因為從mysql抽取的數據為json格式。如：

{"BinLogFilename":"mysql-bin.000001","Type":"UPDATE","Table":"weblog","ServerId":2,"BinLogPosition":6408,"Database":"test","OldData":{"args":"123","id":10,"method":"12df34","create_time":1625251291000,"url":"name1qw0","ip":"123"},"Data":{"args":"123","id":10,"method":"12df34","create_time":1625251291000,"url":"name3423qw0","ip":"123"},"Timestamp":1626261330000,"Offset":"mysql-bin.000001:6408"}

 

3、mysql驅動一定要正確，這個看日志，如果顯示

表示你缺少對應高版本的mysql驅動包