filebeat+kafka失敗

filebeat端配置

#----------------------------- Kafka output --------------------------------
output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["10.88.128.88:9092"]
  version: "0.8.2"

  topic: 'abc'
  #partition.round_robin:
  #  reachable_only: false

  required_acks: 1
  compression: none 
  #max_message_bytes: 1000000

 

報錯如下：

2017/04/30 08:14:24.060269 log.go:12: WARN producer/broker/0 state change to [closing] because dial tcp: lookup test43 on 172.16.2.1:53: no such host
2017/04/30 08:14:24.060361 log.go:12: WARN producer/leader/rokid/0 state change to [retrying-48]
2017/04/30 08:14:24.060405 log.go:12: WARN producer/leader/rokid/0 abandoning broker 0
2017/04/30 08:14:24.060450 log.go:12: WARN producer/broker/0 shut down
x2017/04/30 08:14:24.160582 log.go:12: WARN client/metadata fetching metadata for [rokid] from broker 10.88.128.88:9092
2017/04/30 08:14:24.164350 log.go:12: WARN producer/broker/0 starting up
2017/04/30 08:14:24.164388 log.go:12: WARN producer/broker/0 state change to [open] on rokid/0
2017/04/30 08:14:24.164405 log.go:12: WARN producer/leader/rokid/0 selected broker 0
2017/04/30 08:14:24.164505 log.go:12: WARN producer/leader/rokid/0 state change to [flushing-48]
2017/04/30 08:14:24.164516 log.go:12: WARN producer/leader/rokid/0 state change to [normal]
i2017/04/30 08:14:24.185439 log.go:12: WARN Failed to connect to broker test43:9092: dial tcp: lookup test43 on 172.16.2.1:53: no such host 2017/04/30 08:14:24.185488 log.go:12: WARN producer/broker/0 state change to [closing] because dial tcp: lookup test43 on 172.16.2.1:53: no such host
2017/04/30 08:14:24.185534 log.go:12: WARN producer/leader/rokid/0 state change to [retrying-49]
2017/04/30 08:14:24.185552 log.go:12: WARN producer/leader/rokid/0 abandoning broker 0
2017/04/30 08:14:24.185592 log.go:12: WARN producer/broker/0 shut down
2017/04/30 08:14:24.285740 log.go:12: WARN client/metadata fetching metadata for [rokid] from broker 10.88.128.88:9092
2017/04/30 08:14:24.287170 log.go:12: WARN producer/broker/0 starting up
2017/04/30 08:14:24.287191 log.go:12: WARN producer/broker/0 state change to [open] on rokid/0
2017/04/30 08:14:24.287205 log.go:12: WARN producer/leader/rokid/0 selected broker 0
2017/04/30 08:14:24.287300 log.go:12: WARN producer/leader/rokid/0 state change to [flushing-49]
2017/04/30 08:14:24.287311 log.go:12: WARN producer/leader/rokid/0 state change to [normal]
2017/04/30 08:14:24.304521 log.go:12: WARN Failed to connect to broker test43:9092: dial tcp: lookup test43 on 172.16.2.1:53: no such host

上網查，filebeat解析的時候要解析到主機名，

所以綁定hosts如下

　　10.88.128.88 test43

問題搞定

按照報錯，filebeat應該是先以配置的IP獲取hostname，然后按照hostname連接kafka，如果通過hostname找不到對方服務器，則報錯，所以需要在filebeat機器上綁定對方機器的hosts，暫不理解filebeat為什么要這么做。

如果目標機器中hosts文件書寫有誤，也會報錯（filebeat向kafka中傳遞數據的時候還會檢查目標機器的hosts文件）

 

開始還以為kafka版本太新，修改為filebeat目前支持的穩定版0.8.2都出問題，反復查，綁定hosts問題搞定。

 

參考網址：http://www.iyunv.com/thread-350295-1-1.html