kafka服務端和客戶端均無法消費


一直報錯:

1 partitions have leader brokers without a matching listener, including [cloudeasy.cmdb.cancel.manage-0]

查看zk服務端,報錯如下:

原因是 watch 太多且路徑太長導致重連的時候超過了 ZK maxBuffer 的限制

把 客戶端從 3.4.6 升級到 3.4.7 就能解決這個問題

先部署zookeeper,默認docker不需要參數,域名訪問地址為zookeeper-dev.cmsp-dev.svc.cluster.local:2181

docker 部署kafka(2.12-2.5.0),啟動參數如下:

sh

-c

 exec kafka-server-start.sh /opt/kafka/config/server.properties --override advertised.host.name=zookeeper-dev.cmsp-dev.svc.cluster.local --override broker.id=-1 --override listeners=PLAINTEXT://:9092 --override advertised.listeners=PLAINTEXT://kafka-dev.cmsp-dev.svc.cluster.local:9092 --override zookeeper.connect=zookeeper-dev.cmsp-dev.svc.cluster.local:2181

 

 

./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181

./kafka-topics.sh --list --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181

./kafka-topics.sh  --delete --topic cloudeasy.dev.mess --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181

如果刪不掉,可以在zookeeper里刪除

deleteall /brokers/topics

 

開發環境創建topic:

./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.billing.executor --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cloudmanager.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.cancel.manage --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.delete.resource --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.update.project --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.user.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181

 

查詢有哪些客戶端group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

查詢指定group的消費情況
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cloudeasy --describe

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group  cloudeasy.billing.consumer

kafka-consumer-groups.sh  --bootstrap-server localhost:9092 --describe --group cloudeasy.monitor.group

 

 

 

客戶端kafka報錯問題:
2021-10-19 13:01:47.223 [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=cmsp.broker.group] 1 partitions have leader brokers without a matching listener, including [cmsp.broker.group-0]
2021-10-19 11:39:58.625 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=cloudeasy.log.consumer] 1 partitions have leader brokers without a matching listener, including [cloudeasy.user.event-0]
1 partitions have leader brokers without a matching listener,沒有主broker了
./kafka-topics.sh --list --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181 --describe --topic cloudeasy.cmdb.update.project
顯示Leader:-1 選主有問題

解決leeder為-1問題,連接zookeeper
./zkCli.sh -server localhost:2181
get /brokers/topics/cmsp.broker.user/partitions/0/state
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":9,"leader":1001,"version":1,"leader_epoch":5,"isr":[1001]}

 

 

get /brokers/topics/cloudeasy.billing.ascription/partitions/0/state
get /brokers/topics/cloudeasy.billing.executor/partitions/0/state
get /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state
get /brokers/topics/cloudeasy.jflow.event/partitions/0/state
get /brokers/topics/cloudeasy.optimization.event/partitions/0/state
get /brokers/topics/cloudeasy.order.budget/partitions/0/state
get /brokers/topics/cloudeasy.order.event/partitions/0/state
get /brokers/topics/cloudeasy.prod.mess/partitions/0/state
get /brokers/topics/cloudeasy.user.event/partitions/0/state
get /brokers/topics/cmsp.broker.account/partitions/0/state
get /brokers/topics/cmsp.broker.group/partitions/0/state
get /brokers/topics/cmsp.broker.user/partitions/0/state

 

set /brokers/topics/cloudeasy.billing.ascription/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.billing.executor/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.jflow.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.optimization.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.budset/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.prod.mess/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.user.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.account/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.group/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM