一直報錯:
1 partitions have leader brokers without a matching listener, including [cloudeasy.cmdb.cancel.manage-0]
查看zk服務端,報錯如下:
原因是 watch 太多且路徑太長導致重連的時候超過了 ZK maxBuffer 的限制
把 客戶端從 3.4.6 升級到 3.4.7 就能解決這個問題
先部署zookeeper,默認docker不需要參數,域名訪問地址為zookeeper-dev.cmsp-dev.svc.cluster.local:2181
docker 部署kafka(2.12-2.5.0),啟動參數如下:
sh
-c
exec kafka-server-start.sh /opt/kafka/config/server.properties --override advertised.host.name=zookeeper-dev.cmsp-dev.svc.cluster.local --override broker.id=-1 --override listeners=PLAINTEXT://:9092 --override advertised.listeners=PLAINTEXT://kafka-dev.cmsp-dev.svc.cluster.local:9092 --override zookeeper.connect=zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --list --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --delete --topic cloudeasy.dev.mess --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
如果刪不掉,可以在zookeeper里刪除
deleteall /brokers/topics
開發環境創建topic:
./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.billing.executor --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cloudmanager.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.cancel.manage --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.delete.resource --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.update.project --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.user.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
查詢有哪些客戶端group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
查詢指定group的消費情況
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cloudeasy --describe
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group cloudeasy.billing.consumer
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group cloudeasy.monitor.group
客戶端kafka報錯問題:
2021-10-19 13:01:47.223 [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=cmsp.broker.group] 1 partitions have leader brokers without a matching listener, including [cmsp.broker.group-0]
2021-10-19 11:39:58.625 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=cloudeasy.log.consumer] 1 partitions have leader brokers without a matching listener, including [cloudeasy.user.event-0]
1 partitions have leader brokers without a matching listener,沒有主broker了
./kafka-topics.sh --list --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181 --describe --topic cloudeasy.cmdb.update.project
顯示Leader:-1 選主有問題
解決leeder為-1問題,連接zookeeper
./zkCli.sh -server localhost:2181
get /brokers/topics/cmsp.broker.user/partitions/0/state
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":9,"leader":1001,"version":1,"leader_epoch":5,"isr":[1001]}
get /brokers/topics/cloudeasy.billing.ascription/partitions/0/state
get /brokers/topics/cloudeasy.billing.executor/partitions/0/state
get /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state
get /brokers/topics/cloudeasy.jflow.event/partitions/0/state
get /brokers/topics/cloudeasy.optimization.event/partitions/0/state
get /brokers/topics/cloudeasy.order.budget/partitions/0/state
get /brokers/topics/cloudeasy.order.event/partitions/0/state
get /brokers/topics/cloudeasy.prod.mess/partitions/0/state
get /brokers/topics/cloudeasy.user.event/partitions/0/state
get /brokers/topics/cmsp.broker.account/partitions/0/state
get /brokers/topics/cmsp.broker.group/partitions/0/state
get /brokers/topics/cmsp.broker.user/partitions/0/state
set /brokers/topics/cloudeasy.billing.ascription/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.billing.executor/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.jflow.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.optimization.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.budset/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.prod.mess/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.user.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.account/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.group/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}