一直报错:
1 partitions have leader brokers without a matching listener, including [cloudeasy.cmdb.cancel.manage-0]
查看zk服务端,报错如下:
原因是 watch 太多且路径太长导致重连的时候超过了 ZK maxBuffer 的限制
把 客户端从 3.4.6 升级到 3.4.7 就能解决这个问题
先部署zookeeper,默认docker不需要参数,域名访问地址为zookeeper-dev.cmsp-dev.svc.cluster.local:2181
docker 部署kafka(2.12-2.5.0),启动参数如下:
sh
-c
exec kafka-server-start.sh /opt/kafka/config/server.properties --override advertised.host.name=zookeeper-dev.cmsp-dev.svc.cluster.local --override broker.id=-1 --override listeners=PLAINTEXT://:9092 --override advertised.listeners=PLAINTEXT://kafka-dev.cmsp-dev.svc.cluster.local:9092 --override zookeeper.connect=zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --list --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --delete --topic cloudeasy.dev.mess --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
如果删不掉,可以在zookeeper里删除
deleteall /brokers/topics
开发环境创建topic:
./kafka-topics.sh --create --topic cloudeasy.dev.mess --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.billing.executor --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cloudmanager.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.cancel.manage --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.delete.resource --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.cmdb.update.project --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --create --topic cloudeasy.user.event --replication-factor 1 --partitions 1 --zookeeper zookeeper-dev.cmsp-dev.svc.cluster.local:2181
查询有哪些客户端group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
查询指定group的消费情况
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cloudeasy --describe
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group cloudeasy.billing.consumer
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group cloudeasy.monitor.group
客户端kafka报错问题:
2021-10-19 13:01:47.223 [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=cmsp.broker.group] 1 partitions have leader brokers without a matching listener, including [cmsp.broker.group-0]
2021-10-19 11:39:58.625 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=cloudeasy.log.consumer] 1 partitions have leader brokers without a matching listener, including [cloudeasy.user.event-0]
1 partitions have leader brokers without a matching listener,没有主broker了
./kafka-topics.sh --list --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181
./kafka-topics.sh --zookeeper zookeeper.cmsp-dev.svc.cluster.local:2181 --describe --topic cloudeasy.cmdb.update.project
显示Leader:-1 选主有问题
解决leeder为-1问题,连接zookeeper
./zkCli.sh -server localhost:2181
get /brokers/topics/cmsp.broker.user/partitions/0/state
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":9,"leader":1001,"version":1,"leader_epoch":5,"isr":[1001]}
get /brokers/topics/cloudeasy.billing.ascription/partitions/0/state
get /brokers/topics/cloudeasy.billing.executor/partitions/0/state
get /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state
get /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state
get /brokers/topics/cloudeasy.jflow.event/partitions/0/state
get /brokers/topics/cloudeasy.optimization.event/partitions/0/state
get /brokers/topics/cloudeasy.order.budget/partitions/0/state
get /brokers/topics/cloudeasy.order.event/partitions/0/state
get /brokers/topics/cloudeasy.prod.mess/partitions/0/state
get /brokers/topics/cloudeasy.user.event/partitions/0/state
get /brokers/topics/cmsp.broker.account/partitions/0/state
get /brokers/topics/cmsp.broker.group/partitions/0/state
get /brokers/topics/cmsp.broker.user/partitions/0/state
set /brokers/topics/cloudeasy.billing.ascription/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.billing.executor/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cloudmanager.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.add.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.cancel.manage/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.delete.resource/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.cmdb.update.project/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.jflow.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.optimization.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.budset/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.order.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.prod.mess/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cloudeasy.user.event/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.account/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.group/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}
set /brokers/topics/cmsp.broker.user/partitions/0/state {"controller_epoch":4,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}