kafka關於修改副本數和分區的數的案例實戰（也可用作leader節點均衡案例）

　　　　　　　　　　　　　　kafka關於修改副本數和分區的數的案例實戰（也可用作leader節點均衡案例）

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正傑

版權聲明：原創作品，謝絕轉載！否則將追究法律責任。

 

 

 

一.關於topic分區數的修改

1>.創建1分區1個的topic，名稱為yinzhengjie-channel

[root@node101 ~]# kafka-topics.sh --zookeeper node102.yinzhengjie.org.cn:2181 --create --replication-factor 1 -partitions 1 --topic yinzhengjie-channel
Created topic "yinzhengjie-channel".
[root@node101 ~]# 

2>.查看topic的信息

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103                              #可以很明顯的看出kafka 的分區數和副本數都是1
[root@node101 ~]# 

3>.將之前創建的topic修改為3個分區

[root@node101 ~]# kafka-topics.sh --alter --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@node101 ~]# 

4>.再次查看topic的分區數

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel                   
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103                                    #這是第一個分區，它的副本數依然是1一個，當前0號分區的副本數存放在103這個節點上。說明你的數據修改成功啦！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 

 

二.關於topic副本數的修改

1>.編寫分配腳本

[root@node101 ~]# cat addReplicas.json 
{"topics":
    [{"topic":"yinzhengjie-channel"}],
    "version": 1
}
[root@node101 ~]# 

2>.執行分配計划，用於生成json格式的文件

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --topics-to-move-json-file addReplicas.json --broker-list "101,102,103" --generate
Current partition replica assignment            　　　　　　#這是當前的分區情況，你可以結合--describe參數查看當前的分區情況
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[101]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[102]}]}

Proposed partition reassignment configuration     　　　　#這是推薦分區計划
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[102]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[101]}]}
[root@node101 ~]# 
[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel　　　　　　#查看當前分區的情況
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 

3>.Proposed partition reassignment configuration 后是根據命令行的指定的brokerlist生成的分區分配計划json格式。將 Proposed partition reassignment configuration的配置copy保存到一個文件中 topic-reassignment.json並對它進行相應的修改

[root@node101 ~]# cat topic-reassignment.json　　　　#注意，我在復制下來之后，對副本數進行了修改，由之前的1個副本升級為2個副本。
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[102,101]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[102,103]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[101,103]}]}
[root@node101 ~]# 

4>.根據上一步生成的分配計划配置json文件topic-reassignment.json，進行topic的重新分配。

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --reassignment-json-file topic-reassignment.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[101]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[102]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@node101 ~]# 

5>.查看分配的進度

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --reassignment-json-file topic-reassignment.json --verify
Status of partition reassignment: 
Reassignment of partition [yinzhengjie-channel,0] completed successfully　　　　　　#如果這里的參數是：is still in progress，說明正在進行分配，如果看到當前的提示說明分配完成。
Reassignment of partition [yinzhengjie-channel,1] completed successfully
Reassignment of partition [yinzhengjie-channel,2] completed successfully
[root@node101 ~]# 

　　　　　溫馨提示，上述的方法不僅僅可以用來修改副本數，還可以用來修改你的leader節點，下圖我就是我在生產環境中用來均衡leader節點的實操截圖：（是不是上面我提到的2種狀態都有呢？）

6>.如果分配完成，我們再次查看

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel　　　　　　#查看當前分區的情況，這是還沒有重新分配的時候
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103　　　　　　　　#這里的副本數只有一個！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 
root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:2    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 102    Replicas: 102,101    Isr: 102,101　　　　　　#副本數編程了2個！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 102    Replicas: 102,103    Isr: 103,102
    Topic: yinzhengjie-channel    Partition: 2    Leader: 101    Replicas: 101,103    Isr: 103,101
[root@node101 ~]#