Kafka遷移與擴容工具用法

１．遷移topic到新增的node上

假如現在一個kafka集群運行三個broker，broker.id依次為101,102,103,后來由於業務數據突然暴增，需要新增三個broker,broker.id依次為104,105,106.目的是要把chatmessage遷移到新增node上。腳本(json格式)如下所示：

 

kafka-reassign-partitions.sh --zookeeper 192.168.1.10:2181 --topics-to-move-json-file  migration-chatmessage-topic.json  --broker-list  "104,105,106"  --generate

 

migration-chatmessage-topic.json文件內容如下：

 

{
    "topics":
        [
            {
                "topic": "chatmessage"
            }
        ],
    "version":1
}

 

生成分配partitions的json腳本：

{
    "version":1,
    "partitions":
        [
            {
                "topic":"chatmessage",
                "partition":10,"replicas":[8]
            },
            {
                "topic":"chatmessage",
                "partition":5,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":3,
                "replicas":[5]
            },
            {
                "topic":"chatmessage", 
                "partition":4, 
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":9,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":1,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":11,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":7,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":2,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":0,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":6,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":8,
                "replicas":[4]
            }
        ]
    }
}

重新分配parttions的json腳本如下：migration-topic-chatmessage-topic.json

 

{"version":1,
    "partitions":
        [
            {"topic":"cluster-switch-topic","partition":10,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":5,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":4,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":3,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":9,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":1,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":11,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":7,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":2,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":0,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":6,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":8,"replicas":[5]}
        ]
}

 

執行一下腳本

./kafka-reassign-partitions.sh --zookeeper 192.168.1.10:2181  --reassignment-json-file migration-topic-chatmessage-topic.json --execute

 

topic修改(replicats-factor)副本個數

　　假如初始時chatmessage為一個副本，為了提高可用性，需要改為2副本模式。腳本replicas-chatmessage-topic.json文件內容如下:

 

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": 0,
                        "replicas": [101,102,104]
                },
                {
                        "topic": "chatmessage",
                        "partition": 1,
                        "replicas": [102,103,106]
                },
              .......
        "version":1
}

 

編輯好json文件后執行一下命令：

./kafka-reassign-partitions.sh --zookeeper  192.168.1.10:2181 --reassignment-json-file  replicas-chatmessage-topic.json  --execute

3.topic的分區擴容用法

　　先擴容分區數量，腳本如下:

./kafka-topics.sh --zookeeper 192.168.1.10:2181  --alter   --partitions 15   --topic   chatmessage

 

　　把topic為chatmessage的分區數量擴展到１５個。

　　設置topic分區副本：

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": 12,
                        "replicas": [101,102]
                },
                {
                        "topic": "chatmessage",
                        "partition": 13,
                        "replicas": [103,104]
                },
                {
                        "topic": "chatmessage",
                        "partition": 14,
                        "replicas": [105,106]
                }
                ],             
        "version":1
}

 

　　編輯json文件，執行一下腳本：

./bin/kafka-reassign-partitions.sh --zookeeper  192.168.１.10:2181 --reassignment-json-file partitions-extension-chatmessage-topic.json  --execute