如何徹底刪除Kafka中的topic (marked for deletion)


工作中因為各種原因, 例如topic中消息堆積的太多,或者kafka所在磁盤空間滿了等等,可能需要徹底清理一下kafka topic,那么如何徹底刪除topic?
方法一(配置delete.topic.enable=true)
  修改kafaka配置文件server.properties, 添加delete.topic.enable=true,重啟kafka,之后通過kafka命令行就可以直接刪除topic
  通過命令行刪除topic:

    ./bin/kafka-topics.sh --delete --zookeeper {zookeeper server} --topic {topic name}

方法二(沒有配置delete.topic.enable=true)

1、通過命令行刪除topic:

./bin/kafka-topics.sh --delete --zookeeper {zookeeper server} --topic {topic name}

  因為kafaka配置文件中server.properties沒有配置delete.topic.enable=true,此時的刪除並不是真正的刪除,只是把topic標記為:marked for deletion
  你可以通過命令:./bin/kafka-topics --zookeeper {zookeeper server} --list 來查看所有topic
2、刪除kafka存儲目錄(server.properties文件log.dirs配置,默認為"/tmp/kafka-logs")相關topic目錄


3, 若想真正刪除它,需要登錄zookeeper客戶端:

  命令:./bin/zkCli.sh

  找到topic所在的目錄:ls /brokers/topics

   執行命令:rmr /brokers/topics/{topic name}即可,此時topic被徹底刪除。

  另外被標記為marked for deletion的topic你可以在zookeeper客戶端中通過命令獲得:ls /admin/delete_topics/{topic name},如果你刪除了此處的topic,那么marked for deletion 標記消失

zookeeper 的config中也有有關topic的信息: ls /config/topics/{topic name}暫時不知道有什么用

總結

徹底刪除topic:
1、確保kafka的配置文件server.proeprties中設置delete.topic.enable=true,如果沒有, 確保cluster的所有kafka配置文件設置該參數並重啟,然后直接通過命令刪除,如果命令刪除不了,直接通過zookeeper命令行刪除掉broker下的topic。

2、刪除kafka存儲目錄(server.properties文件log.dirs配置,默認為"/tmp/kafka-logs")相關topic目錄

參考文檔:
1, https://stackoverflow.com/questions/17730905/is-there-a-way-to-delete-all-the-data-from-a-topic-or-delete-the-topic-before-ev

stop zookeeper & Kafka server,
then go to ‘kafka-logs’ folder , there you will see list of kafka topic folders, delete folder with topic name
go to ‘zookeeper-data’ folder , delete data inside that.
start zookeeper & kafka server again.
2, https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Isitpossibletodeleteatopic?
Deleting a topic is supported since 0.8.2.x. You will need to enable topic deletion (setting delete.topic.enable to true) on all brokers first.

3, https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default

4, https://github.com/darrenfu/bigdata/issues/6

*** The only way to delete a topic permanently is as follows: ***

stop the brokers sudo supervisorctl stop kafka-broker remove the directories on disk sudo rm -rf <kafka_data_dir>/<topic_name>* remove the topic from zookeeper: bin/zkCli.sh 


rmr /config/topics/<topic_name>
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
 

verify the topic directory is deleted in zookeeper:
start the brokers back up sudo supervisorctl start kafka-broker re-create the kafka topic if needed 

轉自https://blog.csdn.net/russle/article/details/82881297


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM