kafka 日志結構

1.kafka日志結構

直接舉例子：

例如kafka有個名字叫 haha 的topic，那么kafka日志下面有kafka-0,kafka-1,kafka-2...,kafka-n，具體多少個，創建分區的時候有幾個分區，n就是幾。

[root@haha kafka]# ls haha-
haha-0/ haha-1/ haha-2/

每個目錄下面有三個日志文件，xxx.log是消息集文件， xxx.index 偏移量索引文件 ，xxx.timeindex 時間戳索引文件

[root@haha kafka]# ls haha-1
00000000000000000000.index  00000000000000000000.log  00000000000000000000.timeindex

 

創建主題：

[root@haha kafka]# kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic log-format

 

[root@haha kafka]# ls /data/appData/kafka/log-format-
log-format-0/ log-format-1/ log-format-2/

[root@haha kafka]# ls /data/appData/kafka/log-format-0
00000000000000000000.index  00000000000000000000.log  00000000000000000000.timeindex

 

修改字段大小及索引跨度配置

[root@haha kafka]# kafka-topics.sh --zookeeper localhost:2181 --alter --topic log-format --config segment.bytes=1024 --config index.interval.bytes=100
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
         Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "log-format".

 

查看分topic區狀態

[root@haha kafka]# /data/app/zookeeper/bin/zkCli.sh -server localhost:2181 

[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids

[zk: localhost:2181(CONNECTED) 3] get /brokers/topics/log-format/partitions/1/state
cZxid = 0x245
ctime = Thu Nov 01 11:29:47 CST 2018
mZxid = 0x245
mtime = Thu Nov 01 11:29:47 CST 2018
pZxid = 0x245
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 72
numChildren = 0

 

如果集群的日志則分布如下

 

 

 

通過kafka自帶的腳本查看topic信息

[root@haha kafka]# kafka-topics.sh --describe --zookeeper localhost:2181 --topic log-format
Topic:log-format	PartitionCount:3	ReplicationFactor:1	Configs:index.interval.bytes=100,segment.bytes=1024
	Topic: log-format	Partition: 0	Leader: 5	Replicas: 5	Isr: 5
	Topic: log-format	Partition: 1	Leader: 5	Replicas: 5	Isr: 5
	Topic: log-format	Partition: 2	Leader: 5	Replicas: 5	Isr: 5

 

執行一下命令將二進制分段日志文件轉化為字符類型的文件

[root@haha kafka]# kafka-run-class.sh kafka.tools.DumpLogSegments --files log-format-0/00000000000000000000.log --print-data-log
Dumping log-format-0/00000000000000000000.log
Starting offset: 0