0.8
[consumers, admin, config, controller, brokers, controller_epoch]
這些節點都歸於誰管理
- consumers: 0.8版本的客戶端Consumer
- admin:管理命令的腳本:kafka-preferred-replica-election.sh,
- config:管理命令的腳本:kafka-topics.sh
- controller:所有的broker監控這個。
- brokers:kafka-topics.sh建立topic
- controller_epoch:
1.consumers
1.1 ids 代表消費者線程
/consumers/[groupId]/ids/[consumerIdString]:每個consumer都有一個唯一的ID(consumerId可以通過配置文件指定,也可以由系統生成),此id用來標記消費者信息.
[zk: 10.39.1.66:22181(CONNECTED) 17] ls /consumers/guest_tblog_expo/ids
[guest_tblog_expo_yz4834.hadoop.data.sina.com.cn-1472452954131-13232511, guest_tblog_expo_yz4879.hadoop.data.sina.com.cn-1472452955003-3ff3d939]
這兩個節點是臨時的znode。
1.2 Consumer owner 表示partition和consumer線程的對應關系
/consumers/[groupId]/owners/[topic]/[partitionId] -> consumerIdString + threadId 索引編號
[zk: 10.39.1.66:22181(CONNECTED) 23] get /consumers/guest_tblog_expo/owners/openapi_exposure/0
guest_tblog_expo_yz4845.hadoop.data.sina.com.cn-1472452954464-2f40ad23-0
1.3 Consumer offset 表示partition被消費到的Offset位置
用來跟蹤每個consumer目前所消費的partition中最大的offset
此znode為持久節點,可以看出offset跟group_id有關,以表明當消費者組(consumer group)中一個消費者失效,
重新觸發balance,其他consumer可以繼續消費.
/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)
[zk: 10.39.1.66:22181(CONNECTED) 26] get /consumers/guest_tblog_expo/offsets/openapi_exposure/0
50793695368
2 admin 表示Controller管理的命令
常有的目錄結構
/admin/reassign_partitions重新分配partition/admin/preferred_replica_election重新preffed partition leader/admin/delete_topics刪除topic
3 config 配置
主要包括三個的配置
- topic : 存儲topic集群屬性之外的屬性，比如修改了一個新的topic屬性
- clients：Client and Topic configuration overrides: The content of both znodes has the same structure（consumer或者Producer的配置地方）
- changes: Config Change notification (Topic and Client config修改通知的地方)
clients 和 changes 在0.8 版本中也許沒有。
[zk: localhost:2181(CONNECTED) 3] get /kafka10/config/topics/test2
{"version":1,"config":{"retention.ms":"3600000"}}
4 controller
/controller -> int (broker id of the controller) 存儲center controller中央控制器所在kafka broker的信息
- 0.8 版本的存儲內容
[zk: 10.39.1.66:22181(CONNECTED) 34] get /controller
3
- 0.10 版本的存儲內容
[zk: localhost:2181(CONNECTED) 12] get /kafka10/controller
{"version":1,"brokerid":40107,"timestamp":"1479456552014"}
兩個版本的存儲內容有差別了。
6 controller_epoch 表示controller選舉的次數
/controller_epoch -> int (epoch) 此值為一個數字,kafka集群中第一個broker第一次啟動時為1，以后只要集群中center controller中央控制器所在broker變更或掛掉，就會重新選舉新的center controller，每次center controller變更controller_epoch值就會 + 1;
[zk: localhost:2181(CONNECTED) 13] get /kafka10/controller_epoch
9
- 什么時候會重新選舉controller？ 有broker與zk失去聯系的時候。
5 brokers
存集群broker和topic的元信息:
/brokers/topics/[topic]:存儲某個topic的partitions所有分配信息- ``
5.1 topic注冊信息
/brokers/topics/[topic]: 存儲某個topic的partitions所有分配信息/brokers/topics/[topic]/partitions/[0...N]: partition的個數/brokers/topics/[topic]/partitions/[partitionId]/state： partition的信息
[zk: localhost:2181(CONNECTED) 20] get /kafka10/brokers/topics/test2/partitions/0/state
{"controller_epoch":9,"leader":40106,"version":1,"leader_epoch":13,"isr":[40106,40107]}
[zk: 10.39.1.66:22181(CONNECTED) 38] get /brokers/topics/openapi_advertise/partitions/0/state
{ "controller_epoch":33, "isr":[ 13, 7 ], "leader":13, "leader_epoch":29, "version":1 }
5.2 Broker注冊信息
/brokers/ids/[0...N] : 每個broker的配置文件中都需要指定一個數字類型的id(全局不可重復),此節點為臨時znode(EPHEMERAL)
0.10 和 0.8 信息不一樣。
[zk: 10.39.1.66:22181(CONNECTED) 39] get /brokers/ids/48152
{ "host":"yz48152.hadoop.data.sina.com.cn", "jmx_port":9999, "port":19092, "version":1 }
[zk: localhost:2181(CONNECTED) 22] get /kafka10/brokers/ids/4099
{"jmx_port":9999,"timestamp":"1479456552292","endpoints":["PLAINTEXT://10.39.40.99:9110"],"host":"10.39.40.99","version":3,"port":9110}
5.3 seqs 0.10 版本特有
未知？？？？
0.10
[consumers, admin, config, controller, brokers, controller_epoch, isr_change_notification, kafka-acl, kafka-acl-changes]
10.1 isr_change_notification
0.10版本多了3組znode
/isr_change_notification/isr_change_x: Gets created when ISR is changed at any broker, controller watches for these notifications and sends MetadataUpdateRequest to all brokers. (ISR改變的時候，通知Controller，發送更新請求給all brokers.)
10.2 kafka-acl
關於授權，一共有三種：
- Cluster:
Create集群權限（官方寫的是“Create on Cluster”） - Group:
ConsumerGroup權限 - Topic：讀寫權限
[zk: localhost:2181(CONNECTED) 10] get /kafka10/kafka-acl/Cluster/kafka-cluster
{"version":1,"acls":[{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"*"},,{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"10.77.*"}]}
[zk: localhost:2181(CONNECTED) 12] get /kafka10/kafka-acl/Group/superConsumer
{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"}]}
[zk: localhost:2181(CONNECTED) 13] get /kafka10/kafka-acl/Topic/client_ad_realread
{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:fengchao1","permissionType":"Allow","operation":"Describe","host":"*"}]}
10.3 kafka-acl-changes
存儲acl的變更信息。由授權的腳本觸發。
[zk: localhost:2181(CONNECTED) 17] get /kafka10/kafka-acl-changes/acl_changes_0000000053
Topic:test2
關於這些節點里的內容格式，請查看kafka zk wiki