Kafka相關問題:
Marking
the coordinator 2147483283 dead.
Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test Auto offset commit failed: Commit cannot be completed due to group rebalance
Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test Auto offset commit failed:
Attempt to join group test failed due to unknown member id, resetting and retrying.
Attempt to heart beat failed since the group is rebalancing, try to re-join group.we
問題描述:kafka消費信息時,產生重復消費的情況。
問題原因:
大概意思是,在kafka超時時間內,有消息沒有處理完成,consumer coordinator會因為沒有收到心跳而掛掉,然后自動提交offset失敗,然后重新分配partition給客戶端。導致以下問題:
1、由於自動提交offset失敗,導致重新分配了partition的客戶端又重新消費之前的一批數據;
2、接着consumer重新消費,又出現了消費超時,無限循環下去。
消費者消費不過來,數據信息量大,自動commit超時。
設置:max.partition.fetch.bytes
:一次拉取最大byte這個屬性 小一點.默認1M;
或者設置session.timeout.ms
:超時時間設置大一點,在不設置的情況下,kafka默認自動5000ms提交
Ø 用自己的話說,就是kafka中存儲了大量數據消費不過來,此時又到了心跳監聽階段,無發送心跳,導致異常。所以要盡量減少數據量,使到監聽階段能夠消費完隊列中的數據。
資料網站:
https://blog.csdn.net/gpdsjqws/article/details/81027415
【kafka : Error
UNKNOWN_MEMBER_ID occurred while committing offsets for group alert】https://blog.csdn.net/jinzhencs/article/details/71713707
【kafka 0.9.0.0 部分配置詳解】https://blog.csdn.net/u012191627/article/details/53338503
【apache kafka 遇到 Attempt to heart beat failed since the group is rebalancing, try to
re-join group. 該如何解決?】https://www.oschina.net/question/2417189_2264063?no_redirect=true
【kafka 消費能力不夠?】https://bbs.csdn.net/topics/392512899