Kafka之 vm.max_map_count


Maximum number of memory map areas a process may have (aka vm.max_map_count).
進程可以具有的最大內存映射區域數(也稱為vm.max_map_count)。

See the Linux kernel documentation.
You should keep an eye at this OS-level property when considering the maximum number of partitions a broker may have.
在考慮broker可能具有的最大分區數時,您應該關注此OS級屬性

By default, on a number of Linux systems, the value of vm.max_map_count is somewhere around 65535.
Linux 系統 vm.max_map_count 的默認值是 65535

Each log segment, allocated per partition, requires a pair of index/timeindex files, and each of these files consumes 1 map area.
每個分區分配的每個日志段需要一對index / timeindex文件,每個文件占用1個映射區域。

In other words, each log segment uses 2 map areas.
換句話說,每個日志段使用2個映射區域

Thus, each partition requires minimum 2 map areas, as long as it hosts a single log segment.
因此,每個分區至少需要2個映射區域,只要它承載單個日志段即可。

That is to say, creating 50000 partitions on a broker will result allocation of 100000 map areas
and likely cause broker crash with OutOfMemoryError (Map failed) on a system with default vm.max_map_count.
舉個例子,如果你創建 50000 分區在一個 broker,將會創建 100000 個map areas 映射取;
在具有默認vm.max_map_count的系統上可能導致代理崩潰並出現OutOfMemoryError(映射失敗)

Keep in mind that the number of log segments per partition varies depending on the segment size,
load intensity, retention policy and, generally, tends to be more than one.
請記住,每個分區的日志段數取決於段大小,負載強度,保留策略,通常往往不止一個

 

對於這個問題我是這么考慮的,Kafka使用page cache技術,也就是用戶態內存映射內核態內存,這樣可以讓刷盤快速。前提就是這種映射關系也是有數量限制的,所以一般都會把這個數值配成大於10萬


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM