hbase集群region數量和大小的影響

1、Region數量的影響

通常較少的region數量可使群集運行的更加平穩，官方指出每個RegionServer大約100個regions的時候效果最好，理由如下：

1）Hbase的一個特性MSLAB，它有助於防止堆內存的碎片化，減輕垃圾回收Full GC的問題，默認是開啟的。但是每個MemStore需要2MB（一個列簇對應一個寫緩存memstore）。所以如果每個region有2個family列簇，總有1000個region，就算不存儲數據也要3.95G內存空間.

2）如果很多region，它們中Memstore也過多，內存大小觸發Region Server級別限制導致flush，就會對用戶請求產生較大的影響，可能阻塞該Region Server上的更新操作。

3）HMaster要花大量的時間來分配和移動Region，且過多Region會增加ZooKeeper的負擔。

4）從hbase讀入數據進行處理的mapreduce程序，過多Region會產生太多Map任務數量，默認情況下一個region對應一個map

所以，如果一個HRegion中Memstore過多，而且大部分都頻繁寫入數據，每次flush的開銷必然會很大，因此也建議在進行表設計的時候盡量減少ColumnFamily的個數。

 

計算集群region數量的公式：

((RS Xmx) * hbase.regionserver.global.memstore.size) / (hbase.hregion.memstore.flush.size * (# column families))

假設一個RS有16GB內存，那么16384*0.4/128m 等於51個活躍的region。

如果寫很重的場景下，可以適當調高hbase.regionserver.global.memstore.size，這樣可以容納更多的region數量。

總結，建議分配合理的region數量，根據寫請求量的情況，一般20-200個之間，可以提高集群穩定性，排除很多不確定的因素，提升讀寫性能。監控Region Server中所有Memstore的大小總和是否達到了上限（hbase.regionserver.global.memstore.upperLimit ＊ hbase_heapsize，默認 40%的JVM內存使用量），超過可能會導致不良后果，如服務器反應遲鈍或compact風暴。

 

2、region大小的影響

hbase中數據一開始會寫入memstore，滿128MB（看配置）以后，會flush到disk上而成為storefile。當storefile數量超過觸發因子時（可以配置），會啟動compaction過程將它們合並為一個storefile。對集群的性能有一定影響。而當合並后的storefile大於max.filesize，會觸發分割動作，將它切分成兩個region。

1）當hbase.hregion.max.filesize比較小時，觸發split的機率更大，系統的整體訪問服務會出現不穩定現象。

2）當hbase.hregion.max.filesize比較大時，由於長期得不到split，因此同一個region內發生多次compaction的機會增加了。這樣會降低系統的性能、穩定性，因此平均吞吐量會受到一些影響而下降。

總結，hbase.hregion.max.filesize不宜過大或過小，經過實戰，生產高並發運行下，最佳大小5-10GB！關閉某些重要場景的hbase表的major_compact！在非高峰期的時候再去調用major_compact，這樣可以減少split的同時，顯著提供集群的性能，吞吐量、非常有用。

 

---

169.2.2. Number of regions per RS - upper bound

In production scenarios, where you have a lot of data, you are normally concerned with the maximum number of regions you can have per server. too many regions has technical discussion on the subject. Basically, the maximum number of regions is mostly determined by memstore memory usage. Each region has its own memstores; these grow up to a configurable size; usually in 128-256 MB range, see hbase.hregion.memstore.flush.size. One memstore exists per column family (so there’s only one per region if there’s one CF in the table). The RS dedicates some fraction of total memory to its memstores (see hbase.regionserver.global.memstore.size). If this memory is exceeded (too much memstore usage), it can cause undesirable consequences such as unresponsive server or compaction storms. A good starting point for the number of regions per RS (assuming one table) is:

((RS memory) * (total memstore fraction)) / ((memstore size)*(# column families))

This formula is pseudo-code. Here are two formulas using the actual tunable parameters, first for HBase 0.98+ and second for HBase 0.94.x.

HBase 0.98.x

((RS Xmx) * hbase.regionserver.global.memstore.size) / (hbase.hregion.memstore.flush.size * (# column families))

HBase 0.94.x

((RS Xmx) * hbase.regionserver.global.memstore.upperLimit) / (hbase.hregion.memstore.flush.size * (# column families))+

If a given RegionServer has 16 GB of RAM, with default settings, the formula works out to 16384*0.4/128 ~ 51 regions per RS is a starting point. The formula can be extended to multiple tables; if they all have the same configuration, just use the total number of families.

This number can be adjusted; the formula above assumes all your regions are filled at approximately the same rate. If only a fraction of your regions are going to be actively written to, you can divide the result by that fraction to get a larger region count. Then, even if all regions are written to, all region memstores are not filled evenly, and eventually jitter appears even if they are (due to limited number of concurrent flushes). Thus, one can have as many as 2-3 times more regions than the starting point; however, increased numbers carry increased risk.

For write-heavy workload, memstore fraction can be increased in configuration at the expense of block cache; this will also allow one to have more regions.

169.2.3. Number of regions per RS - lower bound

HBase scales by having regions across many servers. Thus if you have 2 regions for 16GB data, on a 20 node machine your data will be concentrated on just a few machines - nearly the entire cluster will be idle. This really can’t be stressed enough, since a common problem is loading 200MB data into HBase and then wondering why your awesome 10 node cluster isn’t doing anything.

On the other hand, if you have a very large amount of data, you may also want to go for a larger number of regions to avoid having regions that are too large.

169.2.4. Maximum region size

For large tables in production scenarios, maximum region size is mostly limited by compactions - very large compactions, esp. major, can degrade cluster performance. Currently, the recommended maximum region size is 10-20Gb, and 5-10Gb is optimal. For older 0.90.x codebase, the upper-bound of regionsize is about 4Gb, with a default of 256Mb.

The size at which the region is split into two is generally configured via hbase.hregion.max.filesize; for details, see arch.region.splits.

If you cannot estimate the size of your tables well, when starting off, it’s probably best to stick to the default region size, perhaps going smaller for hot tables (or manually split hot regions to spread the load over the cluster), or go with larger region sizes if your cell sizes tend to be largish (100k and up).

In HBase 0.98, experimental stripe compactions feature was added that would allow for larger regions, especially for log data. See ops.stripe.

169.2.5. Total data size per region server

According to above numbers for region size and number of regions per region server, in an optimistic estimate 10 GB x 100 regions per RS will give up to 1TB served per region server, which is in line with some of the reported multi-PB use cases. However, it is important to think about the data vs cache size ratio at the RS level. With 1TB of data per server and 10 GB block cache, only 1% of the data will be cached, which may barely cover all block indices.