轉載請注明地址http://www.cnblogs.com/dongxiao-yang/p/5217754.html
Starting in 0.9, the Kafka cluster has the ability to enforce quotas on produce and fetch requests. Quotas are basically byte-rate thresholds defined per client-id. A client-id logically identifies an application making a request. Hence a single client-id can span multiple producer and consumer instances and the quota will apply for all of them as a single entity i.e. if client-id="test-client" has a produce quota of 10MB/sec, this is shared across all instances with that same id.
從0.9版本開始，kafka集群新增了針對生產和消費請求進行配額（quotas）控制的能力。Quota基本上是一個單client-id的數據byte速率的門檻值的概念。邏輯上一個client-id代表了一個產生請求的應用程序。一個client-id理論上可以擁有多個producer或者consumer實例,qouta會把這些實例當做一整個個體來對待。假如一個client-id為“test-client”的程序quota設置為10MB/s，擁有相同id的實例會共享這個配額。
It is possible for producers and consumers to produce/consume very high volumes of data and thus monopolize broker resources, cause network saturation and generally DOS other clients and the brokers themselves. Having quotas protects against these issues and is all tbe more important in large multi-tenant clusters where a small set of badly behaved clients can degrade user experience for the well behaved ones. In fact, when running Kafka as a service this even makes it possible to enforce API limits according to an agreed upon contract.
生產/消費者有可能會產生非常高的數據吞吐並因此搶占了broker服務器的資源，造成網絡飽和並且通常會DoS掉其他client的連接和broker服務器本身。加入quota機制可以預防上述情況的發生，更可以避免集群在多用戶的場景下由於個別客戶端的異常流量影響其余正常客戶端的使用（就是一顆老鼠屎壞了一鍋湯的意思）。其實，由於kafka本來就是作為一個后台服務運行的，通過api接口強制進行限制約定是很可行的辦法。
By default, each unique client-id receives a fixed quota in bytes/sec as configured by the cluster (quota.producer.default, quota.consumer.default). This quota is defined on a per-broker basis. Each client can publish/fetch a maximum of X bytes/sec per broker before it gets throttled. We decided that defining these quotas per broker is much better than having a fixed cluster wide bandwidth per client because that would require a mechanism to share client quota usage among all the brokers. This can be harder to get right than the quota implementation itself!
默認情況下，每一個單獨的client-id對應一份集群配置的固定quta速度（默認配置在quota.producer.default, quota.consumer.default）。quota是一個被定義到每台broker粒度的概念。每個client在達到限速前可以與單台broker產生最大為X bytes/sec的寫/讀流量請求。決定將quota定義到每台broker粒度比設置一個固定的全集群粒度的帶寬概念更合適，這樣可以省去一個在集群broker間協調quota的機制。這個協調機制可能比quota機制本身的實現更為麻煩！
How does a broker react when it detects a quota violation? In our solution, the broker does not return an error rather it attempts to slow down a client exceeding its quota. It computes the amount of delay needed to bring a guilty client under it's quota and delays the response for that time. This approach keeps the quota violation transparent to clients (outside of client side metrics). This also keeps them from having to implement any special backoff and retry behavior which can get tricky. In fact, bad client behavior (retry without backoff) can exacerbate the very problem quotas are trying to solve.
broker在發現超出quota的情況下會如何處理？我們目前的處理方法是，broker並不會返回錯誤信息而是會嘗試降低客戶端的速度。broker計算出將客戶端速度限制在quota以下需要的delay時間然后在response時先delay這么多時間再響應。這種機制基本實現將quota限速功能對客戶端透明化（無需客戶端一側的配置），同時也避免了客戶端需要實現的復雜麻煩的backoff和retry的邏輯。事實上，異常的客戶端行為（沒有回退機制的重試）可能將quota想要解決的問題更加惡化。
Client byte rate is measured over multiple small windows (for e.g. 30 windows of 1 second each) in order to detect and correct quota violations quickly. Typically, having large measurement windows (for e.g. 10 windows of 30 seconds each) leads to large bursts of traffic followed by long delays which is not great in terms of user experience.
客戶端的byte速率是通過多個小的窗口抽樣（比如說每秒抽樣30次）來准確並迅速的發現quota超限的情況。通常來說，長時間的抽樣窗口（比如30秒抽樣10次）會由於延遲較大可能會發生流量擁堵的狀況，這會影響用戶體驗。
Quota overrides（quota 重設）
It is possible to override the default quota for client-ids that need a higher (or even lower) quota. The mechanism is similar to the per-topic log config overrides. Client-id overrides are written to ZooKeeper under/config/clients. These overrides are read by all brokers and are effective immediately. This lets us change quotas without having to do a rolling restart of the entire cluster. See here for details.
可以修改某個客戶端默認的quota，方法與每個topic的config重設類似。客戶端的設置位於zookeeper服務/config/clients路徑下。這個改動會被所有broker讀取並且立即生效。
這可以使我們無需滾動重啟整個集群就可以改變quota。參考如下連接 here。