MQ隊列堆積太長，消費不過來怎么辦(轉)

轉自：http://windwrite.com/archives/603

我們現有的業務就面臨此問題，消息生產太快，消費不過來，導致隊列堆積很長，把服務器內存耗盡，這時RabbitMQ的處理能力很低下。

我在RabbitMQ社區發郵件問了下：

 

We have the queue publisher and subscriber, the subscriber is going with multi-threads.

The problem is the queue has been increasing too quickly, the consumers can not follow the increment speed, as you see:

root@ubuntu:~# for i in `seq 1 10`;do rabbitmqctl list_queues -p /actionlog;sleep 1;done
Listing queues …
queue_storm_actionlog 12665787
…done.
Listing queues …
queue_storm_actionlog 12670873
…done.
Listing queues …
queue_storm_actionlog 12667975
…done.
Listing queues …
queue_storm_actionlog 12672645
…done.
After one day or about, the queue size can be more than 20 millions, thus RabbitMQ becomes high load and the consuming process is very slow.

收到的答復郵件內容如下：

Increase # of throughput of consumers or reduce publishing rate. If consumers cannot
catch up, hardware upgrades will only offset the moment when systems starts running
low on resources.

If messages can be lost, 3.1 introduces a new feature, queue length limit:
http://www.rabbitmq.com/maxlength.html

Further to the suggestion of imposing a maximum queue length, you could
also consider imposing a maximum message age and instruct the broker to
discard messages that become stale: http://www.rabbitmq.com/ttl.html

總結起來解決方案大體包括：

• 增加消費者的處理能力，或減少發布頻率
• 單純升級硬件不是辦法，只能起到一時的作用
• 考慮使用隊列最大長度限制，RabbitMQ 3.1支持
• 給消息設置年齡，超時就丟棄