celery的安裝和使用

 celery是python開發的分布式任務調度模塊，接口簡單，開發容易，五分鍾就寫出一個異步發送郵件的服務，celery本身不含消息服務，它使用第三方消息服務來傳遞任務，目前，celery支持的消息服務有RabbitMQ,redis甚至是數據庫，redis是最佳選擇

 

Windows使用celery只能安裝 3.1.25版

pip install celery==3.1.25  

 

編寫tasks.py

# tasks.py
import time
from celery import Celery

celery = Celery('tasks', broker='redis://localhost:6379/0')

@celery.task
def sendmail(mail):
    print('sending mail to %s...' % mail['to'])
    time.sleep(2.0)
    print('mail sent.')

 

啟動celery處理任務

celery -A tasks worker --loglevel=info

tasks是任務文件名，worker任務角色，--loglevel=info 任務日志級別

上面的命令行實際上啟動的是worker，如果要放到后台運行，可以扔給supervisor

 

要在目錄下啟動

D:\py3code\jintong_day1\ce>celery -A tasks worker --loglevel=info
[2018-05-30 19:37:15,815: WARNING/MainProcess] d:\py3.6\lib\site-packages\celery
\apps\worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

 -------------- celery@E73-PC v3.1.25 (Cipater)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x395deb8
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . tasks.sendmail

[2018-05-30 19:37:15,926: INFO/MainProcess] Connected to redis://localhost:6379/
0
[2018-05-30 19:37:15,995: INFO/MainProcess] mingle: searching for neighbors
[2018-05-30 19:37:17,160: INFO/MainProcess] mingle: all alone
[2018-05-30 19:37:17,183: WARNING/MainProcess] celery@E73-PC ready.

 

 

怎么發送任務

在當前目錄下打開命令行，進入python

>>> from tasks import sendmail
>>> sendmail.delay(dict(to='[e-mail]'))
<AsyncResult: 1a0a9262-7858-4192-9981-b7bf0ea7483b>

在worker里就可以看到任務處理的消息

[2018-05-30 19:36:13,517: INFO/MainProcess] Received task: tasks.sendmail[815178
90-2406-4756-a4b5-d650ea8cd2e2]
[2018-05-30 19:36:13,517: WARNING/Worker-1] sending mail to celery@hhh
[2018-05-30 19:36:15,524: WARNING/Worker-1] mail sent
[2018-05-30 19:36:15,524: INFO/MainProcess] Task tasks.sendmail[81517890-2406-47
56-a4b5-d650ea8cd2e2] succeeded in 1.9970000000030268s: None

 celery默認設置就能滿足基本要求，worker以pool模式啟動，默認大小為CPU核心數量，缺省化機制為pickle,但可以指定為json,由於python調用UNIX/Linux太容易，所以，用celery作為異步任務框架非常合適

celery還有一些高級用法，比如把多個任務組合成一個原子任務，還有一個完善的監控接口