airflow worker 出現 (2006, 'MySQL server has gone away')以及raise AirflowException('Celery command failed')airflow 設置分布式部署主從worker不能協調運行


OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away')
[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status

重啟能解決上述問題

[2020-04-20 17:25:40,165: ERROR/ForkPoolWorker-1] None
[2020-04-20 17:25:40,172: ERROR/ForkPoolWorker-1] Task airflow.executors.celery_executor.execute_command[5253e28b-4219-42bc-81ca-2b8951a7123e] raised unexpected: AirflowException('Celery command failed',)
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python2.7/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/ubuntu/.local/lib/python2.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 72, in execute_command
raise AirflowException('Celery command failed')
AirflowException: Celery command failed

 

上述問題,未找到對應的執行程序,認真檢查執行程序是否在所在目錄;另外確保slave worker能執行master程序請設置如下操作:

sudo airflow worker -cn=celery@hosttname -p -D

 

master schduler -p 

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM