airflow 安裝配置celery+rabbitmq celery+redis

AirFlow的安裝可以參考：https://www.cnblogs.com/braveym/p/11378851.html

這里介紹的是AirFlow 安裝配置celery+rabbitmq   和   celery+redis

 

一、airflow 安裝配置celery+rabbitmq

 

1、安裝celery和rabbitmq組件

sudo pip install apache-airflow[celery]

sudo pip install apache-airflow[rabbitmq]

 

 

2、系統安裝rabbit

安裝 erlang

sudo yum install erlang

 

 

安裝 rabbitmq

sudo wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-3.6.15-1.el6.noarch.rpm
sudo rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
sudo yum install rabbitmq-server-3.6.15-1.el6.noarch.rpm

 

 

配置rabbitmq（設置用戶名root，密碼rabbitmq，創建虛擬主機airflow-rabbitmq）

[bigdata@bigdata airflow]$ su
Password: 
[root@bigdata airflow]# rabbitmqctl add_user root rabbitmq
Creating user "root"
[root@bigdata airflow]# rabbitmqctl add_vhost airflow-rabbitmq
Creating vhost "airflow-rabbitmq"
[root@bigdata airflow]# rabbitmqctl set_user_tags root airflow-rabbitmq
Setting tags for user "root" to ['airflow-rabbitmq']
[root@bigdata airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"
Setting permissions for user "root" in vhost "airflow-rabbitmq"
[root@bigdata airflow]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  amqp_client
  cowlib
  cowboy
  rabbitmq_web_dispatch
  rabbitmq_management_agent
  rabbitmq_management

Applying plugin configuration to rabbit@bigdata... started 6 plugins.
[root@bigdata airflow]# 

 

 

 

修改airflow配置

executor = CeleryExecutor

broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

result_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

 

 

 

 

 

 在啟動worker 時，提示不能用root用戶權限啟動。修改方法：
    echo  "export C_FORCE_ROOT="True"" >> /etc/profile
    source /etc/profile

 

二、airflow 安裝配置celery+redis

 

安裝redis

sudo yum -y install redis

 

 

修改配置文件

 

sudo vim /etc/redis.conf
#bind 127.0.0.1     // 注釋掉，使redis允許遠程訪問
requirepass 密碼  // 修改這行，redis登錄密碼

 

 

 

 

 

開啟redis服務

sudo systemctl start redis

 

 

 

python下安裝redis庫

sudo pip install redis

 

 

安裝celery

sudo pip install celery

 

 

 

配置airflow.cfg

executor = CeleryExecutor
broker_url = redis://127.0.0.1:6379/0
result_backend = redis://127.0.0.1:6379/0