airflow分布式部署(一)Python環境


一、Python環境安裝

airflow依賴Python3,所以安裝之前需要保證服務器有Python環境

#安裝Python3
#安裝依賴環境
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
#下載安裝包
wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
#創建python3目錄
mkdir -p /usr/local/python3
#解壓縮
tar -zxvf Python-3.6.8.tgz
#編譯安裝,需要c環境
yum install gcc
cd Python-3.6.8
./configure --prefix=/usr/local/python3
make
make install
#建立python3的軟鏈
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
#將/usr/local/python3/bin加入PATH
...
PATH=$PATH:$HOME/bin:/usr/local/python3/bin
export PATH
...
source ~/.bash_profile
#檢查是否生效
python3 -V
pip3 -V

 

 

#systemctl自啟動報錯解決
#Q:No such file or directory: 'airflow': 'airflow'
#A:將python3的路徑加入到PATH:which python3

export PATH=$PATH:/root/miniconda3/envs/py36/bin/
source /etc/profile

二、依賴包安裝

airflow依賴大量第三方包,如果都一一安裝較為困難,本次采用安裝miniconda包,可以達到事半功倍的效果

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash Miniconda3-latest-Linux-x86_64.sh

source ~/.bashrc

conda create -n py36 python=3.6 anaconda

conda activate py36

安裝完miniconda之后,就可以安裝airflow的包了

pip install 'apache-airflow[celery]'
pip install 'apache-airflow[redis]'
pip install 'apache-airflow[mysql]'

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM