Centos7 安裝部署 Airflow

本人在centos7 的環境下部署，怎么在centos7 下配置靜態 IP 關閉防火牆 以及安裝jdk在這里不多贅述，

centos7 配置靜態ip可以參考:https://www.cnblogs.com/braveym/p/8523100.html  和  https://www.cnblogs.com/braveym/p/9096402.html

 

Airflow 基礎安裝

1、默認自帶python2環境，自行安裝pip

sudo yum -y install epel-release
sudo yum -y install python-pip

 

 

 

 

 

 

 

 

 

2、進行pip的更新，否則很多安裝會報錯

sudo pip install --upgrade pip
sudo pip install --upgrade setuptools

 

 

 

 

 

 

3、安裝開發庫

sudo yum install python-devel
sudo yum install libevent-devel
sudo yum install mysql-devel

 

 

 

 

 

 

 

 

4、安裝centos7下的mysql

在安裝mysql前建議先更新一下源，不然下載老失敗

 

打開centos的yum文件夾（需要root權限或者擁有該目錄的操作權限才可以）

輸入命令cd  /etc/yum.repos.d

 

用wget下載repo文件

輸入命令sudo wget  http://mirrors.aliyun.com/repo/Centos-7.repo（如果wget命令不生效，說明還沒有安裝wget工具，輸入sudo yum -y install wget 回車進行安裝）

 

 備份系統原來的repo文件

sudo mv  CentOS-Base.repo CentOS-Base.repo.bak

 

 

 

替換系統原理的repo文件

sudo wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo

 

 

 在更新源之前先把自帶的mariadb刪除，會報類似這樣的錯誤

 

 

 

 

[bigdata@bigdata ~]$ rpm -qa|grep mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
[bigdata@bigdata ~]$ rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)
[bigdata@bigdata ~]$ sudo rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
[bigdata@bigdata ~]$ rpm -qa|grep mariadb
[bigdata@bigdata ~]$ 

 

 

 

執行yum源更新命令

sudo yum clean all

sudo yum makecache

sudo yum -y update （需要比較長的時間）

 

 

 

 在https://dev.mysql.com/downloads/mysql/下載mysql的離線安裝包

 

 

 

 下載完之后把包上傳到集群

 

 解壓

 

 把以上解壓包每一個都安裝一遍，但是，安裝的時候建議安裝順序來，不要隨便拿起一個就安裝，不然會報這樣的錯誤

 

 所以建議按照下面的順序進行安裝

bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-common-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-5.7.27-1.el7################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-compat-5.7.2################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-embedded-compat-5################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-devel-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-devel-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-devel-5.7.27-1.el################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-client-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-server-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$

 

先關閉mysql服務

 

 

修改MySql配置

skip-grant-tables #添加這句話，這時候登入mysql就不需要密碼

 

 

啟動mysql

[bigdata@bigdata mysql]$ sudo systemctl start mysqld.service
[bigdata@bigdata mysql]$ sudo systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-20 11:49:49 CST; 3s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 4759 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 4679 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 4762 (mysqld)
    Tasks: 27
   CGroup: /system.slice/mysqld.service
           └─4762 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Aug 20 11:49:45 bigdata systemd[1]: Starting MySQL Server...
Aug 20 11:49:49 bigdata systemd[1]: Started MySQL Server.

 

 

登錄mysql，輸入密碼處直接回車就可以

 

 

5、初始化配置mysql，可以設定允許root遠程登錄

先通過root登錄mysql,密碼是剛剛給root用戶設置的密碼我這里是：Admin123!

 

 Mysql默認不允許遠程登錄，我們需要設置下，並且防火牆開放3306端口；(備注 mysql5.7默認密碼策略要求密碼必須是大小寫字母數字特殊字母的組合，至少8位)

 

mysql> set password for root@localhost = password('Admin123!');
ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement
mysql> flush privileges; 
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@localhost = password('Admin123!');
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> create user 'airflow'@'localhost' identified by 'Airflow123!';
Query OK, 0 rows affected (0.05 sec)

mysql> grant all on airflow.* to 'airflow'@'%';
Query OK, 0 rows affected (0.02 sec)

mysql>  flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> quit;
Bye

 

 

 修改配置文件

 

設置MySql開機自啟

sudo systemctl enable mysqld

 

 

對於防火牆開啟的情況下，就需要開發3306端口，我這里是把防火牆關閉

[bigdata@bigdata mysql]$ sudo firewall-cmd --zone=public --add-port=3306/tcp --permanent
FirewallD is not running
[bigdata@bigdata mysql]$ sudo firewall-cmd --reload
FirewallD is not running

 

 

6、配置Mysql默認編碼為UTF-8

 修改/etc/my.cnf配置文件，在[mysqld]下添加編碼配置

 

 

character_set_server=utf8
init_connect='SET NAMES utf8'

 

 編輯保存完 重啟mysql服務: sudo systemctl restart mysqld.service

 

 

 查看下編碼：

 

[bigdata@bigdata mysql]$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.27 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%character%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

mysql> 

 

 

 

7、安裝AirFlow

給airflow設置環境變量

sudo vim /etc/profile

#airflow
export AIRFLOW_HOME=/opt/modules/airflow
export SLUGIFY_USES_TEXT_UNIDECODE=yes

 

是環境變量生效

 

 安裝airflow

sudo pip install apache-airflow

 

 

可以看到出現以下錯誤

 

 這樣解決：

sudo pip install six --user -U
sudo pip install ipython --user -U
sudo pip install --ignore-installed dnspython

  sudo yum install python-devel

  sudo yum install openldap-devel

  sudo pip install --ignore-installed python-ldap

  sudo pip install --ignore-installed enum34

 

 

前面的問題解決了，又開始新的問題

 

 這個問題這樣解決：

sudo pip install --ignore-installed requests

 

 

最后再重新安裝一次airflow,可以看到成功了

 

 

8、 設置mysql數據庫連接

sudo pip install apache-airflow[mysql]

 

 

 創建airflow用戶,創建airflow數據庫並給出所有權限給此用戶（注意在設置密碼的時候還是前面講的問題，不能太簡單，否則不能通過）

mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'airflow'@'%' identified by 'AirFlow123!';
Query OK, 0 rows affected (0.02 sec)

mysql> GRANT all privileges on airflow.* TO 'airflow'@'%'  IDENTIFIED BY 'AirFlow123!';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

 

 

修改airflow配置文件，指向數據庫mysql

airflow.cfg 文件通常在~/airflow目錄下

 

 

 

 

 

executor =  LocalExecutor
sql_alchemy_conn = mysql://airflow:AirFlow123!@192.168.189.11/airflow
對應字段解釋如下： dialect+driver://username:password@host:port/database

 

 修改mysql的配置文件 

 vim /etc/my.cnf

 添加這一句

explicit_defaults_for_timestamp=1

 

 

 重啟mysql

 

 

 9、初始化airflow並啟動web服務

sudo airflow initdb
sudo airflow webserver
sudo airflow scheduler
sudo airflow worker 

 

 

 

 

 在瀏覽器打開地址 192.168.189.11:8080