在docker容器下安裝airflow

本文轉載自查看原文 2019-10-15 21:13 575 AirFlow

本人的環境是基於centos7下來安裝的

一、安裝docker

下載docker安裝包，下載地址：https://download.docker.com/linux/static/stable/x86_64/

下載到本地后解壓

tar -zxf docker-18.09.6.tgz

將解壓出來的docker文件內容移動到 /usr/bin/ 目錄下

cp docker/* /usr/bin/

將docker注冊為service

新建文件

vim /etc/systemd/system/docker.service

並添加以下內容

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
 
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
 
[Install]
WantedBy=multi-user.target

添加文件權限

chmod +x /etc/systemd/system/docker.service
systemctl daemon-reload

啟動docker

systemctl start docker

驗證

systemctl status docker			#查看Docker狀態
docker -v			#查看Docker版本

二、安裝airflow容器

先在宿主機安裝mysql

上傳mysql安裝包

先卸載自帶的mysql相關的包

 rpm -qa|grep mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
$ rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)
$ sudo rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
$ rpm -qa|grep mariadb

再依次安裝

bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-common-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-5.7.27-1.el7################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-compat-5.7.2################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-embedded-compat-5################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-devel-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-devel-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-devel-5.7.27-1.el################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-client-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-server-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$

skip-grant-tables #添加這句話，這時候登入mysql就不需要密碼

登錄mysql，輸入密碼處直接回車就可以

配置msyql

mysql> flush privileges; 
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@localhost = password('Admin123!');
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> create user 'airflow'@'localhost' identified by 'Airflow123!';
Query OK, 0 rows affected (0.05 sec)

mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'airflow'@'%' identified by 'AirFlow123!';
Query OK, 0 rows affected (0.02 sec)

mysql> GRANT all privileges on airflow.* TO 'airflow'@'%'  IDENTIFIED BY 'AirFlow123!';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> GRANT all privileges on airflow.* TO 'root'@'172.17.0.2' IDENTIFIED BY 'Admin123!';
Query OK, 0 rows affected, 1 warning (0.04 sec)

//172.17.0.2 是我的docker容器的地址

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

修改/etc/my.cnf配置文件，在[mysqld]下添加編碼配置

character_set_server=utf8
init_connect='SET NAMES utf8'

vim /etc/my.cnf

添加這一句

explicit_defaults_for_timestamp=1

編輯保存完重啟mysql服務: sudo systemctl restart mysqld.service

三、docker上安裝airflow

git clone https://github.com/puckel/docker-airflow.git /root/airflow //下載源碼到airflow文件夾

docker run --name airflow6  -v /usr/local/airflow/dags:/usr/local/airflow/dags -v /usr/local/airflow/airflowSql:/usr/local/airflow/airflowSql  -v /home/airflow/airflow.cfg:/usr/local/airflow/airflow.cfg -d -p 8086:8080 --privileged puckel/docker-airflow   //運行容器

/* 我這里把本地的目錄映射到容器里面去 通過 -v 參數進行映射**/

進入容器