azkaban 編譯部署

安裝azkaban單機版和集群版

一、安裝單機版

1、安裝git

yum install git 

2、拉取azkaban源碼

git clone https://github.com/azkaban/azkaban.git

　　

3、開始編譯

cd azkaban; 
開始編譯
./gradlew build installDist

　　

編譯過程很漫長，一直在下載包，可能出現超時的情況，然后再繼續編譯就好

如果出現JCE報錯，則需要手動下載相關的包

testV1_1 FAILED
    java.lang.RuntimeException: java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at azkaban.crypto.Crypto.decrypt(Crypto.java:76)
        at azkaban.crypto.DecryptionTest.testV1_1(DecryptionTest.java:35)

        Caused by:
        java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
            at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:57)
            at azkaban.crypto.Crypto.decrypt(Crypto.java:74)
            ... 1 more

            Caused by:
            org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1050)
                at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
                at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:55)
                ... 2 more
azkaban.crypto.EncryptionTest > testEncryption FAILED
    org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:924)
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)
        at azkaban.crypto.CryptoV1_1.encrypt(CryptoV1_1.java:42)
        at azkaban.crypto.Crypto.encrypt(Crypto.java:58)
        at azkaban.crypto.EncryptionTest.testEncryption(EncryptionTest.java:28)

　　解決方法：
下載JCE
https://www.oracle.com/technetwork/cn/java/javase/downloads/jce8-download-2133166-zhs.html
我這邊使用的JDK8，包含了JCE所需要的jre8 ，解壓出來 ，放到jdk安裝目錄的 jre/lib/security下面

參考 https://yq.aliyun.com/articles/648399

4、啟動服務

cd azkaban-solo-server/build/install/azkaban-solo-server; 
bin/start-solo.sh

　　

注意，這里不能直接進入到bin目錄啟動，否則會報找不到database.properties的錯誤

java.io.IOException: Cannot find 'database.properties' file in /opt/local/azkaban-solo-server/build/install/azkaban-solo-server/bin/sql/database.properties
	at azkaban.database.AzkabanDatabaseSetup.loadDBProps(AzkabanDatabaseSetup.java:178)
	at azkaban.database.AzkabanDatabaseSetup.loadTableInfo(AzkabanDatabaseSetup.java:102)
	at azkaban.database.AzkabanDatabaseUpdater.runDatabaseUpdater(AzkabanDatabaseUpdater.java:82)
	at azkaban.soloserver.AzkabanSingleServer.start(AzkabanSingleServer.java:93)
	at azkaban.soloserver.AzkabanSingleServer.main(AzkabanSingleServer.java:58)　　

 

參考： https://blog.csdn.net/SUDDEV/article/details/101369524

5、查看服務

訪問 http://ip:8081 

默認賬號密碼是 azkaban azkaban

6、停止服務 

bin/shutdown-solo.sh

　　

7、如果是離線安裝，可以在可以上網的機器編譯，然后將編譯好后的

azkaban/azkaban-solo-server/build/install/azkaban-solo-server  復制到安裝機器上，然后啟動即可。

 二、安裝集群版

1、在上面編譯好的目錄中，azkaban-solo-server是單機版服務，azkaban-exec-server是后台服務，azkaban-web-server 是網頁服務
我們現在需要在上面編譯好的文件中，將后台服務和網頁服務復制到安裝目錄

cp -r azkaban/azkaban-exec-server/build/install/azkaban-exec-server /opt/local
cp -r azkaban/azkaban-web-server/build/install/azkaban-web-server /opt/local

　　

2、創建azkaban依賴的MySQL數據庫

mysql> CREATE DATABASE azkaban;

　　

3、生成初始化MySQL數據庫文件

cd azkaban/azkaban-db
../gradlew build installDist

　　

文件在azkaban/azkaban-db/build/sql，放到MySQL執行

4、配置和啟動后台服務

cd /opt/local/azkaban-exec-server/conf
vim azkaban.properties，修改MySQL配置、時區

default.timezone.id=Asia/Shanghai
executor.port=12321
mysql.port=3306
mysql.host=數據庫ip
mysql.database=azkaban
mysql.user=數據庫賬號
mysql.password=數據庫密碼

　

啟動服務

bin/start-exec.sh

5、配置和啟動網頁服務

cd /opt/local/azkaban-web-server/conf

vim azkaban.properties

default.timezone.id=Asia/Shanghai

mysql.port=3306
mysql.host=數據庫ip
mysql.database=azkaban
mysql.user=數據庫賬號
mysql.password=數據庫密碼

　　

啟動服務 bin/start-web.sh 

然后發現報錯了，發現沒有可用的executor

2020/05/15 15:05:46.300 +0800  INFO [AzkabanWebServer] [main] [Azkaban] ************* loginLoggerObjName is null, m
ake sure there is a logger with name azkaban.webapp.servlet.LoginAbstractAzkabanServlet
2020/05/15 15:05:46.301 +0800  INFO [ExecutorManager] [main] [Azkaban] Initializing executors from database.
2020/05/15 15:05:46.302 +0800 ERROR [ExecutorManager] [main] [Azkaban] No active executors found
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Exception in thread "main" 
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] azkaban.executor.ExecutorManagerException:
 No active executors found
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ActiveExecutors
.setupExecutors(ActiveExecutors.java:52)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.setupExecutors(ExecutorManager.java:192)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.initialize(ExecutorManager.java:127)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.start(ExecutorManager.java:141)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.webapp.AzkabanWebServer.
launch(AzkabanWebServer.java:234)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.webapp.AzkabanWebServer.
main(AzkabanWebServer.java:227)

　　

我們改一下數據庫，激活executor

use azkaban;
update azkaban.executors set active=1;

　　

再啟動一次服務，即可

對於集群部署，將azkaban-web-server按照上面的步驟部署，然后執行 curl -G "http:ip:port/executor?action=activate" 激活節點，或者去MySQL改數據激活

　　

　　

1、
executor became inactive before setting up the flow

描述：重啟web server的時候，無法執行舊的flow，日志報錯如上

原因：Azkaban Executor 啟動后不會自動激活

解決：

執行命令手動激活executor：
# 在MySQL表中看具體的端口 select * from executors;
curl http://localhost:2663/executor?action=activate
要求，先啟動exec，然后激活，再啟動web

2、
azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib
在執行節點的conf配置文件查看該配置項
azkaban-exec-server/conf
找到該配置項的文件，
azkaban.jobtype.plugin.dir
修改不加載本地原生庫
execute.as.user=false
azkaban.native.lib=false

3、
Unable to get scheduled flow triggers

描述：在執行任務的時候，任務始終處於Preparing（等待）狀態

原因：web server在執行前會對executor進行一個過濾，我們需要修改這個過濾器規則，通常是內存限制

解決：

修改：azkaban-web-server/conf/azkaban.properties

#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus