Azkaban 2.5.0 job type 插件安裝

一、環境及軟件

安裝環境：

• 安裝目錄： /usr/local/ae/ankaban
• Hadoop 安裝目錄 export HADOOP_HOME=/usr/local/ae/hadoop-1.2.1
• azkaban-executor-2.5.0安裝目錄：/usr/local/ae/azkaban/azkaban-executor-2.5.0

安裝軟件：

• azkaban-jobtype-2.5.0.tar.gz

Azkaban jobtype plugin source：github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype

二、配置jobtype

1. 解壓 azkaban-jobtype-2.5.0.tar.gz 到 $AZKABAN-EXECUTOR_HOME/plugins/jobtypes 目錄下
   

   user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0/plugins/jobtypes$ tar -zxvx azkaban--jobtype-2.5.0.tar.gz

2. jobtype properties 配置
   common.properties
   

   ## everything that the user job can know
   
   hadoop.home=/usr/local/ae/hadoop-1.2.1
   #hive.home=
   #pig.home=
   
   azkaban.should.proxy=false
   jobtype.global.classpath=${hadoop.home}/hadoop-core-1.2.1.jar,${hadoop.home}/*,${hadoop.home}/conf/,${hadoop.home}/lib/*

   commonprivate.properties

   ## hadoop security manager setting common to all hadoop jobs
   hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0
   
   ## hadoop security related settings
   
   # proxy.keytab.location=YOUR_AZKABAN_KERBEROS_PRINCIPAL
   #  proxy.user=KEYTAB_LOCATION
   
   azkaban.should.proxy=false
   obtain.binary.token=false
   # obtain.namenode.token=true
   # obtain.jobtracker.token=true
   
   # global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf
   #jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf
   
   # global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path
   #jobtype.global.jvm.args=
   
   # hadoop
   hadoop.home=/usr/local/ae/hadoop-1.2.1
   #pig.home=
   #hive.home=

三、配置 jobtype 插件安裝目錄到 Azkaban-executor

1. 配置 azkaban.properties
   在 azkaban.properties 文件中添加以下配置
   

   # Azkaban JobTypes Plugins
   azkaban.jobtype.plugin.dir=plugins/jobtypes

四、重啟 Azkaban-executor

user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0$ sh bin/azkaban-executor-shutdown.sh
user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0$ sh bin/azkaban-executor-start.sh

五、HadoopJava jobtype profile

type=hadoopJava
#jvm.args
job.class=×××××××××××××××××××××****
main.args=×××××××××
method.run=run
#method.cancel
classpath=./lib/*,/usr/local/ae/hadoop-1.2.1/*,/usr/local/ae/hadoop-1.2.1/lib/*,/usr/local/ae/hbase-0.94.8/*,/usr/local/ae/hbase-0.94.8/lib/*
#dependencies
#getJobGeneratedProperties

Note：1. 若 ${job.class} 繼承自 AbstractHadoopJob 則需要設置 FileIutPutFormat 與 FileOutPutFormat

         2. Azkaban 在啟動是會生成兩個日志文件azkaban-access.log/azkaban-webserver.log，他們的生成位置是在你執行腳本的目錄，所以建議你最好還是在AZKABAN_HOME 目錄下執行啟動腳本，如果你喜歡在 ./bin 目錄下啟動，你需要將上文第3.1步驟的紅色標記處修改目錄位置為 ../${dir}