oozie4.3.0的安裝與配置 + hadoop2.7.3

安裝步驟

• mysql的配置
• oozie的安裝
• oozie的配置
• oozie的啟動與登錄
• 常用oozie的命令

1. mysql的配置

mysql的安裝自行解決，然后在mysql上

創建oozie數據庫，創建oozie用戶名和密碼，並賦值oozie賬號登陸oozie數據庫的權限。

mysql -u root -proot
create database oozie;  
//(設置oozie數據庫的訪問權限)  
grant all privileges on oozie.* to 'oozie'@'%' identified by 'password';             
FLUSH PRIVILEGES;  

注意要刪除msyql中兩個空的用戶，否則會一直報用戶無權限。

 

2. oozie的安裝

2.1 oozie的下載與編譯

https://mirrors.tuna.tsinghua.edu.cn/apache/oozie/4.3.0/oozie-4.3.0.tar.gz

官方提供的oozie只是源碼，需要自己編譯，解壓縮，使用以下命令編譯：

./mkdistro.sh -Phadoop-2 -Dhadoop.auth.version=2.7.3 -Ddistcp.version=2.7.3 -Dhadoop.version=2.7.3 -Dsqoop.version=1.4.6 -DskipTests 

編譯過程中會有3處錯誤(點這里參考這篇文章)，都是鏡像庫中包找不到，所以需要自己下載，然后放到本地maven倉庫目錄下即可。

編譯成功后，打包的文件路徑為：oozie-4.3.0/distro/target/oozie-4.3.0-distro.tar.gz

2.2 oozie的安裝

1）  解壓縮包oozie-4.3.0-distro.tar.gz到/usr/local/目錄下，然后進入目錄oozie-4.3.0,分別解壓縮里面的三個壓縮包oozie-client-4.3.0.tar.gz、oozie-examples.tar.gz、oozie-sharelib-4.3.0.tar.gz

完成后的文件列表如下：

2） 在hdfs上創建/user/oozie目錄，然后將share目錄上傳到hdfs中的/user/oozie目錄。

將mysql驅動和oracle驅動放到share/lib目錄下, 后面使用sqoop的時候，會使用hdfs的/user/oozie/share/lib/sqoop/目錄下的jar包。

cp ojdbc*.jar /usr/local/oozie-4.3.0/share/lib/sqoop/
cp mysql-connector-java-5.1.35-bin.jar /usr/local/oozie-4.3.0/share/lib/sqoop/
hdfs dfs -copyFromLocal /usr/local/oozie-4.3.0/share/ /user/oozie

3）  在/usr/local/oozie目錄下創建libext文件夾，然后復制hadoop的lib目錄下的文件到/usr/local/oozie/libext下。

ln -s oozie-4.3.0 oozie
cd oozie
mkdir libext
cp ${HADOOP_HOME}/share/hadoop/*/*.jar libext/
cp ${HADOOP_HOME}/share/hadoop/*/lib/*.jar libext/

添加ext-2.2.zip和mysql驅動包、oracle驅動包到libext

cp ext-2.2.zip /usr/local/oozie/libext/
cp mysql-connector-java-5.1.35-bin.jar /usr/local/oozie/libext/

4）  修改oozie-4.3.0/oozie-server/conf/server.xml文件，注釋掉下面的記錄

<!--<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />-->

5）  打war包

在bin目錄下執行命令：

./oozie-setup.sh prepare-war

war文件最終保存在/usr/local/oozie/oozie-server/webapps目錄下

 

 

3. oozie的配置

3.1 設置環境變量/etc/profile

#180112 oozie path
export OOZIE_HOME=/usr/local/oozie
export PATH=$OOZIE_HOME/bin:$PATH
export OOZIE_CONFIG=/usr/local/oozie/conf

# 這里要注意地址后面要帶/oozie，否則報404錯誤, 踩過的坑只有自己知道痛苦

export OOZIE_URL=http://dwtest-name1:11000/oozie

  3.2 修改配置文件/usr/local/oozie/conf/oozie-site.xml

默認conf文件夾下的oozie-site.xml文件都是注釋的，需要自己添加以下內容。

<?xml version="1.0"?>
<configuration>

    <!--
        Refer to the oozie-default.xml file for the complete list of
        Oozie configuration properties and their default values.
    -->

    <!-- Proxyuser Configuration -->

    <property>
        <name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name>
        <value>*</value>
        <description>
            List of hosts the '#USER#' user is allowed to perform 'doAs'
            operations.

            The '#USER#' must be replaced with the username o the user who is
            allowed to perform 'doAs' operations.

            The value can be the '*' wildcard or a list of hostnames.

            For multiple users copy this property and replace the user name
            in the property name.
        </description>
    </property>

    <property>
        <name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name>
        <value>*</value>
        <description>
            List of groups the '#USER#' user is allowed to impersonate users
            from to perform 'doAs' operations.

            The '#USER#' must be replaced with the username o the user who is
            allowed to perform 'doAs' operations.

            The value can be the '*' wildcard or a list of groups.

            For multiple users copy this property and replace the user name
            in the property name.
        </description>
    </property>

    <!-- 20180110 add -->
    <property>
        <name>oozie.service.JPAService.create.db.schema</name>
          <value>false</value>
        </property>
    <property>
        <name>oozie.service.JPAService.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>oozie.service.JPAService.jdbc.url</name>
        <value>jdbc:mysql://dwtest-name1:33061/oozie?createDatabaseIfNotExist=true</value>
    </property>
     
    <property>
        <name>oozie.service.JPAService.jdbc.username</name>
        <value>oozie</value>
    </property>
     
    <property>
        <name>oozie.service.JPAService.jdbc.password</name>
        <value>password</value>
        <description>
                DB user password.
                IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
                if empty Configuration assumes it is NULL.
        </description>
    </property>

    <property>
        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
        <value>*=/usr/local/hadoop/etc/hadoop</value>
    </property>

      <property>
        <name>oozie.service.HadoopAccessorService.action.configurations</name>
        <value>*=/usr/local/hadoop/etc/hadoop</value>
    </property>

    <property>
         <name>oozie.service.SparkConfigurationService.spark.configurations</name>
         <value>*=/usr/local/spark/conf</value>
    </property>

    <!-- 這里是保存在hdfs上的路徑 -->
    <property>
         <name>oozie.service.WorkflowAppService.system.libpath</name>
         <value>/user/oozie/share/lib</value>
    </property>

    <property>
        <name>oozie.use.system.libpath</name>
        <value>true</value>
        <description>
                Default value of oozie.use.system.libpath. If user haven't specified =oozie.use.system.libpath=
                in the job.properties and this value is true and Oozie will include sharelib jars for workflow.
        </description>
    </property>

    <property>
        <name>oozie.subworkflow.classpath.inheritance</name>
        <value>true</value>
    </property>


</configuration>

 

 3.3 創建元數據表

在/usr/local/oozie/bin目錄下執行以下命令生成sql文件，並創建元數據表

bin/ooziedb.sh create -sqlfile oozie.sql -run

 

可以在mysql的oozie數據庫中看到以下表被創建：

3.4 修改core-site.xml文件(視具體環境設置)

 如果調用oozie job的賬號與hadoop不一致，則需要修改hadoop的配置文件core-site.xml，添加調用oozie job的用戶組，由於我使用的都是hadoop賬戶，則不需要做任何修改。

修改完后，拷貝到Second name節點。使用以下命令來刷新，不需要重啟hadoop集群。

scp core-site.xml hadoop@dwtest-name2:/usr/local/hadoop/etc/hadoop/
hdfs dfsadmin -refreshSuperUserGroupsConfiguration  
yarn rmadmin -refreshSuperUserGroupsConfiguration

  

4. oozie的啟動與登錄

啟動與停止的腳本

bin/oozied.sh start
bin/oozied.sh stop

 啟動時，顯示如下：

 

登錄: http://dwtest-name1:11000/oozie/

不兼容包的刪除：

 oozie啟動后會自動將war包解壓生成一個oozie文件夾。

此時需要將/usr/local/oozie/oozie-server/webapps/oozie/WEB-INF/lib中包含hadoop2.6.0版本的包刪除或者移走，

否則啟動job會提示錯誤Error, java.lang.NoSuchFieldError: HADOOP_CLASSPATH

5. 常用oozie的命令

desc	command
查看共享庫pig包	oozie admin -oozie http://localhost:11000/oozie -shareliblist spark
查看共享庫	oozie admin -oozie http://localhost:11000/oozie -shareliblist
提交任務	oozie job -oozie http://localhost:11000/oozie -config job.properties -submit
執行任務	oozie job -oozie http://localhost:11000/oozie -config job.properties -run
殺死任務	oozie job -oozie http://localhost:11000/oozie -kill jobid
重新運行任務	oozie job -oozie http://localhost:11000/oozie -config job.properties  -rerun jobid
改變作業參數	oozie job -oozie http://localhost:11000/oozie -change jobid -value concurrency=1000;endtime=2018-01-10
檢查作業狀態	oozie job -oozie http://localhost:11000/oozie -info jobid
查看作業日志	oozie job -oozie http://localhost:11000/oozie -log jobid
檢查XML是否符合規范	oozie calidate myapp/workflow.xml

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

help查詢：

oozie help calidata //查詢所有命令
oozie help admin
oozie help job