HUE中Oozie執行Sqoop

Oozie執行Sqoop，傳入參數（注意，在使用--query時，參數中一定不要帶有空格，否則出錯）
1. 新建一個workflow

 

2. 拖入一個sqoop

 

3. sqoop抽取命令如下（建議先在命令行試下，因為oozie的錯誤提示很委婉，基本看不出錯誤原因）

sqoop import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by '`' --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

這是在命令行可執行的命令，其中有三個動態參數，${date}、${start_time}、${end_time}，此處做了一個import的導入操作，把指定數據抽取到HDFS路徑下，用了--query寫SQL語句抽取想要的數據，此處需要注意，如果使用--query，必須要帶\$CONDITIONS，否則執行失敗。
4. 填入oozie-sqoop中，需要填入的內容如下

import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by ` --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

注意：指定分隔符時，要把單引號去掉，下邊還有一個大坑
5. 添加參數

 

6. 啟動，填入參數

 

7. 那么問題來了，報錯是必須的
oozie在解析sqoop語句的時候，是根據空格來分割的，那就惡心了，日期字段中間帶有空格，就像這樣

Setting up log4j2
log4j2 configuration file created at /data1/yarn/nm/usercache/fengz/appcache/application_1552354446915_0759/container_1552354446915_0759_01_000001/sqoop-log4j2.xml
Sqoop command arguments :
import
--connect
jdbc:mysql://localhost:3306/spider_new
--username
root
--password
********
--target-dir
/user/fengz/brand/spider_data/amac/amac_fund_20190315
--delete-target-dir
--fields-terminated-by
`
--query
"select
*
from
amac_fund
where
update_time
between
'2019-03-14
10:57:08'
and
'2019-03-14
14:08:45'
and
\$CONDITIONS"
-m
1
Fetching child yarn jobs
tag id : oozie-3e829587c1f3a050998935d2c2ac3ed4

 

再看下oozie的錯誤提示，呵呵呵....發現一點鳥用都沒有

No child hadoop job is executed.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
at java.lang.Runtime.exit(Runtime.java:107)
at java.lang.System.exit(System.java:971)
at org.apache.sqoop.Sqoop.main(Sqoop.java:251)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:214)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:199)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
... 16 more
Intercepting System.exit(1)
Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

以后用的時候注意下就好了。那這個問題怎么解決？用shell，shell腳本里邊寫sqoop抽取。