1. oozie 調用sql文件的workflow 錯誤匯總:
1)hive2server密碼錯誤。(有時設置可以無密碼,有時需要登陸密碼,有時是單獨的hive2server密碼)
Connecting to jdbc:hive2://spark-02:10000/default
Error: Could not open client transport with JDBC Uri: jdbc:hive2://spark-02:10000/default: Peer indicated failure: Error validating the login (state=08S01,code=0)
No current connection
Error: Could not open client transport with JDBC Uri: jdbc:hive2://spark-02:10000/default: Peer indicated failure: Error validating the login (state=08S01,code=0)
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
2)空格無法識別。
INFO : Table sales_analysis.sa_item_sku_day_sales stats: [numFiles=1, numRows=0, totalSize=98, rawDataSize=0]
Error: Error while compiling statement: FAILED: ParseException line 2:8 cannot recognize input near ',' 'dp_idstring' ',' in column type (state=42000,code=40000)
Closing: 0: jdbc:hive2://spark-02:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
原因:
dp_id與string之間的空格無法識別。
所以多加了幾個空格,query執行正常。
3)腳本開頭的亂碼字符無法識別。
可能是腳本出現了亂碼,復制到ue等編輯器下,刪除開頭試試。
2. oozie 無法調用impala sql
原因貌似是,無法訪問文件路徑,可能是要文件放在oozie 主機上。
解決辦法:
1)把簡單的impala sql 放在impala的主機上,然后crontab 調用
2)復雜的impala sql,可以把源表和結果表存成 parquet表,然后hive sql執行后,進行元數據同步
3)kudu表只能用 impala 讀取到parquet表后,hive才能訪問數據
刷新元數據: invalidate metadata; refresh + 表名
3. oozie 調用shell腳本
sync impala腳本內容
#!/bin/bash export PYTHON_EGG_CACHE=./myeggs /usr/bin/impala-shell -q 'invalidate metadata'