一.基本功能:
1.啟動hive時報錯
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.init(JDBCStatsPublisher.java:265)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:412)
Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:388)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:417)
解決方案:
將mysql-connector-java-5.1.6-bin.jar包導入到$HIVE_HOME/lib目錄下
####2.啟動hive時報錯: ``` [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.TerminalFactory.create(TerminalFactory.java:101) at jline.TerminalFactory.get(TerminalFactory.java:158) Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.console.ConsoleReader.
####解決方案: ``` 將當前hive版本的$HIVE_HOME/lib目錄下的jline-2.12.jar包拷貝到$HADOOP_HOME/share/hadoop/yarn/lib目錄下, 並將舊版本的Hive的Jline包從$HADOOP_HOME/etc/hadoop/yarn/lib目錄下刪除 ```
####3.啟動hive時報錯 ``` Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.
解決方案:
1.查看hive-site.xml配置，會看到配置值含有"system:java.io.tmpdir"的配置項
2.新建文件夾${HIVE_HOME}/hive/logs
3.將含有"system:java.io.tmpdir"的配置項的值修改為${HIVE_HOME}/hive/logs
即: 新添屬性為
<property>
<name>hive.exec.local.scratchdir</name>
<value>${HIVE_HOME}/logs/HiveJobsLog</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>${HIVE_HOME}/logs/ResourcesLog</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<value>${HIVE_HOME}/logs/HiveRunLog</value>
<description>Location of Hive run time structured log file</description>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>${HIVE_HOME}/logs/OpertitionLog</value>
<description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>
####4.啟動hive時報錯 ``` Caused by: java.sql.SQLException: Access denied for user 'root'@'master' (using password: YES) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3269) ```
解決方案:
mysql密碼不正確, 查看hive-site.xml配置與mysql的密碼是否一致
####5.操作表數據時(向表中導入數據), 報錯: ``` FAILED: RuntimeException org.apache.hadoop.security.AccessControlException: Permission denied: user=services02, access=EXECUTE, inode="/tmp":services01:supergroup:drwx------ ```
解決方案:
user=services02與inode="/tmp":services01:supergroup不同時,說明hive登錄的主機與HDFS的active狀態的主機不一樣
應把user=services02的主機變為HDFS的active狀態的主機.
##二.擴展Parquet功能: ####1.創建存儲格式為parquet時,建表語句為: (Hive版本為0.12) ``` create table parquet_test(x int, y string) row format serde 'parquet.hive.serde.ParquetHiveSerDe' stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat' outputformat 'parquet.hive.DeprecatedParquetOutputFormat'; ``` #####報錯: ``` FAILED: SemanticException [Error 10055]: Output Format must implement HiveOutputFormat, otherwise it should be either IgnoreKeyTextOutputFormat or SequenceFileOutputFormat ```
####解決方案: ``` 因為parquet.hive.DeprecatedParquetOutputFormat類並沒有在Hive的CLASSPATH中配置 單獨下載parquet-hive-1.2.5.jar包(此類屬於$IMPALA_HOME/lib目錄下), 在$HIVE_HOME/lib目錄下建立個軟鏈就可以了 ``` ``` cd $HIVE_HOME/lib ln -s $/home/hadoop/soft/gz.zip/parquet-hive-1.2.5.jar ```
####2.繼續提交創表語句: (Hive版本為0.12) ``` create table parquet_test(x int, y string) row format serde 'parquet.hive.serde.ParquetHiveSerDe' stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat' outputformat 'parquet.hive.DeprecatedParquetOutputFormat'; ```
報錯:
Exception in thread "main" java.lang.NoClassDefFoundError: parquet/hadoop/api/WriteSupport
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
####解決方案: ``` 通過yum下載Parquet sodu yum -y install parquet ``` ``` 下載parquet的jar包在/usr/lib/parquet目錄下, 將/usr/lib/parquet目錄下的所有jar(除javadoc.jar和sources.jar外)拷貝到$HIVE_HOME/lib目錄下. 若yum無法下載parquet資源包, 則這是需要配置yum源, 請自行百度尋找資料 ```
####3.繼續提交創表語句:(Hive版本為0.12) ``` create table parquet_test(x int, y string) row format serde 'parquet.hive.serde.ParquetHiveSerDe' stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat' outputformat 'parquet.hive.DeprecatedParquetOutputFormat'; ```
報錯:
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
####解決方案: ``` 先啟動metastore服務: hive --service metastore ```
####4.將存儲格式為textFile的文件數據insert到parquet時, 報錯信息: ``` Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ... 9 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:134) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:386) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377) ... 22 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:323) ... 34 more FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec ```
####解決方案: ``` 在hive-env.sh中添加: JAVA_HOME=/home/hadoop/soft/jdk1.7.0_67 HADOOP_HOME=/home/hadoop/soft/hadoop-2.4.1 HIVE_HOME=/home/hadoop/soft/hive-0.12.0 export HIVE_CONF_DIR=$HIVE_HOME/conf export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib ```
####5.將textFile表的數據插入到parquet表中,報錯: ``` Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"time":1471928602,"uid":687994,"billid":1004,"archiveid":null,"year":"2016","mouth":"2016-08","day":"2016-08-23","hour":"2016-08-23-04"} Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.ArrayWritable ```
解決方案:
(1)配置hive-env.xml文件,添加:
JAVA_HOME=/home/hadoop/soft/jdk1.7.0_67
HADOOP_HOME=/home/hadoop/soft/hadoop-2.4.1
HIVE_HOME=/home/hadoop/soft/hive-0.12.0
export HIVE_CONF_DIR=$HIVE_HOME/conf
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib
(2)通過yum下載Parquet
sodu yum -y install parquet
(3)下載parquet的jar包在/usr/lib/parquet目錄下, 將/usr/lib/parquet目錄下的所有jar(除javadoc.jar和sources.jar外)拷貝到$HIVE_HOME/lib目錄下.
(4)文件壓縮格式問題:
在insert插入數據之前, 要先設置壓縮格式, 要三種格式可選, 通常選用SNAPPY:
CompressionCodecName.UNCOMPRESSED
CompressionCodecName.SNAPPY
CompressionCodecName.GZIP
(5)在Hive命令行中執行set parquet.compression = SNAPPY語句; 然后再進行insert數據插入操作
####6.創建存儲格式為parquet的表時,報錯 ``` FAILED: SemanticException [Error 10055]: Output Format must implement HiveOutputFormat, otherwise it should be either IgnoreKeyTextOutputFormat or SequenceFileOutputFormat ```
解決方案:
查看hive版本, 如果hive是0.13版本以下的, 則創表時用: stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat' outputformat 'parquet.hive.DeprecatedParquetOutputFormat';
若hive是0.13版本以上, 則創表時用: stored as parquet;
##三.Hive1.x升級到2.x異常匯總 --- ####1.啟動hive時報錯 ``` Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.
解決方案:
(1)刪除HDFS上的hive數據與hive數據庫
hadoop fs -rm -r -f /tmp/hive
hadoop fs -rm -r -f /user/hive
(2)刪除MySQL上的hive的元數據信息
mysql -uroot -p
drop database hive
隨之而來了一個問題:
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
解決方案:
初始化hive, 將mysql作為hive的元數據庫
schematool -dbType mysql -initSchema
####2.使用HPL/SQL時報錯: ``` java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused ```
解決方案(若方案一仍然報錯, 則執行方案二):
方案一:
啟動hiveServer2服務,
cd $HIVE_HOME/bin
./hiveserver2
方案二:
- 編輯hplsql-site.xml, 修改以下配置:
<property>
<name>hplsql.conn.default</name>
<value>hive2conn</value>
<description>The default connection profile</description>
</property>
<property>
<name>hplsql.conn.hive2conn</name>
<value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://m1:10000</value>
<description>HiveServer2 JDBC connection</description>
</property>
2.然后啟動hiveServer2服務
sh $HIVE_HOME/bin/hiveserver2
3.使用beeline測試連接
sh $HIVE_HOME/bin/beeline
!connect jdbc:hive2://m1:10000
####3.在hive2.0中使用hplsql, 對存儲過程文件進行執行時, 報錯: ``` java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive at org.apache.hive.jdbc.HiveConnection.
解決方案:
(1)暫停HDFS服務
stop-dfs.sh
(2)修改core-site.xml文件
在$HADOOP_HOME/etc/hadoop/core-site.xml文件中增加以下配置, 注意:Hadoop.proxyuser.centos.hosts配置項名稱中centos部分為報錯User:* 中的用戶名部分
<property>
<name>hadoop.proxyuser.centos.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.centos.hosts</name>
<value>*</value>
</property>
(3)將core-site.xml文件分發到其他主機
(4)啟動HDFS服務
start-dfs.sh
####4.在Hive2.0中使用HPL/SQL, 對存儲過程文件進行執行時, 報錯: ``` java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Permission denied: user=anonymous, access=EXECUTE, inode="/tmp":centos:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:208) ```
解決方案:
1.修改有問題的目錄權限
hadoop fs -chmod 755 /tmp
2.在hive-site.xml文件中修改以下配置項
<property>
<name>hive.scratch.dir.permission</name>
<value>755</value>
</property>
####5.使用HPL/SQL存儲過程時報錯, 報錯信息如下: ``` 在HPL/SQL2.2.1版本中, 使用from語句時, 總是從配置文件里找hplsql.dual.table配置項的值, 總是報錯找不到dual表, 或找不到select語句中指定cloumn ```
####解決方案: HPL/SQL官網 ``` http://www.hplsql.org/doc ``` ``` 注意: 一定得是0.3.17或以上版本的HPL/SQL 下載一個HPL/SQL 0.3.17的tan.gz文件, 解壓后將hplsql-0.3.17.jar包放入$HIVE_HOME包下, 並改名為hive-hplsql-*.jar格式的包,如:hive-hplsql-0.3.17.jar ```
####五.Hive On Spark作為引擎: --- ``` 官方參考配置項: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark 遇坑解決方案指南: http://www.cnblogs.com/breg/p/5552342.html 搭建教程及部分遇坑解決指南 http://www.cnblogs.com/linbingdong/p/5806329.html ``` ---
####1.啟動hive時報錯 ``` Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226) at org.apache.hadoop.hive.ql.metadata.Hive.
解決方案:
啟動metastore服務:
hive --service metastore
####2.調用spark引擎計算時, 報錯: ``` Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask ```
####解決方案: ``` 錯誤日志中出現Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask 該錯誤, 大部分是因為spark的安裝包是包含Hive引用包的, 出現此類問題應自己手動編譯一個spark包 ```
####3.調用spark引擎計算時, 報錯: ``` 2016-12-19T20:19:15,491 ERROR [main] client.SparkClientImpl: Error while waiting for client to connect. java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancel client 'dcee57ba-ea77-4e92-bd43-640e8385e2e7'. Error: Child process exited before connecting back with error log Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=200000 Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8 Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000 Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256 Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800 16/12/19 20:19:15 INFO client.RemoteDriver: Connecting to: m1:48286 Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS at org.apache.hive.spark.client.rpc.RpcConfiguration.
####解決方案: ``` 參考資料: http://www.cnblogs.com/breg/p/5552342.html ```
日志中出現Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS該錯誤, 大部分是因為spark的安裝包是包含Hive引用包的, 出現此類問題應自己手動編譯一個spark包
####4.使用編譯好的spark安裝包, 安裝好后, 啟動master時報錯: ``` Spark Command: /home/centos/soft/jdk1.7.0_67/bin/java -cp /home/centos/soft/spark/conf/:/home/centos/soft/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/home/centos/soft/hadoop/etc/hadoop/:/home/centos/soft/hadoop/etc/hadoop/:/home/centos/soft/hadoop/lib/spark-assembly-1.6.0-hadoop2.6.0.jar -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip m1 --port 7077 --webui-port 8080 ======================================== Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2570) at java.lang.Class.getMethod0(Class.java:2813) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) ```
解決方案:
出現此類問題, 均是spark源碼編譯時出錯, 推薦使用Maven編譯
####5.使用編譯好的spark安裝包, 安裝好后, 啟動master時報錯: ``` Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2570) at java.lang.Class.getMethod0(Class.java:2813) Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) ```
####解決方案: ``` 出現此類問題, 均是spark源碼編譯時出錯, 推薦使用Maven編譯 ```
####6.Hive On Spark使用hplsql存儲過程時報錯, **Beeline**中的報錯信息: ``` Unhandled exception in HPL/SQL java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://m1:10000: null at org.apache.hive.jdbc.HiveConnection.
Hive-log.log的報錯信息:
ERROR [HiveServer2-Handler-Pool: Thread-43] server.TThreadPoolServer:
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
####解決方案: 將```hive.server2.authentication```配置項改為```NONE``` ```
####7.使用spark-submit --master yarn提交任務時沒問題, 以HPL/SQL用存儲過程提交任務執行Spark任務時卻報錯: ``` 16/12/26 16:45:01 WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered 16/12/26 16:45:16 WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered ```
####解決方案: 出現這種問題的原因有很多種, 我所遇到的問題的解決方案是: 將```/etc/hosts```下的```127.0.0.1```注釋, 然后設置一個新值 ``` sudo vi /etc/hosts #127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 localhost ```
####8.報錯信息如下: ``` java.lang.StackOverflowError at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53) ```
####解決方案： SQL語句的`where`條件過長，字符串棧溢出
####9.報錯信息如下: ``` Error: Could not find or load main class org.apache.hive.beeline.BeeLine ```
####解決方案： 重新編譯Hive，並帶上參數`-Phive-thriftserver`
####10.報錯信息如下: ``` check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 ```
####解決方案： 使用新版`mysql-connector`包
####11.報錯信息如下: ``` java.lang.NoSuchMethodError: org.apache.parquet.schema.Types$MessageTypeBuilder.addFields([Lorg/apache/parquet/schema/Type;)Lorg/apache/parquet/schema/Types$BaseGroupBuilder; ```
####解決方案： ``` 版本沖突所致，統一`hive`和`spark`中`parquet`組件版本 ```
####12.報錯信息如下: ``` Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' ```
####解決方案： 不要官網編譯好的spark，自己下載Spark源碼重新編譯，並保證編譯的Spark版本滿足Hive源碼中pom.xml文件中對spark的一個大版本要求(如: 1.6.X), 若使用Hive on Spark,則在編譯過程中不帶Phive有關的任何參數
####13.報錯信息如下: ``` java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS at org.apache.hive.spark.client.rpc.RpcConfiguration.
####解決方案： 不要官網編譯好的spark，自己下載Spark源碼重新編譯，並保證編譯的Spark版本滿足Hive源碼中pom.xml文件中對spark的一個大版本要求, 若使用Hive on Spark,則在編譯過程中不帶Phive有關的任何參數