Hive(一)安裝與配置

hive安裝方式有2種，第一種是最簡單安裝，直接使用hive內嵌的derby作為元數據庫，第二種是標准安裝，將MySQL數據庫作為元數據庫

以下介紹標准安裝方式：

1、安裝包下載

下載地址：http://archive.apache.org/dist/hive/  （或者 http://hive.apache.org/downloads.html ）

2、解壓

tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /root/opt/hive

3、配置

1. hive-env.sh

   # HADOOP_HOME=${bin}/../../hadoop
   HADOOP_HOME=/root/opt/hadoop/hadoop-3.2.1
   # Hive Configuration Directory can be controlled by:
   export HIVE_CONF_DIR=/root/opt/hive/hive-3.1.2/conf

   # Folder containing extra libraries required for hive compilation/execution can be controlled by:
   export HIVE_AUX_JARS_PATH=/root/opt/hive/hive-3.1.2/lib

2. hive-site.xml

   #創建hive-site.xml文件
   vim hive-site.xml

   添加如下內容：

   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   <configuration>
       <property>
           <name>javax.jdo.option.ConnectionURL</name>
            <value>jdbc:mysql://centos3:3306/hive?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionDriverName</name>
           <value>com.mysql.cj.jdbc.Driver</value> <!--MySQL8之后的數據庫驅動名稱-->
       </property>
       <property>
           <name>javax.jdo.option.ConnectionUserName</name>
           <value>root</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionPassword</name>
           <value>123456</value>
       </property>
       <property>
           <name>datanucleus.readOnlyDatastore</name>
           <value>false</value>
       </property>
       <property>
           <name>datanucleus.fixedDatastore</name>
           <value>false</value>
       </property>
       <property>
           <name>datanucleus.autoCreateSchema</name>
           <value>true</value>
       </property>
       <property>
           <name>datanucleus.schema.autoCreateAll</name>
           <value>true</value>
       </property>
       <property>
           <name>datanucleus.autoCreateTables</name>
           <value>true</value>
       </property>
       <property>
           <name>datanucleus.autoCreateColumns</name>
           <value>true</value>
       </property>
       <property>
           <name>hive.metastore.local</name>
           <value>true</value>
       </property>
       <!-- 顯示表的列名 -->
       <property>
           <name>hive.cli.print.header</name>
           <value>true</value>
       </property>
       <!-- 顯示數據庫名稱 -->
       <property>
           <name>hive.cli.print.current.db</name>
           <value>true</value>
       </property>
   </configuration>

4、上傳mysql的驅動jar包到hive的安裝目錄的lib文件夾中，同時在mysql中創建hive的元表

5、配置環境變量

vi /etc/profile

export HIVE_HOME=/root/opt/hive/hive-3.1.2

export PATH=$PATH:$HIVE_HOME/bin

source /etc/profile

 6、進入shell命令

hive

如果報如下錯誤：

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
	at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
	at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

這是hive和hadoop的jar包沖突引起，com.google.common.base.Preconditions.checkArgument 指 guava.jar

hadoop的依賴位置：hadoop-3.2.1/share/hadoop/common/lib

hive的依賴位置：apache-hive-3.1.2-bin/lib

通常用hadoop的高版本覆蓋hive的低版本，版本一致即可。

 

如果出現如下異常：

2020-10-09 16:31:35,849 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
2020-10-09 16:31:35,849 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
2020-10-09 16:31:35,849 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
2020-10-09 16:31:35,849 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
2020-10-09 16:31:35,849 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
2020-10-09 16:31:36,912 WARN  [4259aeab-b356-437e-a541-76f33bbe2693 main] DataNucleus.Query: Query for candidates of org.apache.hadoop.hive.metastore.model.MVersionTable and subclasses resulted in no possible candidates
Required table missing : "`VERSION`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables"
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : "`VERSION`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables"
	at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:606)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3385)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896)
	at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672)
	at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:425)
	at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:865)
	at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:347)
	at org.datanucleus.store.query.Query.executeQuery(Query.java:1816)
	at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
	at org.datanucleus.store.query.Query.execute(Query.java:1726)
	at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:374)
	at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:9101)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:9085)
	at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9042)
	at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy36.verifySchema(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:697)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:690)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:767)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:538)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8667)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:94)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4299)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4367)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4347)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
	at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
	at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
	at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331)
	at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.init(HiveMaterializedViewsRegistry.java:133)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive (default)>

則，重新初始化數據庫：

bin/schematool -dbType mysql -initSchema

7、啟動hive服務

1.使用服務的形式啟動hive

nohup hiveserver2 >/dev/null 2>&1 &

2.鏈接方式

nohup hiveserver2 &

1：beenline

hive --service beeline !connect jdbc:hive2://centos2:10000 root 123456

2:beeline -u jdbc:hive2://centos2:10000 -n root

set hive.exec.mode.local.auto=true;