hive部署手冊

安裝環境:    

      機器 只需要安裝一台機器
      操作系統:Ubuntu 11.04 64操作系統
      hadoop:版本是1.0.2,安裝在/usr/local/hadoop
      sun jdk:版本是1.6.0_31 64bit,安裝在/usr/local/jdk
      hive:版本是0.8.1,安裝在/usr/local/hive

安裝步驟:

1.下載  

    下載hive:http://labs.mop.com/apache-mirror/hive/hive-0.8.1/hive-0.8.1.tar.gz

2.安裝

(1)上傳hive安裝包到機器上,使用root用戶登陸:   

 tar -xvf hive-0.8.1.tar.gz

(2)將解壓的hive分別移動並改名為/usr/local/hive

rm -rf /usr/local/hive
mv hive-0.8.1  /usr/local/hive

 

3.配置hive

(1)修改/usr/local/hive/bin/hive-config.sh

    在文件末尾加入

export JAVA_HOME=/usr/local/jdk  
export HIVE_HOME=/usr/local/hive  
export HADOOP_HOME=/usr/local/hadoop  

(2) 根據hive-default.xml復制hive-site.xml

cp /usr/local/hive/conf/hive-default.xml /usr/local/hive/conf/hive-site.xml

(3)配置hive-site.xml,主要配置項如下:
        hive.metastore.warehouse.dir：（HDFS上的）數據目錄
        hive.exec.scratchdir：（HDFS上的）臨時文件目錄
        hive.metastore.warehouse.dir默認值是/user/hive/warehouse
        hive.exec.scratchdir默認值是/tmp/hive-${user.name}
        以上是默認值，暫時不改。
(4)改變 /usr/local/hive的目錄所有者為hadoop

chown -R hadoop:hadoop /usr/local/hive

(5)配置hive的log4j:
      cp /usr/loca/hive/conf/hive-log4j.properties.template  /usr/loca/hive/conf/hive-log4j.properties
   修改/usr/loca/hive/conf/hive-log4j.properties將org.apache.hadoop.metrics.jvm.EventCounter改為org.apache.hadoop.log.metrics.EventCounter
(6)啟動hive

      使用hadoop用戶登陸,執行/usr/local/hive/bin/hive
(7)測試hive

hive> create TABLE pokes( id INT, name string);  
hive> SHOW TABLES; 
hive> select * from pokes; 
hive> drop table pokes;

4.優化hive

默認meta數據庫為derby ，為了避免使用默認的Derby數據庫（有並發訪問和性能的問題），通常還需要配置元數據庫為MySQL

修改配置文件conf/hive-site.xml 

<property>
  <name>hive.metastore.local</name>
  <value>false</value>
  <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://mysql_server_host:3306/hivedb?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=latin1</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>mysql_username</value>
  <description>username to use against metastore database</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>mysql_password</value>
  <description>password to use against metastore database</description>
</property>

<property>
  <name>hive.stats.dbconnectionstring</name>
  <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=latin1&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value>
  <description>The default connection string for the database that stores temporary hive statistics.</description>
</property>

<property>
  <name>hive.stats.dbconnectionstring</name>
  <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=utf8&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value>
  <description>The default connection string for the database that stores temporary hive statistics.</description>
</property>

<property>
  <name>hive.stats.dbclass</name>
  <value>jdbc:mysql</value>
  <description>The default database that stores temporary hive statistics.</description>
</property>

<property>
  <name>hive.stats.jdbcdriver</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>The JDBC driver for the database that stores temporary hive statistics.</description>
</property>

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://127.0.0.1:9083</value>
</property>

添加metastore啟動腳本bin/hive-metastore.sh

#!/bin/sh
nohup ./hive --service metastore >> metastore.log 2>&1 &
echo $! > hive-metastore.pid

添加hive server啟動腳本bin/hive-server.sh

nohup ./hive --service hiveserver >> hiveserver.log 2>&1 &
echo $! > hive-server.pid

啟動metastore和hive server

./hive-metastore.sh
./hive-server.sh

使用客戶端連接和測試

參考之前的hive測試步驟一一執行。

如果出現
FAILED: Error in metadata: javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.
NestedThrowables:
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

或者是
FAILED: Error in metadata: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

以上錯誤，主要是在select操作、drop操作時，就會出錯。

這樣的錯誤，可以修改hivedb數據庫編碼:

mysql> alter database hivedb character set latin1;

然后重啟hive metastore和hive server就可以了

mysql數據庫的編碼目前最好設置為latin1,否則使用hive會出現莫名其妙的問題