【hive】——metastore的三種模式

Hive中metastore（元數據存儲）的三種方式：

• 內嵌Derby方式
• Local方式
• Remote方式

[一]、內嵌Derby方式

這個是Hive默認的啟動模式，一般用於單元測試，這種存儲方式有一個缺點：在同一時間只能有一個進程連接使用數據庫。

hive-site.xml 中jdbc URL、驅動、用戶名、密碼等的配置信息如下：

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

<property>   <name>javax.jdo.option.ConnectionURL</name>   <value>jdbc:derby:;databaseName=metastore_db;create=true</value>   <description>JDBC connect string for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionDriverName</name>   <value>org.apache.derby.jdbc.EmbeddedDriver</value>   <description>Driver class name for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionUserName</name>   <value>APP</value>   <description>username to use against metastore database</description> </property>   <property>   <name>javax.jdo.option.ConnectionPassword</name>   <value>mine</value>   <description>password to use against metastore database</description> </property>   <property>   <name>hive.metastore.warehouse.dir</name>   <value>file:///Users/micmiu/tmp/hive/warehouse</value>   <description>unit test data goes in here on your local filesystem</description> </property>   <!-- micmiu.com -->

執行初始化命令：schematool -dbType derby -initSchema

查看初始化后的信息： schematool -dbType derby -info

配置完成后就可在shell中以CLI的方式訪問hive 進行操作驗證。

[二]、Local方式

以本地Mysql數據庫為例：創建好用戶：hive；database：hive。

配置文件 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置如下：

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

<property>   <name>javax.jdo.option.ConnectionURL</name>   <value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value>   <description>JDBC connect string for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionDriverName</name>   <value>com.mysql.jdbc.Driver</value>   <description>Driver class name for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionUserName</name>   <value>hive</value>   <description>username to use against metastore database</description> </property>   <property>   <name>javax.jdo.option.ConnectionPassword</name>   <value>micmiu</value>   <description>password to use against metastore database</description> </property>   <property>   <name>hive.metastore.warehouse.dir</name>   <!-- base hdfs path -->   <value>/user/hive/warehouse</value>   <description>location of default database for the warehouse</description> </property> <!-- micmiu.com -->

ps：需要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中

如果是第一次需要執行初始化命令：schematool -dbType mysql -initSchema

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

micmiu-mbp:mysql micmiu$ schematool -dbType mysql -initSchema 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative Metastore connection URL: jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive Starting metastore schema initialization to 0.12.0 Initialization script hive-schema-0.12.0.mysql.sql Initialization script completed schemaTool completeted

查看初始化后信息 schematool -dbType mysql -info

初始化后查看mysql中表情況：show tables;

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

mysql> show tables; +---------------------------+ | Tables_in_hive            | +---------------------------+ | BUCKETING_COLS            | | CDS                       | | COLUMNS_V2                | | DATABASE_PARAMS           | | DBS                       | | DB_PRIVS                  | | DELEGATION_TOKENS         | | GLOBAL_PRIVS              | | IDXS                      | | INDEX_PARAMS              | | MASTER_KEYS               | | NUCLEUS_TABLES            | | PARTITIONS                | | PARTITION_EVENTS          | | PARTITION_KEYS            | | PARTITION_KEY_VALS        | | PARTITION_PARAMS          | | PART_COL_PRIVS            | | PART_COL_STATS            | | PART_PRIVS                | | ROLES                     | | ROLE_MAP                  | | SDS                       | | SD_PARAMS                 | | SEQUENCE_TABLE            | | SERDES                    | | SERDE_PARAMS              | | SKEWED_COL_NAMES          | | SKEWED_COL_VALUE_LOC_MAP  | | SKEWED_STRING_LIST        | | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES             | | SORT_COLS                 | | TABLE_PARAMS              | | TAB_COL_STATS             | | TBLS                      | | TBL_COL_PRIVS             | | TBL_PRIVS                 | | TYPES                     | | TYPE_FIELDS               | | VERSION                   | +---------------------------+ 41 rows in set (0.00 sec)

配置完成后就可在shell中以CLI的方式訪問hive 進行操作驗證。

[三]、Remote方式

以Mysql數據庫(192.168.6.77)為例：創建好用戶：hive；database：hive_meta。Remote方式需要分別配置服務端和客戶端的配置文件：

服務端的 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置如下：

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

<property>   <name>javax.jdo.option.ConnectionURL</name>   <value>jdbc:mysql://192.168.6.77/hive_meta?createDatabaseIfNotExist=true</value>   <description>JDBC connect string for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionDriverName</name>   <value>com.mysql.jdbc.Driver</value>   <description>Driver class name for a JDBC metastore</description> </property>   <property>   <name>javax.jdo.option.ConnectionUserName</name>   <value>hive</value>   <description>username to use against metastore database</description> </property>   <property>   <name>javax.jdo.option.ConnectionPassword</name>   <value>micmiu</value>   <description>password to use against metastore database</description> </property>   <property>   <name>hive.metastore.warehouse.dir</name>   <!-- base hdfs path -->   <value>/user/hive/warehouse</value>   <description>base hdfs path :location of default database for the warehouse</description> </property> <!-- micmiu.com -->

ps：需要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中

如果是第一次需要執行初始化命令：schematool -dbType mysql -initSchema

客戶端中配置內容修改如下：

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13

<!-- thrift://<host_name>:<port> 默認端口是9083 --> <property>   <name>hive.metastore.uris</name>   <value>thrift://192.168.6.77:9083</value>   <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property>   <!--  hive表的默認存儲路徑 --> <property>   <name>hive.metastore.warehouse.dir</name>   <value>/user/hive/warehouse</value>   <description>location of default database for the warehouse</description> </property>

hive metastore 服務端啟動命令：
hive --service metastore -p <port_num>
如果不加端口默認啟動：hive --service metastore，則默認監聽端口是：9083 ，注意客戶端中的端口配置需要和啟動監聽的端口一致。服務端啟動正常后，客戶端就可以執行hive操作了。

參考：

https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin