記一次hive版本升級


  之前的項目中hive版本用的是 apache-hive-1.2.2-bin,最近需要將hive版本升級到apache-hive-2.3.4-bin, 本來想着元數據在mysql中,直接部署好hive啟動就好了,

結果提示元數據未初始化,查找一下資料,發現不同版本hive的元數據信息有所不同,而官方也給了升級方法,這里記錄一下。

  首先,找到我們要升級的hive,找到apache-hive-2.3.4-bin/scripts/metastore/upgrade/mysql  這個目錄

 

 

 這個目錄下有一個README,里面講了如何升級

Hive MetaStore Upgrade HowTo
============================

This document describes how to upgrade the schema of a MySQL backed
Hive MetaStore instance from one release version of Hive to another
release version of Hive. For example, by following the steps listed
below it is possible to upgrade a Hive 0.5.0 MetaStore schema to a
Hive 0.7.0 MetaStore schema. Before attempting this project we
strongly recommend that you read through all of the steps in this
document and familiarize yourself with the required tools.

MetaStore Upgrade Steps
=======================

1) Shutdown your MetaStore instance and restrict access to the
   MetaStore's MySQL database. It is very important that no one else
   accesses or modifies the contents of database while you are
   performing the schema upgrade.

2) Create a backup of your MySQL metastore database. This will allow
   you to revert any changes made during the upgrade process if
   something goes wrong. The mysqldump utility is the easiest way to
   create a backup of a MySQL database:

   % mysqldump --opt <metastore_db_name> > metastore_backup.sql

   Note that you may need also need to specify a hostname and username
   using the --host and --user command line switches.

3) Dump your metastore database schema to a file. We use the mysqldump
   utility again, but this time with a command line option that
   specifies we are only interested in dumping the DDL statements
   required to create the schema:

   % mysqldump --skip-add-drop-table --no-data <metastore_db_name> > my-schema-x.y.z.mysql.sql

4) The schema upgrade scripts assume that the schema you are upgrading
   closely matches the official schema for your particular version of
   Hive. The files in this directory with names like
   "hive-schema-x.y.z.mysql.sql" contain dumps of the official schemas
   corresponding to each of the released versions of Hive. You can
   determine differences between your schema and the official schema
   by diffing the contents of the official dump with the schema dump
   you created in the previous step. Some differences are acceptable
   and will not interfere with the upgrade process, but others need to
   be resolved manually or the upgrade scripts will fail to complete.

  * Missing Tables: Hive's default configuration causes the MetaStore
    to create schema elements only when they are needed. Some tables
    may be missing from your MetaStore schema if you have not created
    the corresponding Hive catalog objects, e.g. the PARTITIONS table
    will probably not exist if you have not created any table
    partitions in your MetaStore. You MUST create these missing tables
    before running the upgrade scripts. The easiest way to do this is
    by executing the official schema DDL script against your
    schema. Each of the CREATE TABLE statements in the schema script
    include an IF NOT EXISTS clause, so tables which already exist in
    your schema will be ignored, and those which don't exist will get
    created.

  * Extra Tables: Your schema may include a table named NUCLEUS_TABLES
    or a table named SEQUENCE_TABLE. These tables are managed
    by the DataNucleus ORM layer and will be created automatically if
    they don't exist. No action on your part is required.

  * Reversed Column Constraint Names in the Same Table: Tables with
    multiple constraints may have the names of the constraints
    reversed. For example, the PARTITIONS table contains two foreign
    key constraints named PARTITIONS_FK1 and PARTITIONS_FK2 which
    reference SDS.SD_ID and TBLS.TBL_ID respectively. However, in your
    schema you may find that PARTITIONS_FK1 references TBLS.TBL_ID and
    PARTITIONS_FK2 references SDS.SD_ID. Either version is acceptable
    -- the only requirement is that these constraints actually exist.

  * Differences in Column/Constraint Names: Your schema may contain
    tables with columns named "IDX" or unique keys named
    "UNIQUE<tab_name>". If you find either of these in your schema you
    will need to change the names to "INTEGER_IDX" and
    "UNIQUE_<tab_name>" before running the upgrade scripts. For more
    background on this issue please refer to HIVE-1435.

5) You are now ready to run the schema upgrade scripts. If you are
   upgrading from Hive 0.5.0 to Hive 0.6.0 you need to run the
   upgrade-0.5.0-to-0.6.0.mysql.sql script, but if you are upgrading
   from 0.5.0 to 0.7.0 you will need to run the 0.5.0 to 0.6.0 upgrade
   script followed by the 0.6.0 to 0.7.0 upgrade script.

   % mysql --verbose
   mysql> use <metastore_db_name>;
   Database changed
   mysql> source upgrade-0.5.0-to-0.6.0.mysql.sql
   mysql> source upgrade-0.6.0-to-0.7.0.mysql.sql

   These scripts should run to completion without any errors. If you
   do encounter errors you need to analyze the cause and attempt to
   trace it back to one of the preceding steps.

6) The final step of the upgrade process is validating your freshly
   upgraded schema against the official schema for your particular
   version of Hive. This is accomplished by repeating steps (3) and
   (4), but this time comparing against the official version of the
   upgraded schema, e.g. if you upgraded the schema to Hive 0.7.0 then
   you will want to compare your schema dump against the contents of
   hive-schema-0.7.0.mysql.sql

  直接說主要步驟:

       一,最好先停止我們的hive相關服務,禁止訪問hive的元數據 ,防止升級過程中其他應用使用修改數據。

       二,備份元數據

       三,開始升級,這里升級需要說明一下,hive提供的升級腳本是不能跨版本運行的,比如直接從1.2.2升級到2.3.4,所以我們需要根據它提供的腳本(腳本在scripts/metastore/upgrade/mysql),一個大版本,一個大版本升級

如我這次  需要按照如下順序執行

upgrade-1.2.0-to-2.0.0.mysql.sql
upgrade-2.0.0-to-2.1.0.mysql.sql
upgrade-2.1.0-to-2.2.0.mysql.sql
upgrade-2.2.0-to-2.3.0.mysql.sql

 

 

 

     執行完后,再打開我們的apache-hive-2.3.4-bin就可以了。


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM