最近有個省級項目需要遷移大批量數據(大於2T;主要的表均為分區表,並且有二級分區,單表有129個分區,7-8個二級分區)並且遷移過程中需要轉換字符集,從GBK轉換到UTF-8。而且,涉及到新項目功能上線,遷移時間(停機時間)需要盡快能的短。數據字符集轉換和導入時間盡量控制在10H以內。
基於以上考慮,數據遷移和導入分一下幾步操作:
1.從現有的測試庫上導出表結構數據,導入到目標庫,除去索引和約束;沒有索引和約束的額外開銷,單純導數據會很快。
2.現有生產庫上數據導出,盡可能采用高並發;考慮到新舊服務器CPU核數較多,准備采用parallel為32以上;
3.數據分兩步導入目標庫:
3.1 第一步純數據導入,即content=data_only;
3.2 第二步追加索引和約束,即include=index,constraint
4.核對數據,表分析。
以上主要步驟1和3分別驗證如下:
impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 exclude=index,constraint
不會導入索引和約束
impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics
不會導入索引和約束;經過驗證:TRIGGER,TABLE_STATISTICS 包含在metadata里面,建議去除trigger,statistics
impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only
索引和約束也會導入
impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only
僅導入表數據,不含index,constraint,trigger,statistics
impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics
追加index,constraint,trigger,statistics
實驗操作過程記錄如下:
[ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics
Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:52:03 2021
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 content=metadata_only exclude=index,constraint,trigger,statistics
Processing object type TABLE_EXPORT/TABLE/TABLE
Job "SYSTEM"."20210311_ahxnb_ad71" successfully completed at Thu Mar 11 20:52:08 2021 elapsed 0 00:00:05
[ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only
Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:52:36 2021
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71:AAB301_341182_P3 content=data_only
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
. . imported "AHXNB"."AD71":"AAB301_341182"."AAB301_341182_P3" 11.20 MB 1334358 rows
Job "SYSTEM"."20210311_ahxnb_ad71" successfully completed at Thu Mar 11 20:52:43 2021 elapsed 0 00:00:07
[ora11g@dbsrv1 ~]$ impdp system/****** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics
Import: Release 11.2.0.4.0 - Production on Thu Mar 11 20:54:05 2021
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Master table "SYSTEM"."20210311_ahxnb_ad71" successfully loaded/unloaded
Starting "SYSTEM"."20210311_ahxnb_ad71": system/******** job_name=20210311_ahxnb_ad71 directory=DPDATA dumpfile=ahxnb_20210309_%U.dmp logfile=20210311_ahxnb_impdp_ad71_idx.log PARALLEL=6 remap_schema=AHXNB:AHXNB remap_tablespace=JHK_DATA:XNB_SYSTEM,USERS:XNB_SYSTEM tables=ahxnb.AD71 include=index,constraint,trigger,statistics
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/TRIGGER
ORA-39082: Object type TRIGGER:"AHXNB"."SBTBI_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTBI_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAI_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAI_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAU_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAU_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAD_AD71" created with compilation warnings
ORA-39082: Object type TRIGGER:"AHXNB"."SBTAD_AD71" created with compilation warnings
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Job "SYSTEM"."20210311_ahxnb_ad71" completed with 8 error(s) at Thu Mar 11 20:54:40 2021 elapsed 0 00:00:34
后續:
采用該辦法后,data_only導入大約3小時,追加索引和約束大約10小時。
遇到的問題1:
ORA-31693: Table data object "AHXNB"."AC63":"AAB301_341602"."AAB301_341602_P2014" failed to load/unload and is being skipped due to error:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-39077: unable to subscribe agent ORA-39077: unable to subscribe agent KUPC$A_1_163050148848000 to queue "KUPC$C_1_20210312150031"
ORA-24067: exceeded maximum number of subscribers for queue SYS.KUPC$C_1_20210312150031
后續發現該問題跟impdp導入過程中parallel參數有關系(過大?),后面通過配置parallel=1重新導入前面因出錯未導入的數據。懷疑ORA-24067是一個BUG,留待后續繼續觀察測試。
遇到的問題2:
在解決問題1,補數據的時候遇到:
ORA-31693: Table data object "AHXNB"."AC63":"AAB301_341602"."AAB301_341602_P2014" failed to load/unload and is being skipped due to error:
ORA-04098: trigger 'AHXNB.SBTBI_AC63' is invalid and failed re-validation
將該觸發器disable后,數據順利導入。后續重新enable該觸發器。
