實際運維中，由於OGG同步的數據，未使用DDL同步的情況下，並且DDL新增字段並不規范的情況下，導致數據不一致，本次實驗測試不規范的新增字段，會導致什么問題？
數據庫版本11.2.0.4， OGG 19.1.0.0.4
OGG測試源端新增字段,目標端並未及時同步字段，源端dml操作測試
OGG Version 19.1.0.0.4 Oracle 11.2.0.4 RAC 2節點。
測試3個表：分別為D1,D2,D3，用戶源端YZ,目標端BAK_YZ。
如下分別測試3中情況：
1.復制進程使用assumetargetdefs；
2.復制進程使用OGG 默認情況下；
3.復制進程使用指定的Def文件。
最后：建議生產環境正確的添加字段的步驟。
為了避免閱讀乏困，特在此文章前面進行總結：
1.oracle 19 ogg有優化,優先讓保障OGG進程不中斷，因此新增字段不規范的情況下，會導致新增字段的數據不會同步到ogg目標端，雖然數據不一致，但是OGG軟件同步鏈路正常；
*OGG11的情況下,在不同步DDL的情況下,源端新增一個字段，目標端未新增，此時源端對新增字段insert,目標端OGG進程會報錯！但是到了OGG 19版本不報錯，OGG優先保障進程正常同步，數據不一致自行處理！ 包括OGG 19目標端使用def文件的情況下，照樣會跳過新增字段的數據同步，OGG進程正常運行！
2.正確的步驟
源端關閉抽取進程；
執行DDL腳本；
DELETE,ADD TRANDATA OGG表；
目標端執行DDL腳本；
源端啟動EXT抽取進程，完成。
如果使用defgen
源端關閉抽取進程；
執行DDL腳本；
對變更的表，重新輸出表結構def文件；
DELETE,ADD TRANDATA OGG表；
目標端關閉復制進程；
執行DDL腳本；
根據源端def文件，更新或更換DEF文件
源端啟動EXT抽取進程，完成。
測試1.復制進程使用assumetargetdefs
測試1.復制進程使用assumetargetdefs 源端目標端均有此前同步的一個DD表的數據,復制使用他們的數據進行測試。 SQL> select * from dd; ID CC_NA WITTIME ---------- ----- ------------------------------ 2 2 03-JUN-20 02.34.37.000000 PM SQL> create table d1 as select * from dd; SQL> create table d2 as select * from dd; SQL> create table d3 as select * from dd; 添加最小補充日志 GGSCI (t1 as ogg@t1) 18> dblogin USERID ogg,PASSWORD ogg GGSCI (t1 as ogg@t1) 21> add trandata yz.d1 GGSCI (t1 as ogg@t1) 21> add trandata yz.d2 GGSCI (t1 as ogg@t1) 21> add trandata yz.d3 GGSCI (t1 as ogg@t1) 24> info trandata yz.d1 Logging of supplemental redo log data is enabled for table YZ.D1. Columns supplementally logged for table YZ.D1: "CC_NAME", "ID", "WITTIME". Prepared CSN for table YZ.D1: 2815050 源端測試OGG同步性 SQL> delete d1; SQL> delete d2; SQL> delete d3; SQL> commit; 目標端驗證[OK] 目標端如果沒有對應的表，OGG復制進程報錯，同步表過去就可以了。 2020-07-29 09:29:13 INFO OGG-06505 MAP resolved (entry YZ.D1): MAP "YZ"."D1",TARGET BAK_YZ.D1. 2020-07-29 09:29:13 WARNING OGG-00869 No data found when executing SQL statement <SELECT object_name FROM all_objects WHERE owner=:owner_name and object_name=upper(:object_name) and object_type in ('TABLE', 'VIEW')>. 源端新增字段 alter table d1 add status varchar2(10); 此時無論如何測試，OGG復制進程正常，為啥？ OGG源端 trandata只抽取3個列,因此本次測試需要排除次干擾。 源端OGG 抽取進程重啟，trandata重置。 GGSCI (t1 as ogg@t1) 32> stop exta GGSCI (t1 as ogg@t1) 33> delete trandata yz.d1 GGSCI (t1 as ogg@t1) 34> add trandata yz.d1 GGSCI (t1 as ogg@t1) 35> info trandata yz.d1 Logging of supplemental redo log data is enabled for table YZ.D1. Columns supplementally logged for table YZ.D1: "CC_NAME", "ID", "STATUS", "WITTIME". GGSCI (t1 as ogg@t1) 36> start exta 源端INSERT SQL> insert into d1 values(1,1,sysdate,'Y'); SQL> commit; 目標端驗證 OK 無報錯，只是忽略了STATUS列的信息！ SQL> select * from d1; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 09.37.23.000000 AM 源端UPDATE SQL> UPDATE D1 SET ID=2,STATUS='N' WHERE CC_NAME=1 AND STATUS='Y'; SQL> commit; 目標端驗證 OK 無報錯，只是忽略了STATUS列的信息！ SQL> select * from d1; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 2 1 29-JUL-20 09.37.23.000000 AM 源端DELETE 一樣 SQL> delete d1; SQL> commit; 目標端驗證 OK 無報錯 SQL> select * from d1; no rows selected 那么如果使用了assumetargetdefs 參數，同步表結構之后呢？ alter table d1 add status varchar2(10); 再次測試INSERT SQL> insert into d1 values(1,1,sysdate,'Y'); SQL> commit; 目標端驗證 status 無值！ SQL> select * from d1; ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 1 1 29-JUL-20 09.42.04.000000 AM GGSCI (t2) 20> info repa Log Read Checkpoint File /u01/ogg/base/dirdat/t1000000031 2020-07-29 09:42:06.493758 RBA 3889 [ogg@t2 base]$ ./logdump Logdump >open /u01/ogg/base/dirdat/t1000000031 Logdump >pos 0 Logdump >DETAIL data 省略若干個N 及事務下一個下一個，走到最后一個 Logdump >n 2020/07/29 09:42:06.493.758 Insert Len 62 RBA 3721 Name: YZ.D1 (TDR Index: 4) After Image: Partition x0c G s 0000 0500 0000 0100 3101 0005 0000 0001 0031 0200 | ........1........1.. 1f00 0000 3230 3230 2d30 372d 3239 3a30 393a 3432 | ....2020-07-29:09:42 3a30 342e 3030 3030 3030 3030 3003 0005 0000 0001 | :04.000000000....... 0059 | .Y Column 0 (x0000), Len 5 (x0005) 0000 0100 31 | ....1 Column 1 (x0001), Len 5 (x0005) 0000 0100 31 | ....1 Column 2 (x0002), Len 31 (x001f) 0000 3230 3230 2d30 372d 3239 3a30 393a 3432 3a30 | ..2020-07-29:09:42:0 342e 3030 3030 3030 3030 30 | 4.000000000 Column 3 (x0003), Len 5 (x0005) 0000 0100 59 | ....Y 可以發現存在Clolumn 0,1,2,3 四個列，數據對應1,1,2020-07-29:09:42:04.000000000,Y與插入數據相同！因此源端抽取投遞進程無異常！有問題的是復制進程！！！ 重啟復制進程 GGSCI (t2) 2> stop repa GGSCI (t2) 3> start repa 源端再次UPdate update d1 set cc_name=2 where status='Y'; SQL> commit; 復制進程終於ABEND了！！！ GGSCI (t2) 4> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING REPLICAT ABENDED REPA 00:00:00 00:00:22 2020-07-29 09:49:12 WARNING OGG-01004 Aborted grouped transaction on BAK_YZ.D1, Database error 1403 (No data found). [ogg@t2 base]$ tail -200f /u01/ogg/base/dirrpt/repa.dsc Oracle GoldenGate Delivery for Oracle process started, group REPA discard file opened: 2020-07-29 09:47:56.707549 Current time: 2020-07-29 09:49:12 Discarded record from action ABEND on error 1403 No data found Aborting transaction on /u01/ogg/base/dirdat/t1 beginning at seqno 31 rba 3,889 error at seqno 31 rba 3889 Problem replicating YZ.D1 to BAK_YZ.D1. Record not found Mapping problem with unified PK update record (target format) SCN:0.3.3.1505... * ID = 1 000000: 31 |1 | CC_NAME = 1 000000: 31 |1 | WITTIME = 2020-07-29 09:42:04.000000000 000000: 32 30 32 30 2d 30 37 2d 32 39 20 30 39 3a 34 32 |2020-07-29 09:42| 000010: 3a 30 34 2e 30 30 30 30 30 30 30 30 30 |:04.000000000 | STATUS = Y 000000: 59 |Y | ID = 1 000000: 31 |1 | CC_NAME = 2 000000: 32 |2 | WITTIME = 2020-07-29 09:42:04.000000000 000000: 32 30 32 30 2d 30 37 2d 32 39 20 30 39 3a 34 32 |2020-07-29 09:42| 000010: 3a 30 34 2e 30 30 30 30 30 30 30 30 30 |:04.000000000 | STATUS = Y 000000: 59 |Y | * Process Abending : 2020-07-29 09:49:12.260433 使用了HANDLECOLLISIONS 參數，但是帶來了副作用 源端 ID CC_NA WITTIME STATUS ---------- ----- ------------------------------ ---------- 1 2 29-JUL-20 09.42.04.000000 AM Y 目標端，數據已經和源端不一致了，update不存在的記錄，轉換為了insert操作！ ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 1 1 29-JUL-20 09.42.04.000000 AM 1 2 29-JUL-20 09.42.04.000000 AM Y 本次不測試HANDLECOLLISIONS，跳過測試其它。 注釋HANDLECOLLISIONS參數，重啟進程。 源端DELETE操作 delete d1 where status='Y'; commit; 目標端同步OK
測試2.復制進程不使用def參數,默認情況下；
源端新增字段 alter table d2 add status varchar2(10); 此時無論如何測試，OGG復制進程正常，為啥？ OGG源端 trandata只抽取3個列,因此本次測試需要排除次干擾。 源端OGG 抽取進程重啟，trandata重置。 GGSCI (t1 as ogg@t1) 32> stop exta GGSCI (t1 as ogg@t1) 33> delete trandata yz.d2 GGSCI (t1 as ogg@t1) 34> add trandata yz.d2 GGSCI (t1 as ogg@t1) 35> info trandata yz.d2 Columns supplementally logged for table YZ.D2: "CC_NAME", "ID", "STATUS", "WITTIME". GGSCI (t1 as ogg@t1) 36> start exta 源端INSERT SQL> insert into d2 values(1,1,sysdate,'Y'); SQL> commit; 目標端驗證 OK 無報錯，只是忽略了STATUS列的信息！ SQL> select * from d2; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 09.37.23.000000 AM 源端UPDATE SQL> UPDATE D2 SET ID=2,STATUS='N' WHERE CC_NAME=1 AND STATUS='Y'; SQL> commit; 目標端驗證 OK 無報錯，只是忽略了STATUS列的信息！ 並且update操作忽略了,並未修改。 SQL> select * from d2; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 09.37.23.000000 AM ？？？ ID=1並未同步修改操作 再次測試INSERT操作能否同步 SQL> insert into d2 values(2,2,sysdate,'N'); SQL> commit; 目標端驗證 SQL> select * from d2; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 2 2 29-JUL-20 10.02.43.000000 AM 2 1 29-JUL-20 10.00.05.000000 AM 源端DELETE SQL> DELETE D2 WHERE STATUS='N'; 2 rows deleted. SQL> COMMIT; Commit complete. SQL> select * from d2; no rows selected 目標端驗證OK SQL> select * from d2; no rows selected 添加字段 alter table d2 add status varchar2(10); 源端INSERT SQL> insert into d2 values(1,1,sysdate,'Y'); SQL> commit; 目標端驗證 OK SQL> select * from d2; ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 1 1 29-JUL-20 10.30.10.000000 AM Y 使用默認的情況，目標端同步源端表的DDL操作后,在進行Insert操作正常同步。 源端UPDATE UPdate d2 set id=6,status='N' where status='Y' and id=1; commit; 目標端驗證 SQL> select * from d2; ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 1 1 29-JUL-20 10.30.10.000000 AM Y SQL> r 1* select * from d2 ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 6 1 29-JUL-20 10.30.10.000000 AM N
測試3.復制進程使用指定def參數,默認情況下；
echo "defsfile ./dirdef/source_007.def,purge USERID ogg,PASSWORD ogg table YZ.B; table YZ.DD; table YZ.RANGE_PART_TAB; table YZ.SYS_EXPORT_SCHEMA_01; table YZ.TEST; table YZ.TESTA; table YZ.TESTC; table YZ.TEST_NEWLONG; table YZ.TT_PK; table YZ.TEST_LOB; TABLE YZ.D1; TABLE YZ.D2; TABLE YZ.D3;" >>./dirdef/source_007.prm ./defgen paramfile ./dirdef/source_007.prm 復制進程添加 sourcedefs /u01/ogg/base/dirdef/repa.def 同步測試 SQL> insert into d3 values(1,1,sysdate); commit; SQL> SELECT * FROM D3; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 11.02.34.000000 AM OK 此時源端新增一個字段！ SQL> alter table d3 add status varchar2(10); SQL> insert into d3 values(2,2,sysdate,'Y'); commit; 目標端,無影響。 ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 11.02.34.000000 AM 2 2 29-JUL-20 11.03.50.000000 AM GGSCI (t1) 3> dblogin USERID ogg,PASSWORD ogg GGSCI (t1 as ogg@t1) 4> delete trandata yz.d3 GGSCI (t1 as ogg@t1) 5> add trandata yz.d3 SQL> insert into d3 values(3,3,sysdate,'Y'); commit; SQL> SELECT * FROM D3; ID CC_NAME WITTIME ---------- ---------- ------------------------------ 1 1 29-JUL-20 11.02.34.000000 AM 2 2 29-JUL-20 11.03.50.000000 AM 3 3 29-JUL-20 11.06.08.000000 AM 不受影響！ 目標端同步表結構變更。 SQL> alter table d3 add status varchar2(10); 源端再次進行dml操作 SQL> insert into d3 values(4,4,sysdate,'Y'); SQL> SELECT * FROM D3; ID CC_NAME WITTIME STATUS ---------- ---------- ------------------------------ ---------- 1 1 29-JUL-20 11.02.34.000000 AM 2 2 29-JUL-20 11.03.50.000000 AM 3 3 29-JUL-20 11.06.08.000000 AM --重啟復制進程 SQL> insert into d3 values(5,5,sysdate,'N'); GGSCI (t2) 54> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING REPLICAT ABENDED REPA 00:00:00 00:00:25 2020-07-30 04:07:12 ERROR OGG-00918 Key column STATUS is missing from map. [ogg@t1 base]$ ./defgen paramfile ./dirdef/source_007.prm OVERRIDE 參數使用
正確做法
1.當使用assumetargetdefs時
驗證OGG同步 SQL> insert into d1 values(1,1,sysdate,'o'); OK 正確做法 源端停抽取進程 GGSCI (t1 as ogg@t1) 16> stop exta 源端新增表字段 SQL> alter table d1 add status1 varchar2(10); 源端重新delete,add trandata GGSCI (t1) 3> dblogin USERID ogg,PASSWORD ogg GGSCI (t1 as ogg@t1) 4> info trandata yz.d1 Columns supplementally logged for table YZ.D1: "CC_NAME", "ID", "STATUS", "WITTIME". GGSCI (t1 as ogg@t1) 5> delete trandata yz.d1 --測試delete trandata后，add之前插入數據，能否同步！ SQL> insert into d1 values(6,6,sysdate,'0','0'); SQL> commit; -- GGSCI (t1 as ogg@t1) 6> add trandata yz.d1 GGSCI (t1 as ogg@t1) 8> info trandata yz.d1 Logging of supplemental redo log data is enabled for table YZ.D1. Columns supplementally logged for table YZ.D1: "CC_NAME", "ID", "STATUS", "STATUS1", "WITTIME". 目標端新增OGG字段 SQL> alter table d1 add status1 varchar2(10); 源端啟動OGG抽取進程 GGSCI (t1 as ogg@t1) 24> start exta 驗證 目標端SQL> select * from d1 where id=6; ID CC_NAME WITTIME STATUS STATUS1 ---------- ---------- ------------------------------ ---------- ---------- 6 6 03-AUG-20 08.11.30.000000 AM 0 0 源端SQL> update d1 set status1=6 where status1='0'; SQL> commit; 目標端SQL> select * from d1 where id=6; ID CC_NAME WITTIME STATUS STATUS1 ---------- ---------- ------------------------------ ---------- ---------- 6 6 03-AUG-20 08.11.30.000000 AM 0 6
2.當使用DEF文件時
源端測試 SQL> insert into d3 values(1,2,sysdate,'o'); SQL> commit; ogg同步正常。 1.源端停抽取進程 GGSCI (t1) 14> info exta,detail 【關閉，啟動抽取進程，會自動切換seq文件】 Seqno 40 RBA 2207 GGSCI (t1) 15> stop exta 添加字段 SQL> alter table d3 add status1 varchar2(10); 重新配置DEF文件 [ogg@t1 base]$ vi ./dirdef/source_007.prm defsfile ./dirdef/source_007.def,purge USERID ogg,PASSWORD ogg table YZ.D1; [ogg@t1 base]$ ./defgen paramfile ./dirdef/source_007.prm [ogg@t1 base]$ cat ./dirdef/source_007.def * Definition for table YZ.D1 Record length: 310 Syskey: 0 Columns: 4 ID 64 50 0 0 0 1 0 50 50 50 0 0 0 0 1 0 1 2 2 -1 0 0 0 CC_NAME 64 200 56 0 0 1 0 200 200 0 0 0 0 0 1 0 1 0 1 -1 0 0 0 WITTIME 192 29 262 0 0 1 0 29 29 29 0 6 0 0 1 0 1 0 187 -1 0 0 0 STATUS 64 10 294 0 0 1 0 10 10 0 0 0 0 0 1 0 1 0 1 -1 0 0 0 End of definition 源端對表進行delete,add trandata GGSCI (t1) 3> dblogin USERID ogg,PASSWORD ogg GGSCI (t1 as ogg@t1) 4> info trandata yz.d3 Columns supplementally logged for table YZ.D3: "CC_NAME", "ID", "STATUS", "WITTIME". GGSCI (t1 as ogg@t1) 5> delete trandata yz.d3 --測試delete trandata后，add之前插入數據，能否同步！ SQL> insert into d3 values(6,6,sysdate,'0','0'); SQL> commit; -- GGSCI (t1 as ogg@t1) 6> add trandata yz.d3 GGSCI (t1 as ogg@t1) 8> info trandata yz.d3 Logging of supplemental redo log data is enabled for table YZ.D3. Columns supplementally logged for table YZ.D3: "CC_NAME", "ID", "STATUS", "STATUS1", "WITTIME". 目標端更新def文件,肯定也需要關閉復制進程。 GGSCI (t2) 31> stop repa [ogg@t2 base]$ vi /u01/ogg/base/dirdef/repa.def 替換上述獲得的最新def文件。 OGG 列手工同步 SQL> alter table d3 add status1 varchar2(10); 啟動復制進程 GGSCI (t2) 2> start repa 源端啟動抽取進程 GGSCI (t1 as ogg@t1) 13> start exta 驗證？ OK SQL> select * from d3 where id=6; ID CC_NAME WITTIME STATUS STATUS1 ---------- ---------- ------------------------------ ---------- ---------- 6 6 03-AUG-20 08.03.50.000000 AM 0 0