碼上歡樂
相關內容    簡體    繁體

Oracle CDC簡介及異步在線日志CDC部署示例

本文轉載自   查看原文   2017-05-26 19:27   10019    Database/ Oracle CDC/ ETL/ Database Replication

摘要

最近由於工作需要，花時間研究了一下Oracle CDC功能和LogMiner工具，希望能找到一種穩定、高效的技術來實現Oracle增量數據抽取功能。以下是個人的部分學習總結和部署實踐。

1. Oracle CDC 簡介

很多人都認為，只要是涉及到數據庫數據復制和增量數據抽取，都是需要購買收費軟件的。實際上，我們通過Oracle提供的CDC和LogMiner等免費工具也能實現數據庫數據復制和增量數據抽取，各種數據復制軟件只是使得獲取增量數據更加便捷，或者是可以支持更多的擴展功能（例如：異構數據庫之間的同步，ETL過程的數據清洗、裝換），但實際Oracle本身是支持CDC機制，只是很少有人關注，操作起來也有些復雜，而且據傳言並不穩定，常常見到論壇上爆出一些莫名其妙的問題。
Oracle11gR2提供給我們以下幾種CDC機制：

1.1 Synchronous Change Data Capture Configuration（同步復制）

 原理很簡單，原表、目標表必須是同一個庫，采用觸發器的機制（設置同步CDC后，並看不到觸發器，但實際運行機理還是觸發器的機制）將原表內容復制到另一個目標表。這個機制就不多說了，和自己給表建觸發器沒什么太大差別。

1.2 Asynchronous HotLog Configuration（異步在線日志CDC）

這個過程已經沒有觸發器了，而是使用Redo Log，但是使用在線日志，並不是歸檔日志。並且原表、目標表仍然必須是同一個庫。這種模式是相對簡單的，同時這種模式是在Oracle 10以上才產生的，9i是沒有這個機制的。

1.3 Asynchronous Distributed HotLog Configuration（異步分布式CDC）

 實際這個模式是對異步在線日志CDC的一種優化，也比較容易理解，就是加入了DB-LINK機制，使原表、目標表不在同一個數據庫。實際是和異步在線日志CDC沒有什么本質區別。

1.4 Asynchronous Autolog Online Change Data Capture Configuration（異步在線日志復制CDC）

異步在線日志復制CDC模式就要高級很多了，使用Standby Redo Log（熱備數據庫日志），實際就是使用Oracle的熱備機制，將日志寫入了熱備數據庫，目標表就可以建立在熱備庫上，這對主數據庫性能影響就進一步降低。

1.5 Asynchronous AutoLog Archive Change Data Capture Configuration（歸檔日志CDC）

歸檔日志CDC模式是最完美的模式，但是需要有機制可以獲取歸檔日志（並行文件系統技術），然后在目標端分析歸檔日志進行變化數據處理，這種模式理論上來講，幾乎可以完全不影響原數據庫的性能。
坦白來說，我對Oracle理解並不深，只是為了解決特定的幾個問題多看了一點，在現實工作中遇到類似問題需要解決的，或對技術痴狂的同學可以研究一下，我貼上了4種模式具體的設置步驟，雖然是英文的，但是還是非常明確的。（我比較推薦使用第二種，因為設置比較簡單，性能上也屬於中規中矩，如果沒有什么特別要求，可以采用異步在線日志CDC。
以下是我對異步在線日志CDC環境的部署測試。

2. 異步在線日志CDC環境部署

2.1 環境配置准備

（1）確認數據庫版本
  1. SQL> select * from v$version;
  3. BANNER
  4. --------------------------------------------------------------------------------
  5. Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
  6. PL/SQL Release 11.2.0.1.0 - Production
  7. CORE 11.2.0.1.0 Production
  8. TNS for Linux: Version 11.2.0.1.0 - Production
  9. NLSRTL Version 11.2.0.1.0 - Production
（2）配置數據庫參數
  1. SQL> alter system set streams_pool_size=50m;
  3. System altered.
  5. SQL> alter system set java_pool_size=50m;
  7. System altered.
  9. SQL> alter system set undo_retention=3600;
  11. System altered.
  13. SQL> show parameter streams_pool
  15. NAME TYPE VALUE
  16. ------------------------------------ ----------- ------------------------------
  17. streams_pool_size big integer 52M
  18. SQL> show parameter java_pool
  20. NAME TYPE VALUE
  21. ------------------------------------ ----------- ------------------------------
  22. java_pool_size big integer 52M
  23. SQL> show parameter undo_re
  25. NAME TYPE VALUE
  26. ------------------------------------ ----------- ------------------------------
  27. undo_retention integer 3600
（3）開啟歸檔及補充日志
  1. SQL> archive log list
  2. Database log mode Archive Mode
  3. Automatic archival Enabled
  4. Archive destination USE_DB_RECOVERY_FILE_DEST
  5. Oldest online log sequence 487
  6. Next log sequence to archive 489
  7. Current log sequence 489
  8. SQL> alter database force logging;
  10. Database altered.
  12. SQL> alter database add supplemental log data;
  14. Database altered.
  16. SQL> select LOG_MODE,FORCE_LOGGING,SUPPLEMENTAL_LOG_DATA_MIN from v$database;
  18. LOG_MODE FOR SUPPLEME
  19. ------------ --- --------
  20. ARCHIVELOG YES YES
（4）准備測試表employee_info
  1. SQL> create table employee_info(n number,name varchar(20),address varchar(150),department varchar(120),organization varchar(150)) tablespace datafile1;
  3. Table created.
  5. SQL> insert into employee_info values(1, 'bendsha', 'lianhang road, shanghai, China', 'AnyBackup', 'EISOO');
  7. 1 row created.
  9. SQL> insert into employee_info values(2, 'bendsha', 'lianhang road, shanghai, China', 'AnyBackup', 'EISOO');
  11. 1 row created.

2.2 創建發布者和訂閱者

（1）創建發布者並授權
  1. SQL> create tablespace cdc_datafile datafile '/u01/app/oracle/orcl/cdc_datafile.dbf' size 1G;
  3. Tablespace created.
  5. SQL> create user cdc_publisher identified by cdc_publisher default tablespace cdc_datafile temporary tablespace temp;
  7. User created.
  9. SQL> grant create session TO cdc_publisher;
  11. Grant succeeded.
  13. SQL> grant create table TO cdc_publisher;
  15. Grant succeeded.
  17. SQL> grant create sequence TO cdc_publisher;
  19. Grant succeeded.
  21. SQL> grant create procedure TO cdc_publisher;
  23. Grant succeeded.
  25. SQL> grant create any job TO cdc_publisher;
  27. Grant succeeded.
  29. SQL> grant execute_catalog_role TO cdc_publisher;
  31. Grant succeeded.
  33. SQL> grant select_catalog_role TO cdc_publisher;
  35. Grant succeeded.
  37. SQL> grant execute ON dbms_cdc_publish TO cdc_publisher;
  39. Grant succeeded.
  41. SQL> grant execute ON dbms_lock TO cdc_publisher;
  43. Grant succeeded.
  45. SQL> grant unlimited tablespace TO cdc_publisher;
  47. Grant succeeded.
  49. SQL> execute dbms_streams_auth.grant_admin_privilege('CDC_PUBLISHER');
  51. PL/SQL procedure successfully completed.
  53. SQL> grant all on backupuser.employee_info to cdc_publisher;
  55. Grant succeeded.
（2）創建訂閱者並授權
  1. SQL> create user cdc_subscriber identified by cdc_subscriber default tablespace cdc_datafile temporary tablespace temp;
  3. User created.
  5. SQL> grant create session TO cdc_subscriber;
  7. Grant succeeded.

2.3 發布/訂閱具體數據

（1）發布：准備源表(Source Table)
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. BEGIN
  4. DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION(TABLE_NAME => 'backupuser.employee_info');
  5. END;
  6. Connected.
  7. SQL> 2 3 4 /
  9. PL/SQL procedure successfully completed.
（2）發布：創建變更集（Data Set）
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. BEGIN
  4. DBMS_CDC_PUBLISH.CREATE_CHANGE_SET(
  5. change_set_name => 'cdc_employee_info_cs',
  6. description => 'Change set for backupuser.employee_info info',
  7. change_source_name => 'HOTLOG_SOURCE',
  8. stop_on_ddl => 'y'
  9. );
  10. END;
  11. Connected.
  12. SQL> 2 3 4 5 6 7 8 9 /
  14. PL/SQL procedure successfully completed.
（3）發布：創建變更表（Change Table）
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. BEGIN
  4. DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE(
  5. owner => 'cdc_publisher',
  6. change_table_name => 'employee_info_ct',
  7. change_set_name => 'cdc_employee_info_cs',
  8. source_schema => 'backupuser',
  9. source_table => 'employee_info',
  10. column_type_list =>'n number,name varchar(20),address varchar(150)',
  11. capture_values => 'both',
  12. rs_id => 'y',
  13. row_id => 'n',
  14. user_id => 'n',
  15. timestamp => 'n',
  16. object_id => 'n',
  17. source_colmap => 'n',
  18. target_colmap => 'y',
  19. options_string =>'');
  20. END;
  21. 19 /
  23. PL/SQL procedure successfully completed.
（4）發布：激活變更集（Data Set）
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. BEGIN
  4. DBMS_CDC_PUBLISH.ALTER_CHANGE_SET(
  5. change_set_name => 'cdc_employee_info_cs',
  6. enable_capture => 'y');
  7. END;
  8. Connected.
  9. SQL> 2 3 4 5 6 /
  11. PL/SQL procedure successfully completed.
（5）授權給訂閱者
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. GRANT SELECT ON cdc_publisher.employee_info_ct TO cdc_subscriber;
  4. conn / as sysdba
  5. GRANT CREATE TABLE TO cdc_subscriber;
  6. GRANT CREATE SESSION TO cdc_subscriber;
  7. GRANT CREATE VIEW TO cdc_subscriber;
  8. GRANT UNLIMITED TABLESPACE TO cdc_subscriber;
  10. Grant succeeded.
（6）訂閱：創建訂閱集
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.CREATE_SUBSCRIPTION(
  5. change_set_name => 'cdc_employee_info_cs',
  6. description => 'Change data for employee_info',
  7. subscription_name => 'employee_info_sub');
  8. END;
  9. Connected.
  10. SQL> 2 3 4 5 6 7 /
  12. PL/SQL procedure successfully completed.
（7）訂閱：開始訂閱表信息
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.SUBSCRIBE(
  5. subscription_name => 'employee_info_sub',
  6. source_schema => 'backupuser',
  7. source_table => 'employee_info',
  8. column_list => 'n,name,address',
  9. subscriber_view => 'employee_info_view');
  10. END;
  11. Connected.
  12. SQL> 2 3 4 5 6 7 8 9 /
  14. PL/SQL procedure successfully completed.
（8）訂閱：激活訂閱
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.ACTIVATE_SUBSCRIPTION(
  5. subscription_name => 'employee_info_sub');
  6. END;
  7. Connected.
  8. SQL> 2 3 4 5 /
  10. PL/SQL procedure successfully completed.
（9）訂閱：擴展訂閱窗口
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.EXTEND_WINDOW(
  5. subscription_name => 'employee_info_sub');
  6. END;
  7. Connected.
  8. SQL> 2 3 4 5 /
  10. PL/SQL procedure successfully completed.
（10）訂閱：查看訂閱視圖內容
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. Connected.
  4. SQL> select * from employee_info_view;
  6. no rows selected

2.4 測試發布/訂閱

（1）源表employee_info變更
  1. SQL>
  2. conn backupuser/backupuser123
  3. insert into employee_info values(1, 'bendsha', 'lianhang road, shanghai, China', 'SmartData', 'SmartDB');
  4. insert into employee_info values(2, 'bendsha', 'lianhang road, shanghai, China', 'SmartData', 'SmartDB');
  5. insert into employee_info values(3, 'bendsha', 'lianhang road, shanghai, China', 'SmartData', 'SmartDB');
  6. update employee_info set name = 'zhuzi' where n = 2;
  7. delete from employee_info where n = 1;
  8. Connected.
  9. SQL>
  10. 1 row created.
  12. SQL>
  13. 1 row created.
  15. SQL>
  16. 1 row created.
  18. SQL>
  19. 1 row updated.
  21. SQL>
  22. 1 row deleted.
  24. SQL> commit;
  26. Commit complete.
（2）查看數據發布情況
  1. SQL>
  2. conn cdc_publisher/cdc_publisher
  3. Connected.
  4. SQL> select OPERATION$,n,name,address from employee_info_ct;
  6. OP N NAME
  7. -- ---------- --------------------
  8. ADDRESS
  9. --------------------------------------------------------------------------------
  10. I 1 bendsha
  11. lianhang road, shanghai, China
  13. I 2 bendsha
  14. lianhang road, shanghai, China
  16. I 3 bendsha
  17. lianhang road, shanghai, China
  20. OP N NAME
  21. -- ---------- --------------------
  22. ADDRESS
  23. --------------------------------------------------------------------------------
  24. UO 2 bendsha
  25. lianhang road, shanghai, China
  27. UN 2 zhuzi
  28. lianhang road, shanghai, China
  30. D 1 bendsha
  31. lianhang road, shanghai, China
  34. 6 rows selected.
（3）查看數據訂閱情況
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.EXTEND_WINDOW(
  5. subscription_name => 'employee_info_sub');
  6. END;
  7. Connected.
  8. SQL> 2 3 4 5 /
  10. PL/SQL procedure successfully completed.
  12. SQL> select OPERATION$,n,name,address from employee_info_view;
  14. OP N NAME
  15. -- ---------- --------------------
  16. ADDRESS
  17. --------------------------------------------------------------------------------
  18. I 1 bendsha
  19. lianhang road, shanghai, China
  21. I 2 bendsha
  22. lianhang road, shanghai, China
  24. I 3 bendsha
  25. lianhang road, shanghai, China
  28. OP N NAME
  29. -- ---------- --------------------
  30. ADDRESS
  31. --------------------------------------------------------------------------------
  32. UO 2 bendsha
  33. lianhang road, shanghai, China
  35. UN 2 zhuzi
  36. lianhang road, shanghai, China
  38. D 1 bendsha
  39. lianhang road, shanghai, China
  42. 6 rows selected.
（4）清除變更數據集
  1. SQL>
  2. conn cdc_subscriber/cdc_subscriber
  3. BEGIN
  4. DBMS_CDC_SUBSCRIBE.PURGE_WINDOW(
  5. subscription_name => 'employee_info_sub');
  6. END;
  7. Connected.
  8. SQL> 2 3 4 5 /
  10. PL/SQL procedure successfully completed.
  12. SQL> select OPERATION$,n,name,address from employee_info_view;
  14. no rows selected
（5）刪除發布數據
  1. SQL> conn cdc_publisher/cdc_publisher
  3. Connected.
  5. SQL> truncate table employee_info_ct;
  7. Table truncated.
  9. SQL> select OPERATION$,n,name,address from employee_info_ct;
  11. no rows selected

3. 常見問題解決方法

3.1 ORA-31466: 未找到發布內容

執行訂閱表信息時，提示ORA-31466：未找到發布內容，排查發現是沒有將變更表cdc_employee_info的查詢權限賦予訂閱者用戶cdc_subscriber導致。
解決方法：
  1. SQL> conn cdc_publisher/cdc_publisher
  2. Connected.
  3. SQL> grant select on cdc_employee_info to cdc_subscriber;
  5. Grant succeeded.

3.2 激活訂閱之后，對源表進行操作，捕獲不到數據

我一開始遇到這個問題是監控系統用戶SYS用戶的employee_info表，沒有出現任何異常，就是捕獲不到數據，后來我替換給backupuser用戶的employee_info表，按照以上步驟操作，就能正常捕獲到數據了，官網也沒查到相關的文檔說明，很奇怪，還需要進一步研究。

4. 參考文檔


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 Oracle CDC配置案例 SQLSERVER|CDC日志變更捕獲機制 【流數據處理】MySql/PG/Oracle+Kafka+Flink(CDC捕獲) 部署及實時計算 在SQL Server 2012中實現CDC for Oracle CDC spyglass Flink| CDC Flink CDC采集MySQL binlog日志實時寫入ClickHouse Oracle CDC (Change Data Capture)更新數據捕獲——概述 二、sqlserver開啟cdc功能 debezium關於cdc的使用(下)
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM