01、源庫安裝ogg
見OGG 安裝 博客
02、源庫配置oracle環境
環境說明：
| 主機 |
角色 |
Ogg版本 |
| 192.168.192.26 |
源庫 |
191004_fbo_ggs_Linux_x64_shiphome.zip |
| 192.168.193.221 |
目標庫中間件Kafka |
OGG_BigData_Linux_x64_19.1.0.0.5.zip |
01）、創建ogg同步用戶
SQL>create tablespace ogg datafile '/u02/oradata/ogg.dbf' size 5000M; SQL>create user ogg identified by "ogg" default tablespace ogg temporary tablespace TEMP;
02）、權限賦予
GRANT CONNECT TO ogg; GRANT ALTER ANY TABLE TO ogg; GRANT ALTER SESSION TO ogg; GRANT CREATE SESSION TO ogg; GRANT FLASHBACK ANY TABLE TO ogg; GRANT SELECT ANY DICTIONARY TO ogg; GRANT SELECT ANY TABLE TO ogg; GRANT RESOURCE TO ogg; GRANT SELECT ANY TRANSACTION TO ogg;
03）、歸檔模式開啟
已開啟則忽略
SQL>archive log list; SQL>shutdown immediate; SQL>startup mount; SQL>alter database archivelog; SQL>alter database open; SQL>alter system switch logfile;
04）、開啟oracle最小附加日志
Oracle 執行查詢語句，確保顯示結果為yes
Select force_logging,SUPPLEMENTAL_LOG_DATA_MIN, SUPPLEMENTAL_LOG_DATA_PK, SUPPLEMENTAL_LOG_DATA_UI, SUPPLEMENTAL_LOG_DATA_FK, SUPPLEMENTAL_LOG_DATA_ALL from v$database;
Min列不為YES則執行開啟操作：
alter database add supplemental log data ; alter system switch logfile;
ALL 列如果開啟則關閉
alter database drop supplemental log data (ALL) columns;
未成功則執行一下語句關閉：
alter database drop supplemental log data (primary key, unique,foreign key) columns; alter database drop supplemental log data ; alter system switch logfile;
05）、開啟強制日志模式
alter database force logging;
06）、環境變量配置
Vi /home/oracle/.bash_profile export OGG_HOME=/home/oracle/app/ogg/ export LD_LIBRARY_PATH=$OGG_HOME:$ORACLE_HOME/lib:/usr/bin
03、配置源庫ogg環境
01）、初始化ogg
當前ogg 默認需要安裝在oracle 用戶下操作。
Su – oracle . ~/.bash_profile Cd /home/oracle/app/ogg/ [oracle@hso32-db-test ogg]$ . ~/.bash_profile [oracle@hso32-db-test ogg]$ ./ggsci Oracle GoldenGate Command Interpreter for Oracle Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO Linux, x64, 64bit (optimized), Oracle 11g on Oct 17 2019 23:13:12 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (hso32-db-test) 1> create subdirs Creating subdirectories under current directory /home/oracle/app/ogg Parameter file /home/oracle/app/ogg/dirprm: already exists. Report file /home/oracle/app/ogg/dirrpt: already exists. Checkpoint file /home/oracle/app/ogg/dirchk: already exists. Process status files /home/oracle/app/ogg/dirpcs: already exists. SQL script files /home/oracle/app/ogg/dirsql: already exists. Database definitions files /home/oracle/app/ogg/dirdef: already exists. Extract data files /home/oracle/app/ogg/dirdat: already exists. Temporary files /home/oracle/app/ogg/dirtmp: already exists. Credential store files /home/oracle/app/ogg/dircrd: already exists. Masterkey wallet files /home/oracle/app/ogg/dirwlt: already exists. Dump files /home/oracle/app/ogg/dirdmp: already exists. GGSCI (hso32-db-test) 2> exit
當前環境配置成功
04、創建源庫測試表，測試數據
01) 、創建測試用戶，表
SQL> create user test_ogg identified by oracle ; User created. SQL> grant dba to test_ogg; Grant succeeded. SQL> conn test_ogg Enter password: Connected. SQL> create table test_ogg( id int,name varchar(20),primary key(id)); Table created.
02) 、配置Oracle 數據支持腳本
配置squence支持
在/home/oracle/app/ogg/下執行 Sqlplus / as sysdba @seqence.sql 鍵入schemas ：test_ogg
配置支持腳本
SQL> @marker_setup SQL> @ddl_setup SQL> @role_setup SQL> grant GGS_GGSUSER_ROLE to goldengate; SQL> @ddl_enable SQL> @marker_status.sql SQL> @?/rdbms/admin/dbmspool.sql SQL> @ddl_pin.sql ogg
05、配置源庫ogg 配置文件
01）、配置mgr進程
登入ogg
Cd /home/oracle/app/ogg/ ./ggsci GGSCI (hso32-db-test) 1> dblogin userid ogg password ogg; Successfully logged into database. GGSCI (hso32-db-test as ogg@hso32) 9> edit param mgr PORT 7809 DYNAMICPORTLIST 7840-7850 AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 3 PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
Ps:
PORT即mgr的默認監聽端口；
DYNAMICPORTLIST動態端口列表，當指定的mgr端口不可用時，會在這個端口列表中選擇一個，最大指定范圍為256個；
AUTORESTART重啟參數設置表示重啟所有EXTRACT進程，最多5次，每次間隔3分鍾；PURGEOLDEXTRACTS即TRAIL文件的定期清理
02）、配置ext抽取數據進程
GGSCI (hso32-db-test as ogg@hso32) 11> exit param extkafka EXTRACT extkafka dynamicresolution SETENV (ORACLE_SID="hso32") SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8") userid ogg,password ogg EXTTRAIL ./dirdat/to table test_kingle.oggest; table test_ogg.test_ogg;
ps：
第一行指定extract進程名稱；
dynamicresolution動態解析；
SETENV設置環境變量，這里分別設置了Oracle數據庫以及字符集；
userid ggs,password ggs即OGG連接Oracle數據庫的帳號密碼，使用上面創建的賬號；
exttrail定義trail文件的保存位置以及文件名，注意這里文件名只能是2個字母，其余部分OGG會補齊；
table即復制表的表名，支持*通配，必須以;結尾
03）、配置push 發送進程
GGSCI (hso32-db-test as ogg@hso32) 11> exit param pukafka EXTRACT pukafka passthru dynamicresolution userid ogg,password ogg RMTHOST 10.118.193.223, MGRPORT 7809 RMTTRAIL ./dirdat/to table test_kingle.oggest; table test_ogg.test_ogg;
ps，
第一行指定extract進程名稱；
passthru即禁止OGG與Oracle交互，我們這里使用pump邏輯傳輸，故禁止即可；dynamicresolution動態解析；
userid ogg,password ogg即OGG連接Oracle數據庫的帳號密碼rmthost和mgrhost即目標端(kafka)OGG的mgr服務的地址以及監聽端口；
rmttrail即目標端trail文件存儲位置以及名稱。
04）、加入需要監控的表（下面的所有操作都是基於本次操作進行）
下面包含 linux 同步到windwos 都是基於這個完成，添加監控表至關重要。
GGSCI (hso32-db-test as ogg@hso32) 21> add trandata test_ogg.test_ogg 2021-07-29 09:03:01 INFO OGG-15132 Logging of supplemental redo data enabled for table TEST_OGG.TEST_OGG. 2021-07-29 09:03:01 INFO OGG-15133 TRANDATA for scheduling columns has been added on table TEST_OGG.TEST_OGG. 2021-07-29 09:03:01 INFO OGG-15135 TRANDATA for instantiation CSN has been added on table TEST_OGG.TEST_OGG. GGSCI (hso32-db-test as ogg@hso32) 22> info trandata test_ogg.test_ogg Logging of supplemental redo log data is enabled for table TEST_OGG.TEST_OGG. Columns supplementally logged for table TEST_OGG.TEST_OGG: "ID". Prepared CSN for table TEST_OGG.TEST_OGG: 5982609614988
05）、配置defile 文件
Oracle 到其他數據屬於異構操作，需要定義映射關系
GGSCI (hso32-db-test as ogg@hso32) 21>edit param test_ogg defsfile ./dirdef/ test_ogg.test_ogg userid ogg,password ogg table test_ogg. test_ogg
生成完成后，選擇在ogg主目錄下，並且oracle用戶下執行
./defgen paramfile dirprm/test_ogg.prm
執行成功后沒有報錯的話，會在./dirdef下面生成一個文件，我們需要把這個文件拷貝到目標庫ogg主目錄下的diedef目錄下即可
06、目標庫221配置
01）、安裝 java
當前環境為綠色版，我直接解壓到/usr/local/java 目錄下即可，環境根據自己的配置
配置環境變量
Vi ~/.bash_porofile export ZOO_HOME=/root/ogg/zookeeper/ export KAFKA_HOME=/root/ogg/kafka/ export OGG_HOME=/root/ogg/ogg/ export RUN_AS_USER=root export JAVA_HOME=/usr/local/java/ export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin:$ZOO_HOME/bin/:$JAVA_HOME/jre/bin/:$KAFKA_HOME/bin export LD_LIBRARY_PATH=$OGG_HOME/lib/:/usr/bin:$OGG_HOME/:$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOMEjre/lib/amd64/libjsig.so:$JAVA_HOME/jre/lib/amd64/server/libjvm.so:/usr/lib
02）、安裝 配置zookeeper
Tar xf apache-zookeeper-3.6.3.tar.gz –C /root/ogg/ zookeeper Cd /root/ogg/ zookeeper
編輯配置文件
[root@RHEL44223 zookeeper]# cat conf/zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/root/ogg/zoo/data dataLogDir=/root/ogg/zoo/log clientPort=2181
啟動 zoo
Cd /root/ogg/zookeeper/bin ./zkServer.sh start
啟動成功即可
03）、安裝配置kafka
解壓kafka
Tar xf kafka_2.12-2.8.0.tgz –c /roo/ogg/kafka
配置kafka配置文件
Cd kafka Vi config/server.properties broker.id=0 listeners=PLAINTEXT://10.118.193.223:9092 host.name=10.118.193.223 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/root/ogg/zoo/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0
啟動 kafka
kafka-server-start.sh -daemon ./config/server.properties
04）、安裝ogg（root用戶即可）
解壓ogg
Tar xf OGG_BigData_Linux_x64_19.1.0.0.5.tar –C /root/ogg/ogg/
初始化ogg
Cd /root/ogg/ogg/ ./ggsci [oracle@RHEL44223 ogg]$ ./ggsci Oracle GoldenGate Command Interpreter for Oracle Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO Linux, x64, 64bit (optimized), Oracle 11g on Oct 17 2019 23:13:12 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. GGSCI RHEL44223) 1> create subdirs Creating subdirectories under current directory /home/oracle/app/ogg Parameter file /home/oracle/app/ogg/dirprm: already exists. Report file /home/oracle/app/ogg/dirrpt: already exists. Checkpoint file /home/oracle/app/ogg/dirchk: already exists. Process status files /home/oracle/app/ogg/dirpcs: already exists. SQL script files /home/oracle/app/ogg/dirsql: already exists. Database definitions files /home/oracle/app/ogg/dirdef: already exists. Extract data files /home/oracle/app/ogg/dirdat: already exists. Temporary files /home/oracle/app/ogg/dirtmp: already exists. Credential store files /home/oracle/app/ogg/dircrd: already exists. Masterkey wallet files /home/oracle/app/ogg/dirwlt: already exists. Dump files /home/oracle/app/ogg/dirdmp: already exists. GGSCI (RHEL44223) 2> exit
05) 配置ogg 配置文件
<01>、配置mgr
GGSCI (RHEL44223) 4> edit param mgr PORT 7809 DYNAMICPORTLIST 7840-7850 AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 3 PURGEOLDEXTRACTS /root/ogg/ogg/dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
<02>、配置 應用配置文件
GGSCI (RHEL44223) 5> edit param REKAFKA REPLICAT rekafka sourcedefs /root/ogg/ogg/dirdef/test_kingle.oggtest TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props REPORTCOUNT EVERY 1 MINUTES,RATE GROUPTRANSOPS 10000 MAP test_kingle.oggtest,TARGET test_kingle.oggtest; MAP test_ogg.test_ogg,TARGET test_ogg.test_ogg;
REPLICATE rekafka定義rep進程名稱；
sourcedefs即在4.6中在源服務器上做的表映射文件；
TARGETDB LIBFILE即定義kafka一些適配性的庫文件以及配置文件，配置文件位於OGG主目錄下的dirprm/kafka.props；
REPORTCOUNT即復制任務的報告生成頻率；
GROUPTRANSOPS為以事務傳輸時，事務合並的單位，減少IO操作；MAP即源端與目標端的映射關系
<03>、配置checkpoint
GGSCI (RHEL44223) 5>Edit param ./GLOBALS checkpointtable test_kingle.checkpoint
<04>、配置kafka 控制文件
Cd /root/ogg/ogg/dirprm [root@RHEL44223 dirprm]# cat rekafka.prm REPLICAT rekafka sourcedefs /root/ogg/ogg/dirdef/test_kingle.oggtest TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props REPORTCOUNT EVERY 1 MINUTES,RATE GROUPTRANSOPS 10000 MAP test_kingle.oggtest,TARGET test_kingle.oggtest; MAP test_ogg.test_ogg,TARGET test_ogg.test_ogg; [root@RHEL44223 dirprm]# cat custom_kafka_producer.properties bootstrap.servers=10.118.193.223:9092 acks=1 compression.type=gzip reconnect.backoff.ms=1000 value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer batch.size=102400 linger.ms=10000 [root@RHEL44223 dirprm]#、
06) 、創建kafka主題
查看主題有哪些 kafka-topics.sh --list --zookeeper localhost:2181 創建主題testogg kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testogg 再次查看是否存在 kafka-topics.sh --list --zookeeper localhost:2181
07、26同步221進程啟動配置
01）、源庫26 ogg 操作
通過上面編輯的配置文件添加進程，配置操作都是是使用的相對路徑，注意自己的環境是不是一直。
Cd /home/oracle/app/ogg/ GGSCI (hso32-db-test) 1> dblogin userid ogg password ogg; Successfully logged into database. GSCI (hso32-db-test as ogg@hso32) 41> add extract extkafka,tranlog,begin now EXTRACT added. GGSCI (hso32-db-test as ogg@hso32) 42> add exttrail ./dirdat/to,extract extkafka EXTTRAIL added. GGSCI (hso32-db-test as ogg@hso32) 43> edit param pukafka GGSCI (hso32-db-test as ogg@hso32) 44> add extract pukafka,exttrailsource ./dirdat/to EXTRACT added. GGSCI (hso32-db-test as ogg@hso32) 45> add rmttrail ./dirdat/to,extract pukafka RMTTRAIL added.
02)、目標庫221 添加進程
通過剛剛編輯的配置文件配置進程
GGSCI (RHEL44223) 5> add replicat rekafka exttrail ./dirdat/to,checkpointtable test_kingle.checkpoint
所有配置完成后，進程啟動
03）、啟動同步進程
進程啟動注意事項，一定要遵守啟動順序，方式獲取不到數據
啟動1：源庫mgr 啟動
Start mgr
啟動2：目標庫mgr 啟動
Start mgr
啟動3：源庫ext和pu 進程
Start extkafka
Start pukafka
啟動4：目標庫
Start REKAFKA
Info all 查看啟動進程狀態
等待啟動成功后查看kafka消息隊列
[root@RHEL44223 ogg]# kafka-console-consumer.sh --bootstrap-server 10.118.193.223:9092 --topic testogg --from-beginning {"table":"TEST_OGG.TEST_OGG","op_type":"I","op_ts":"2021-07-29 09:15:58.595966","current_ts":"2021-07-29T10:18:15.963000","pos":"00000000020000002057","after":{"ID":1,"NAME":"1"}} {"table":"TEST_OGG.TEST_OGG","op_type":"I","op_ts":"2021-07-29 09:15:58.595966","current_ts":"2021-07-29T10:18:16.295000","pos":"00000000020000002191","after":{"ID":2,"NAME":"1"}}
源庫模擬測試插入的時候看是否會有數據出來，如果沒有出現可以查看相應日志文件
Ogg 日志文件 在ogg目錄下ggserrot.log
或者命令行查看
View report rekafka
后面跟隨為進程名稱 可以通過info all 打印出來 group 列就是了。