OGG(Oracle GoldenGate)參數介紹
所有的GoldenGate進程均有參數文件 Manager Extract Replicat Utilities 所有參數均有缺省配置 實際應用只需對小部分參數進行配置 所有參數文件均放在 ./dirprm目錄下 缺省通過進程名進行查找
一、全局參數
MGRSERVNAME Specifies the name of the Manager process when it is installed as a Windows service. CHECKPOINTTABLE Specifies a default checkpoint table. GGSCHEMA Specifies the name of the schema that contains the database objects that support DDL synchronization for Oracle. DDLTABLE Specifies a non-default name for the DDL history table that supports DDL synchronization for Oracle. MARKERTABLE Specifies a non-default name for the DDL marker table that supports DDL synchronization for Oracle. OUTPUTFILEUMASK Specifies a umask that can be used by Oracle GoldenGate processes to create trail files and discard files. SYSLOG Filters the types of Oracle GoldenGate messages that are written to the system logs. UNLOCKEDTRAILFILES Workaround for problems with trail files that remain locked by Collector if there is a network outage that affects the target system.
二、Manager進程相關參數
1、常用參數:
PORT 制定GoldenGate的mgr進程使用哪個 TCP/IP端口偵聽請求. 示例:PORT 7809 DYNAMICPORTLIST 指定GoldenGate可以使用那些端口接受extract發送過來的數據. 示例:DYNAMICPORTLIST 7840-7850 AUTOSTART 指定在mgr啟動時自動啟動那些進程. 示例:AUTOSTART ER * AUTOSTART extract extsz AUTORESTART 指定在mgr可以定時重啟那些進程。可以在網絡中斷等故障恢復后自動重起,避免人工干預. 示例:AUTORESTART ER *, WAITMINUTES 5, RETRIES 3 --每隔5分鍾嘗試啟動一次,嘗試3次 LAGREPORT 指定在ggserr.log中報告延遲的時間間隔. 示例:LAGREPORTHOURS 1 --每隔一小時檢查一次傳輸延遲情況 LAGCRITICAL 指定認為超過此時間即為嚴重錯誤的延遲最大值,如果延遲達到此時間值則會在ggserr.log里面寫入一條error信息 示例:LAGCRITICALMINUTES 45 --傳輸延時超過45分鍾將寫入警告日志 LAGINFO 指定一個延遲時限,如果延遲達到此時間值則會在ggserr.log里面寫入一條info信息 示例:LAGINFOMINUTES 30 --傳輸延時超過30分鍾將寫入錯誤日志 PURGEOLDEXTRACTS 定義自動刪除過時的隊列以節省硬盤空間。
一般按照兩個規則來刪除:首先,要滿足檢查點要求,沒有使用過的隊列不能刪除,保證無數據丟失;其次,可以保留一定的天數。只有當已經使用過且超過設定的天數后的隊列會被自動刪除。示例:purgeoldextracts /backup/goldengate/dirdat/*,usecheckpoints, minkeepdays 7
2、完整示例:
源端Manager配置參數示例:
GGSCI> view params mgr PORT 7839 --通信端口7839。 源端和目標端需要保持一致。 DYNAMICPORTLIST 7840-7914 --動態端口列表的范圍從7840到7914。當制定端口被占用或者出現通信故障,管理進程將會從列表中選擇下一個端口嘗試連接,避免通信端口的單點故障。 AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 7 --當提取進程中斷后嘗試自動重啟,每隔7分鍾嘗試啟動一次,嘗試5次。 PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 10 --定期清理dirdat路徑下的本地隊列(local trail)。保留期限10天,過期后自動刪除。從而控制隊列文件的目錄不會增長過大。 LAGREPORTHOURS 1 --每隔一小時檢查一次傳輸延遲情況 LAGINFOMINUTES 30 --傳輸延時超過30分鍾將寫入錯誤日志 LAGCRITICALMINUTES 45 --傳輸延時超過45分鍾將寫入警告日志
目標端Manager配置參數示例:
GGSCI 4> view params mgr PORT 7839 --通信端口7839。 生產端和災備端需要保持一致 USERID goldengate, PASSWORD YGGJBFJGIH, ENCRYPTKEY default DYNAMICPORTLIST 7840-7914 --動態端口列表的范圍從7840到7914。當制定端口被占用或者出現通信故障,管理進程將會從列表中選擇下一個端口嘗試連接,避免通信端口的單點故障。 AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3 --當提取進程中斷后嘗試自動重啟,每隔7分鍾嘗試啟動一次,嘗試5次。 PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints, minkeepdays 5 --定期清理dirdat路徑下的遠程隊列(remote trail)。保留期限5天,過期后自動刪除。從而控制隊列文件的目錄不會增長過大。 --PURGEDDLHISTORY MINKEEPDAYS 7, MAXKEEPDAYS 10 --刪除DDL歷史表,最小保存7天,最大保存10天。由於此項目沒有開啟DDL,所以該參數被注釋。 --PURGEMARKERHISTORY MINKEEPDAYS 7, MAXKEEPDAYS 10 --刪除MARKER歷史表,最小保存7天,最大保存10天。此項目沒有使用該參數,所以該參數被注釋。 LAGREPORTHOURS 1 --每隔一小時檢查一次傳輸延遲情況 LAGINFOMINUTES 30 --傳輸延時超過30分鍾將寫入錯誤日志 LAGCRITICALMINUTES 45 --傳輸延時超過45分鍾將寫入警告日志
參考:真實環境的OracleGoldenGate配置參數詳解 -- http://blog.itpub.net/29475508/viewspace-1297083/
3、完整可選參數列表
(1)通用參數:
COMMENT | -- Allows insertion of comments in a parameter file.##允許在參數文件中插入注釋信息 SYSLOG Filters the types of Oracle GoldenGate messages that are written to the system logs.
(2)端口管理參數:
DYNAMICPORTLIST Specifies the ports that Manager can dynamically allocate. ##指定Manager可以動態分配的端口列表(這些端口將用來接受extract發送過來的數據) 示例:DYNAMICPORTLIST 7840-7850 DYNAMICPORTREASSIGNDELAY Specifies a time to wait before assigning a port number that was previously assigned. ##當要分配的端口被占用時,指定在再次分配前需要等待的時間 PORT Establishes the TCP/IP port number on which Manager listens for requests. ##指定Manager監聽請求所使用的TCP/IP端口號,示例:PORT 7809
(3)進程管理參數:
AUTORESTART Specifies processes to be restarted by Manager after a failure. ##指定Manager可(定時)重啟哪些進程(可以在網絡中斷等故障恢復后自動重起,避免人工干預) 示例:AUTORESTART ER *, WAITMINUTES 5, RETRIES 3 --每隔5分鍾嘗試啟動一次,嘗試3次 AUTOSTART Specifies processes to be started when Manager starts. ##指定Manager進程啟動時,自動啟動哪些進程 示例:AUTOSTART ER * AUTOSTART extract extsz BOOTDELAYMINUTES Determines how long after system boot time Manager delays until performing main processing activities. This parameter supports Windows. UPREPORT Determines how often process heartbeat messages are reported. ##確定心跳檢測信息多久報告一次
(4)事件管理參數:
DOWNCRITICAL Reports processes that stopped gracefully or abnormally. DOWNREPORT Controls the frequency for reporting stopped processes. LAGCRITICAL Specifies a lag threshold that is considered critical and generates a warning to the error log. 指定認為超過此時間即為嚴重錯誤的延遲最大值,如果延遲達到此時間值則會在ggserr.log里面寫入一條error信息 示例:LAGCRITICALMINUTES 45 --傳輸延時超過45分鍾將寫入警告日志 LAGINFO Specifies a lag threshold at which an informational message is reported to the error log. 指定一個延遲時限,如果延遲達到此時間值則會在ggserr.log里面寫入一條info信息 示例:LAGINFOMINUTES 30 --傳輸延時超過30分鍾將寫入錯誤日志 LAGREPORT Sets an interval for reporting lag time to the error log 指定在ggserr.log中報告延遲的時間間隔. 示例:LAGREPORTHOURS 1 --每隔一小時檢查一次傳輸延遲情況
(5)數據庫登陸:
SOURCEDB Specifies a data source name as part of the login information. 示例:SOURCEDB mydb SOURCEDB mydb, USERID ggs, PASSWORD ggs123 USERID Provides login information for Manager when it needs to access the database. 示例:USERID / USERID ggs, PASSWORD ggs123 USERID ggs@ora1.ora, PASSWORD ggs123 USERID data1 USERID ggs, PASSWORD AACAAAAAAAAAAAIALCKDZIRHOJBHOJUH, ENCRYPTKEY superx128 USERID ggs, PASSWORD AACAAAAAAAAAAAIALCKDZIRHOJBHOJUH, ENCRYPTKEY default
(6)維護相關參數:
CHECKMINUTES Determines how often Manager cycles through maintenance activities. PURGEDDLHISTORY Purges rows from the Oracle DDL history table when they are no longer needed. PURGEMARKERHISTORY Purges Oracle marker table rows that are no longer needed. PURGEOLDEXTRACTS Purges trail data that is no longer needed.##自動刪除過時的隊列以節省硬盤空間。 一般按照兩個規則來刪除: 首先,要滿足檢查點要求,沒有使用過的隊列不能刪除,保證無數據丟失; 其次,可以保留一定的天數。只有當已經使用過且超過設定的天數后的隊列會被自動刪除。 示例:purgeoldextracts /backup/goldengate/dirdat/*,usecheckpoints, minkeepdays 7 PURGEOLDTASKS Purges Extract and Replicat tasks after a specified period of time. STARTUPVALIDATIONDELAY[CSECS] Sets a delay time after which Manager checks that processes are still running after startup.
參考:Oracle GoldenGate Windows and UNIX Reference Guide 11g Release 1(11.1.1) E17791-01
Oracle Golden Gate 系列八 -- GG 參數文件 說明 http://blog.csdn.net/tianlesoftware/article/details/6970183
二、extract進程相關參數
1、常用參數
抽取進程有以下方面參數:
Checkpoint 復制目標 Local System Multiple Remote Systems One to many GoldenGate Trails 數據源 Which Tables Which Rows and Columns Which Operations 數據影射和變換
table參數:
Syntax: TABLE <file name> [, WHERE (<where condition>)] [, FILTER (<expression>)] [, KEYCOLS (<key column specification>)] [, COLS (<column>) | [ , <column> ]) ] [, COLSEXCEPT (<column>) | [ , <column> ]) ] [, FETCHCOLS (<column>) | [ , <column> ]) ] [, FETCHCOLSEXCEPT (<column>) | [ , <column> ]) ] [, SQLEXEC <clause> ] [, FETCHBEFOREFILTER ] [, SQLPREDICATE “WHERE <clause>” ] [, COLMAP (<column mapping specification>)] [, <record type filter>, <record type filter>, …] ;
注意:TABLE的結尾有一個分號。
2、完整示例
參數文件舉例:日志抽取進程
extract extsz userid goldengate, password ******* --REPORT AT 01:59 --reportrollover at 02:00 --transmemory directory (/backup/goldengate/dirtmp,8G,4G),ram 2G,transram 500M tranlogoptions rawdeviceoffset 0 --warnlongtrans 12h, checkintervals 30m exttrail /backup/goldengate/dirdat/sz dynamicresolution numfiles 2000 tableexclude ctais2.TMP_*; tableexclude ctais2.BAK_*; tableexclude ctais2.MLOG$_*; tableexclude ctais2.RUPD$_*; tableexclude ctais2.KJ_*; table ctais2.*;
參數文件舉例:傳輸進程
extract dpesz setenv ( NLS_LANG = AMERICAN_AMERICA.ZHS16GBK ) passthru REPORT AT 01:59 reportrollover at 02:00 transmemory directory (/backup/goldengate/dirtmp,8G,4G), ram 2G,transram 500M --threadoptions iolatencydelay 3000 rmthost 99.16.1.12,mgrport 7809, compress rmttrail /oradata/goldengate/dirdat/sz dynamicresolution numfiles 3000 table ctais2.*;
示例3:
extract extl --抽取進程名 userid ddw,password ddw --連接本機DB的帳號密碼 rmthost 192.168.0.44, mgrport 7801 --目標數據庫服務器地址和GG服務端口號 rmttrail d:\tools\GG\gg10g\dirdat\rl --遠程隊列的位置 dynamicresolution --優化參數,動態分析表結構 gettruncates --抓取truncate數據 table ddw.aatest; --需要抽取的表,可以使用通配符
3、全部可選參數
(1)通用參數
CHECKPARAMS Verifies parameter file syntax. 如果加入該參數,表示下次運行只是檢查一下語法,並不實際運行進程。 注意:該命令只能檢查一些簡單語法,並不能保證參數文件是完全正確的。 COMMENT | -- Denotes comments in a parameter file. ##注釋行,也可以用兩個中划線--代替.--checkparams表示本行已經被注釋掉 ETOLDFORMAT Generates trails in a format that is compatible with Replicat versions prior to Oracle GoldenGate version 6.0. GETENV Retrieves variables that were set with the SETENV parameter. SETENV Specifies a value for a UNIX environment variable from within the GGSCI interface. 針對extract進程設定系統環境變量。 示例:SETENV (NLS_LANG="AMERICAN_AMERICA.UTF8") --設置字符集環境變量為UTF8
OBEY Processes parameter statements contained in a different parameter file. 可以將外部文件的內容包含到本參數文件中來。用於將一些可以重復利用的參數文件部分內容隔離出來,便於統一修改。Obey tables.txt RECOVERYOPTIONS Controls the recovery mode of the Extract process. TCPSOURCETIMER |
NOTCPSOURCETIMER Adjusts timestamps of records transferred to other systems when those systems reflect different times. TRACETABLE |
NOTRACETABLE Causes Extract to ignore database changes generated by Replicat during bidirectional synchronization. Supports Oracle
(2)處理方法Processing method
BEGIN Specifies when to begin a processing run. DSOPTIONS Specifies Extract processing options when a Teradata Access Module (TAM) is being used. END Specifies when to end a processing run. EXTRACT Defines an Extract group as an online process. ##定義抽取進程的名字 GETAPPLOPS |
IGNOREAPPLOPS Controls whether or not operations from all processes except Replicat are written to a trail or file. GETREPLICATES |
IGNOREREPLICATES Controls whether or not replicated operations are captured by an Extract on the same system.
PASSTHRU | Controls whether tables will be processed by a data-pump Extract in pass-through mode or whether data definitions will be required NOPASSTHRU 禁止extract與數據庫交互,適用於Data Pump傳輸進程(dpeXX)。
說明:PASSTHRU直通模式用在兩邊表名、列名一致,可以直接映射的情況,不需要額外配置;
NOPASSTHRU普通模式可以配置表名列名自定義映射,可以加FILTER、transformation等,需要配置一個數據定義文件(data-definitions file)
RMTTASK Creates a processing task on a remote system. SOURCEISTABLE Extracts entire records from source tables. SPECIALRUN Specifies a one-time processing task that does not checkpoint from run to run. VAM Indicates that a Teradata Access Module (TAM) is being used to provide transactional data to the Extract process.
(3)數據庫登陸Database login
SOURCEDB Specifies the data source as part of the login information. USERID Specifies database connection information. [SOURCEDB,] USERID ,PASSWORD 指定所要登陸的數據庫名稱,用戶名和密碼。對於oracle無需指定sourcedb,直接指定用戶名和密碼即可。 Userid goldengate, password goldengate
(4)數據匹配和映射 Selecting and mapping data
ASCIITOEBCDIC Converts ASCII text to EBCDIC for DB2 on z/OS systems running UNIX System Services. COLMATCH Establishes global column-mapping rules. COMPRESSDELETES |
NOCOMPRESSDELETES Controls whether Oracle GoldenGate writes only the key or all columns to the trail for delete operations. COMPRESSUPDATES |
NOCOMPRESSUPDATES Causes only primary key columns and changed columns to be logged for updates. DDL Enables and filters the capture of DDL operations. DDLSUBST Enables string substitution in DDL processing. FETCHOPTIONS Controls certain aspects of the way that Oracle GoldenGate fetches data. 示例:FETCHOPTIONS NOUSESNAPSHOT --不會從閃回日志中獲取數據 FETCHOPTIONS FETCHPKUPDATECOLS --當使用了 HANDLECOLLISIONS 時,請使用該參數。復制進程出現丟失update記錄(missing update)並且更新的是主鍵,update將轉換成insert。由於插入的記錄可能不是完整的行,若要保證完整需要加入此參數 GETDELETES | IGNOREDELETES Controls the extraction of delete operations.是否復制delete操作,缺省復制 GETINSERTS | IGNOREINSERTS Controls the extraction of insert operations.是否復制insert操作,缺省復制 GETTRUNCATES |IGNORETRUNCATES Controls the extraction of truncate statements. 是否復制truncate操作,缺省不復制. GETUPDATEAFTERS | IGNOREUPDATEAFTERS Controls the extraction of update after images.##是否在隊列中寫入后影像,缺省復制 GETUPDATEBEFORES |IGNOREUPDATEBEFORES Controls the extraction of update before images.是否在隊列中寫入前影像,缺省不復制 GETUPDATES | IGNOREUPDATES Controls the extraction of update operations 是否復制update操作,缺省復制 REPLACEBADCHAR Replaces invalid character values with another value. SEQUENCE Specifies sequences for synchronization. TABLE for Extract Specifies tables for extraction and controls column mapping and conversion. TABLEEXCLUDE Excludes tables from the extraction process. 定義所需要排除的表。如果在table里面定義了使用通配符,那么可以使用該參數定義排除掉其中的部分表。如: tableexclude ctais2.TMP_*; tableexclude ctais2.TEMPTAB; TARGETDEFS Specifies a file containing target table definitions for target databases that reside on the NonStop platform. TRIMSPACES |
NOTRIMSPACES Controls whether trailing spaces are trimmed or not when mapping CHAR to VARCHAR columns. WILDCARDRESOLVE Defines rules for processing wildcard table specifications in a TABLE statement.
(5)數據轉發 Routing data
EXTFILE Specifies an extract file to which extracted data is written on the local system. EXTTRAIL Specifies a trail to which extracted data is written on the local system. 指定寫入到本地的哪個隊列。 示例:EXTTRAIL ./dirdat/ya --本地隊列文件路徑 RMTFILE Specifies an extract file to which extracted data is written on a remote system. RMTHOST Specifies the target system and Manager port number. 指定目標系統及其GoldenGate Manager進程的端口號,也用於定義是否使用壓縮進行傳輸。 示例:rmthost 99.16.1.12,mgrport 7809, compress RMTTRAIL Specifies a trail to which extracted data is written on a remote system. 指定寫入到目標端的哪個隊列
(6)數據格式化Formatting data
FORMATASCII Formats extracted data in external ASCII format. FORMATSQL Formats extracted data into equivalent SQL statements. FORMATXML Formats extracted data into equivalent XML syntax. NOHEADERS Prevents record headers from being written to the trail.
(7)自定義處理 Custom processing
CUSEREXIT Invokes a user exit routine during processing. INCLUDE Invokes a macro library. MACRO Defines an Oracle GoldenGate macro. MACROCHAR Defines a macro character other than the default of #. SQLEXEC Executes a stored procedure or query during Extract processing. 在extract運行時首先運行一個sql語句。 示例:sqlexec "Alter session set constraints=deferred" VARWIDTHNCHAR |
NOVARWIDTHNCHAR Controls whether length information is written to the trail for NCHAR columns.
(8)報告信息Reporting
CMDTRACE Displays macro expansion steps in the Extract report file. LIST | NOLIST Displays or suppresses the listing of macros in the report file. REPORT Schedules a statistical report. 定義自動定時報告。REPORT AT 01:59 STATOPTIONS Specifies information to include in statistical displays. 定義每次使用stat時統計數字是否需要重置 REPORTCOUNT Reports the number of records processed.報告已經處理的紀錄條數統計數字。 示例:REPORTCOUNT EVERY 30 MINUTES, RATE --每隔30分鍾報告一次從程序開始到現在的抽取進程或者復制進程的事物記錄數,並匯報進程的統計信息 TRACE/TRACE2 Shows Extract processing information to assist in revealing processing bottlenecks. 打開對GoldenGate進程的跟蹤日志,一般用於調試。 WARNLONGTRANS Defines a long-running transaction and controls the frequency of checking for and reporting them 指定對於超過一定時間的長交易可以在ggserr.log里面寫入警告信息。例如,每隔30分鍾檢查一次長交易,對於超過12個小時的進行告警: 示例:warnlongtrans 12h, checkintervals 30m WARNLONGTRANS 2h, CHECKINTERVAL 3m --每隔3分鍾檢查一下大事務,超過2小時還沒結束的進行報告
(9)錯誤處理Error handling
DDLERROR Controls error handling for DDL extraction. DISCARDFILE Contains records that could not be processed. 定義discard文件位置,如果處理中有紀錄出錯會寫入到此文件中。 示例1:discardfile /oradata/goldengate/repkj.dsc,append,megabytes 100m 示例2:DISCARDFILE ./dirrpt/extya.dsc, APPEND, MEGABYTES 1024 --將執行失敗的記錄保存在discard file中,該文件位於./dirrpt/extya.dsc,大小為1024MB。 文件中已經包含記錄的話,再后面繼續追加,不刪除之前的記錄
(10)協調Tuning
ALLOCFILES Controls the number of incremental memory structures allocated when the value of NUMFILES is reached. BR Controls the Bounded Recovery feature of Extract. CACHEMGR Controls the virtual memory cache manager. 示例:CACHEMGR CACHESIZE 8000MB, CACHEDIRECTORY /goldengate/temp, CACHEDIRECTORY /goldengate/temp2 在/goldengate/temp,/goldengate/temp2 目錄設置虛擬內存 CHECKPOINTSECS Controls how often Extract writes a checkpoint. DBOPTIONS Specifies database options.指定對於某種特定數據庫所需要的特殊參數 示例:DBOPTIONS ALLOWUNUSEDCOLUMN --當抽取進程遇到一個沒有使用的字段時只生成一個警告,進程會繼續執行而不會被異常終止(abend) DDLOPTIONS Specifies DDL processing options. DYNAMICRESOLUTION | NODYNAMICRESOLUTION Suppresses the metadata lookup for a table until Extract encounters transactional data for it. Makes Extract start faster when there are numerous tables specified for synchronization. 示例:DYNAMICRESOLUTION --有時候開啟OGG進程的時候較慢,可能是因為需要同步的表太多,OGG在開啟進程之前會將需要同步的表建立一個記錄並且存入到磁盤中,這樣就需要耗費大量的時間。使用該參數來解決此問題。 EOFDELAY | EOFDELAYCSECS Determines how long Extract delays before searching for more data to process. FLUSHSECS | FLUSHCSECS Determines the amount of time that record data remains buffered before being written to the trail. FUNCTIONSTACKSIZE Controls the size of the memory stack that is used for processing Oracle GoldenGate functions. GROUPTRANSOPS Controls the number of records that are sent to the trail in one batch. LOBMEMORY Controls the amount of memory and temporary disk space available for caching transactions that contain LOB s. MAXFETCHSTATEMENTS Controls the maximum number of prepared queries that Extract can use to fetch data from the database. NUMFILES Controls the initial allocation of memory dedicated to storing information about tables to be processed by Oracle GoldenGate. 定義本extract為最大多少張表預留空間,缺省為500,超過500張表需要設定一個比實際表數略大的值。 示例:numfiles 3000 RMTHOSTOPTIONS Specifies connection attributes other than host information for a TCP/IP connection used by a passive Extract group. THREADOPTIONS Controls aspects of the way that Extract operates in an Oracle Real Application Cluster environment. Supports Oracle. 示例:THREADOPTIONS MAXCOMMITPROPAGATIONDELAY 60000 IOLATENCY 60000 RAC集群中的所有節點必須同步系統時鍾。GoldenGate通過比較本地系統的時間和事務提交的時間點來做出關鍵決策。可以通過NTP來不同系統時間。所有節點上的COMPATIBLE參數設置也必須相同。注:這個參數在Oracle11.2版本后就不再使用了。 TRANLOGOPTIONS Supplies log-processing options. 指定在解析數據庫日志時所需要的特殊參數。例如,對於裸設備,可能需要加入下列參數: 示例1: tranlogoptions rawdeviceoffset 0 示例2:TRANLOGOPTIONS CONVERTUCS2CLOBS --指定在解析數據庫日志時所需要的特殊參數 --本參數只用在extract端 UTF字符類型,並且11.1.1前處理CLOB才需要: 示例3:--tranlogoptions asmuser sys@asm, asmpassword AACAAAAAAAAAAGAIF, ENCRYPTKEY default 示例4:TRANLOGOPTIONS altarchivelogdest primary instance sgpmdb1 /sgpmdb/arch altarchivelogdest instance sgpmdb2 /sgpmdb/arch 指定RAC環境下每個節點的歸檔日志地址 TRANSMEMORY Controls the amount of memory and temporary disk space available for caching uncommitted transaction data. 設定GoldenGate的抽取進程能夠使用的內存大小。如下參數指定本進程最大只能占用2G內存,其中每個事務最大占用內存不能超過500M,如果超過則使用指定目錄作為虛擬內存,該目錄下的單個文件大小為4G,最多只能在該目錄下占用8G空間作為緩存: 示例:transmemory directory (/backup/goldengate/dirtmp,8G,4G),ram 2G,transram 500M
(11)維護Maintenance
DISCARDROLLOVER Controls how often to create a new discard file. 示例:DISCARDROLLOVER AT 3:00 --為了防止discard file被寫滿,每天3:00做一次文件過期設定 PURGEOLDEXTRACTS Purges obsolete trail files.同mgr進程,可以設置自動刪除隊列,建議在mgr設置 REPORTROLLOVER Specifies when to create new report files. 設定切換一個日志的時間和間隔。示例:reportrollover at 02:00 ROLLOVER Specifies the way that trail files are aged.
(12)安全性Security
DECRYPTTRAIL Decrypts data in a trail or extract file. 解密trail文件或extract文件數據 ENCRYPTTRAIL | NOENCRYPTTRAIL Controls encryption of data in a trail or extract file. 加密trail或extract文件的數據
三、replicat相關參數
1、常用參數
replicat提供如下方面參數:
Checkpoint 數據影射關系 Optional row-level selection criteria Optional column mapping facilities Optional transformation services Optional Stored Procedure or SQL query execution 錯誤處理機制 其它可選參數
參數舉例:
replicat repsz setenv ( NLS_LANG = AMERICAN_AMERICA.ZHS16GBK ) userid goldengate, password ***** sqlexec "Alter session set constraints=deferred" REPORT AT 01:59 reportrollover at 02:00 --handlecollisions reperror default,discard discardfile /oradata/goldengate/dirrpt/repsz.dsc,append, megabytes 10 --grouptransops 100 --batchsql assumetargetdefs allownoopupdates dynamicresolution numfiles 3000 --mapexclude CTAIS2.JC_GY_SWWSWH --map CTAIS2.SHOULIXINXI, target CTAIS2.SHOULIXINXI, keycols ( SHOULIBIANHAO ); MAP ctais2.* ,TARGET ctais2.*;
Map參數:
MAP <source table>, TARGET <target table> [, EXCEPTIONSONLY] [, COLMAP (<column mapping specification>)] [, KEYCOLS (<column list>)] [, WHERE (<where condition>)] [, FILTER (<expression>)] ;
注意map語句必須以一個分號結束;
2、完整參數示例:
示例1:
GGSCI 6> view params repsa REPLICAT repsa SETENV (NLS_LANG = "American_America.UTF8") --設置字符集環境變量為UTF8 SETENV (ORACLE_SID = "xxxx" ) --如果系統中存在多個數據庫有時候會用參數SETENV設置ORACLE_HOME、ORACLE_SID等 USERID goldengate, PASSWORD AACAAAAAAAAAAAJAPJYATHVIMGMAOFHIUCUEPFYGGJBFJGIH, ENCRYPTKEY default --SQLEXEC "ALTER SESSION SET CONSTRAINTS=DEFERRED" DBOPTIONS DEFERREFCONST --約束延遲設置。在復制進程的事物被提交之前,延遲級聯刪除、級聯更新時的校驗和實施。 GETTRUNCATES --不捕獲生產端truncate table的操作。 REPORT AT 06:00 --每天早上6點報告 REPORTCOUNT EVERY 30 MINUTES, RATE --每隔30分鍾報告一次從程序開始到現在的抽取進程或者復制進程的事物記錄數,並匯報進程的統計信息 REPORTROLLOVER AT 02:00 --為了防止report file被寫滿,每天2:00做一次文件過期設定 REPERROR DEFAULT, ABEND --除了特殊指定的REPERROR語句,報告所有復制期間出現的錯誤,回滾非正常中斷的事物和進程。 --HANDLECOLLISIONS --當災備端已經存在數據的情況下,解決復制過程中出現的沖突。如果要重新做初始化,可以刪除drop災備端數據庫后再rman恢復,這樣做的話就不需要該參數了。 ####ALLOWNOOPUPDATES --當生產端有某些列但是目標表卻沒有,或者復制進程中配置了COLSEXCEPT 參數 在這些情況下,當生產端對那些列進行更新,目標表將不發生任何變化 assumetargetdefs --使用ASSUMETARGETDEFS參數時,用MAP語句中指定的生產庫源表和災備端目標表具有相同的列結構。它指示的Oracle GoldenGate不在生產端查找源表的結構定義。 DISCARDFILE ./dirrpt/repsa.dsc, APPEND, MEGABYTES 1024M --將執行失敗的記錄保存在discard file中,該文件位於./dirrpt/extya.dsc,大小為1024MB。 文件中已經包含記錄的話,再后面繼續追加,不刪除之前的記錄。 DISCARDROLLOVER AT 02:00 --為了防止discard file被寫滿,每天2:00做一次文件過期設定 —Schema名稱涉及敏感字符,在此統一使用Schema來代替 —table名稱涉及敏感字符,在此統一使用table來代替 map Schema1.table1, target Schema1.table1; map Schema1.table2, target Schema1.table2; map Schema2.table1, target Schema2.table1; map Schema2.table2, target Schema2.table2;
示例2:
replicat repl --復制進程名 userid ddw,password ddw --目標數據庫的帳號密碼 assumetargetdefs --兩台數據庫數據結構一致則使用此參數 reperror default,discard --如果復制數據出錯則忽略, discardfile D:\repsz.dsc,append,megabytes 100 --錯誤信息寫入XXXX文件,追加模式,最大100m gettruncates --復制truncate操作 map ddw.aatest, target ddw.aatest_1; --映射關系,注意target前必須留一個空格 --map ddw.aatest, target ddw.aatest_1, keycols (name), where (TYPE1 = "1") --加兩橫杠在配置文件中進行注釋;keycols指定唯一鍵;where篩選數據
3、全部可選參數
(1)通用參數
CHECKPARAMS Verifies parameter file syntax.檢查參數語法然后停止。 COMMENT | -- Denotes comments in a parameter file. 注釋行。 GETENV Retrieves variables that were set with the SETENV parameter. SETENV Specifies a value for a UNIX environment variable from within the GGSCI interface. 示例:SETENV (NLS_LANG = "American_America.UTF8") --設置字符集環境變量為UTF8 示例:SETENV (ORACLE_SID = "xxxx" ) --如果系統中存在多個數據庫有時候會用參數SETENV設置ORACLE_HOME、ORACLE_SID等 OBEY Processes parameter statements contained in a different parameter file.將外部文件包含到參數文件中。 TRACETABLE | NOTRACETABLE Specifies a trace table to which Replicat adds a record whenever it updates the target database. Supports Oracle.
(2)Processing method
BEGIN Specifies a starting point for Replicat processing. Required when SPECIALRUN is specified. BULKLOAD Loads data directly into the interface of the Oracle SQL*Loader utility. END Specifies a stopping point for Replicat processing. Required when using SPECIALRUN . GENLOADFILES Generates run and control files that are compatible with a database load utility. REPLICAT Specifies a Replicat group for online change synchronization.定義進程名稱。示例:replicat repsz SPECIALRUN Used for one-time processing that does not require checkpointing from run to run.
(3)Database login
TARGETDB Specifies the data source as part of the login information. USERID Specifies database connection information. 指定登陸目標數據庫的用戶名和密碼。 示例:Userid goldengate, password 123456 USERID goldengate, PASSWORD AACAAAAAAAAAAAJAPJYATHVIMGMAOFHIUCUEPFYGGJBFJGIH, ENCRYPTKEY default
(4)Selecting, converting, and mapping data
ALLOWDUPTARGETMAP | NOALLOWDUPTARGETMAP Allows the same source-target MAP statement to appear more than once in the parameter file. ASCIITOEBCDIC Converts incoming ASCII text to EBCDIC for DB2 on z/OS systems running UNIX System Services. ASSUMETARGETDEFS Assumes that source and target tables have the same column structure.假定兩端數據結構一致使用此參數。 示例: assumetargetdefs --使用ASSUMETARGETDEFS參數時,用MAP語句中指定的生產庫源表和災備端目標表具有相同的列結構。它指示的Oracle GoldenGate不在生產端查找源表的結構定義。 COLMATCH Establishes global column-mapping rules. DDL Enables and filters the capture of DDL operations. DDLSUBST Enables string substitution in DDL processing. GETDELETES | IGNOREDELETES Controls the replication of delete operations.是否復制delete操作,缺省為復制。 GETINSERTS | IGNOREINSERTS Controls the replication of insert operations.是否復制insert操作,缺省為復制。 GETUPDATEAFTERS | IGNOREUPDATEAFTERS Controls the replication of update after images. 是否讀取后影像,缺省為讀取 GETUPDATEBEFORES | IGNOREUPDATEBEFORES Controls the replication of update before images. 是否讀取前影像,缺省為不讀取。 GETUPDATES | IGNOREUPDATES Controls the replication of update operations.是否復制update操作,缺省為復制。 GETTRUNCATES | IGNORETRUNCATES Includes or excludes the replication of TRUNCATE statements.是否復制truncate操作,缺省為不復制。 示例:GETTRUNCATES --不捕獲生產端truncate table的操作。 INSERTALLRECORDS Inserts a new record into the target table for every change operation made to a record. INSERTDELETES | NOINSERTDELETES Converts deletes to inserts. INSERTMISSINGUPDATES | NOINSERTMISSINGUPDATES Converts an update to an insert when the target row does not exist. INSERTUPDATES | NOINSERTUPDATES Converts updates to inserts. MAP for Replicat Specifies a relationship between one or more source and target tables and controls column mapping and conversion. MAPEXCLUDE Excludes tables from being processed by a wildcard specification supplied in MAP statements. 用於使用在map中使用*匹配時排除掉指定的表,類似於於源端的tablexclude。 示例:mapexclude CTAIS2.JC_GY_SWWSWH MAP ctais2.* ,TARGET ctais2.*; REPLACEBADCHAR Replaces invalid character values with another value. REPLACEBADNUM Replaces invalid numeric values with another value. SOURCEDEFS Specifies a file that contains source data definitions created by the DEFGEN utility. 假定兩端數據結構不一致,使用此參數指定源端的數據結構定義文件。該文件需要由GoldenGate工具產生。 SPACESTONULL | NOSPACESTONULL Controls whether or not a target column containing only spaces is converted to NULL . TABLE for Replicat Specifies a table or tables for which event actions are to take place when a row satisfies the given filter criteria. TRIMSPACES | NOTRIMSPACES Controls whether trailing spaces are trimmed or not when mapping CHAR to VARCHAR columns. UPDATEDELETES | NOUPDATEDELETES Changes deletes to updates. USEDATEPREFIX Prefixes data values for DATE data types with a DATE literal, as required by Teradata databases. USETIMEPREFIX Prefixes data values for TIME datatypes with a TIME literal, as required by Teradata databases. USETIMESTAMPPREFIX Prefixes data values for TIMESTAMP datatypes with a TIMESTAMP literal, as required by Teradata databases.
(5)Routing data
EXTFILE Defines the name of an extract file on the local system that contains data to be replicated. Used for one-time processing. EXTTRAIL Defines a trail containing data to be replicated. Used for one-time processing.
(6)Custom processing
CUSEREXIT Invokes a user exit routine during processing. DEFERAPPLYINTERVAL Specifies a length of time for Replicat to wait before applying replicated operations to the target database. INCLUDE References a macro library in a parameter file. MACRO Defines an Oracle GoldenGate macro. SQLEXEC Executes a stored procedure or query during Replicat processing. 調用存儲過程或者執行sql語句。可以將返回值作為依據進行過濾條件,或者用戶改變session變量。 示例:SQLEXEC "ALTER SESSION SET CONSTRAINTS=DEFERRED"
(7)Reporting
CMDTRACE Displays macro expansion steps in the report. LIST | NOLIST Displays or suppresses the listing of macros in the report file. REPORT Schedules a statistical report at a specified date or time. 示例:REPORT AT 06:00 --每天早上6點報告 REPORTCOUNT Reports the number of records processed. 示例:REPORTCOUNT EVERY 30 MINUTES, RATE --每隔30分鍾報告一次從程序開始到現在的抽取進程或者復制進程的事物記錄數,並匯報進程的統計信息 SHOWSYNTAX Causes Replicat to print its SQL statements to the report file. STATOPTIONS Specifies information to include in statistical displays. TRACE/TRACE2 Shows Replicat processing information to assist in revealing bottlenecks.
(8)Error handling
CHECKSEQUENCEVALUE | NOCHECKSEQUENCEVALUE Controls whether or not Replicat verifies that a target sequence value is higher than the one on the source and corrects any disparity that it finds. DDLERROR Controls error handling for DDL replication. DISCARDFILE Contains records that could not be processed. 定義出錯數據的輸出文件。當數據出現錯誤后,可以用於查找錯誤原因。 示例:discardfile /oradata/goldengate/dirrpt/repsz.dsc,append, megabytes 10 DISCARDFILE ./dirrpt/repsa.dsc, APPEND, MEGABYTES 1024M --將執行失敗的記錄保存在discard file中,該文件位於./dirrpt/extya.dsc,大小為1024MB。 文件中已經包含記錄的話,再后面繼續追加,不刪除之前的記錄。 HANDLECOLLISIONS | 自動過濾重復時段的數據沖突,用於不能停機執行初始化。打開該參數后不會將數據錯誤報到discard文件中。 NOHANDLECOLLISIONS Handles errors for duplicate and missing records. 示例:--HANDLECOLLISIONS --當災備端已經存在數據的情況下,解決復制過程中出現的沖突。如果要重新做初始化,可以刪除drop災備端數據庫后再rman恢復,這樣做的話就不需要該參數了。 HANDLETPKUPDATE Prevents constraint errors associated with replicating transient primary key updates. OVERRIDEDUPS | NOOVERRIDEDUPS Overlays a replicated insert record onto an existing target record whenever a duplicate-record error occurs. RESTARTCOLLISIONS | Controls whether or not Replicat applies NORESTARTCOLLISIONS HANDLECOLLISIONS logic after Oracle GoldenGate has abended because of a conflict. REPERROR Determines how Replicat responds to database errors. 定義出錯以后replicat的響應,一般可以定義為兩種: Abend,即一旦出現錯誤即停止復制,此為缺省配置; Discard,出現錯誤后繼續復制,只把錯誤的數據放到discard文件中。 示例:REPERROR DEFAULT, DISCARD REPERROR DEFAULT, ABEND --除了特殊指定的REPERROR語句,報告所有復制期間出現的錯誤,回滾非正常中斷的事物和進程。 REPFETCHEDCOLOPTIONS Determines how Replicat responds to operations for which a fetch from the source database was required. SQLDUPERR Specifies the database error number that indicates a duplicate record. Use with OVERRIDEDUPS . WARNRATE Determines how often database errors are reported.
(9)Tuning
ALLOCFILES Controls the amount of incremental memory that is allocated when the amount of memory specified with NUMFILES is reached. BATCHSQL Increases the throughput of Replicat processing by arranging similar SQL statements into arrays and applying them at an accelerated rate. 針對批處理中針對某個表的大批量重復操作進行優化,提高批處理的處理速度。 CHECKPOINTSECS Controls how often Replicat writes a checkpoint when checkpoints are not being generated as the result of transaction commits. DBOPTIONS Specifies database options.定義與數據庫類型相關的特殊處理方式。 示例:DBOPTIONS DEFERREFCONST --約束延遲設置。在復制進程的事物被提交之前,延遲級聯刪除、級聯更新時的校驗和實施。 DDLOPTIONS Specifies DDL processing options. DYNAMICRESOLUTION | 使replicat動態解析表的結構,加快啟動速度。缺省為每次啟動解析所有要復制表的結構。 NODYNAMICRESOLUTION Makes Replicat start faster when there is a large number of tables specified for synchronization. DYNSQL | NODYNSQL Causes Replicat to use literal SQL statements rather than a compile-once, execute-many strategy. EOFDELAY | EOFDELAYCSECS Determines how many seconds Replicat delays before looking for more data to process. FUNCTIONSTACKSIZE Controls the size of the memory stack that is used for processing Oracle GoldenGate functions. GROUPTRANSOPS Controls the number of records that are grouped into a Replicat transaction. 將小的交易合並成為一個大的交易進行提交,減少提交次數,降低系統IO消耗。示例:grouptransops 100 INSERTAPPEND | NOINSERTAPPEND Controls whether or not Replicat uses an APPEND hint when applying INSERT operations to Oracle target tables. MAXDISCARDRECS Limits the number of discarded records reported to the discard file. MAXSQLSTATEMENTS Controls the number of prepared SQL statements that can be used by Replicat. MAXTRANSOPS Divides large source transactions into smaller ones on the target system. 將大交易拆分,每若干條紀錄提交一次。示例:maxtransops 1000 NUMFILES Controls the initial allocation of memory that is dedicated to storing information about tables to be processed by Oracle GoldenGate. 定義進程中表的最大數據量,缺省為 500. RETRYDELAY Specifies the delay between attempts to retry a failed SQL operation. TRANSACTIONTIMEOUT Specifies a time interval after which Replicat will commit its open target transaction and roll back any incomplete source transactions that it contains, saving them for when the entire source transaction is ready to be applied. TRANSMEMORY Controls the amount of memory and temporary disk space available for caching uncommitted transaction data. WILDCARDRESOLVE Alters the rules by which wildcard specifications in a MAP statement are resolved.
(10)Maintenance
DISCARDROLLOVER Specifies when to create new discard files. 示例:DISCARDROLLOVER AT 02:00 --為了防止discard file被寫滿,每天2:00做一次文件過期設定 PURGEOLDEXTRACTS Purges obsolete trail files. 定義自動刪除隊列,一般建議在mgr進程配置。 REPORTROLLOVER Specifies when to create new report files. 示例: REPORTROLLOVER AT 02:00 --為了防止report file被寫滿,每天2:00做一次文件過期設定
(11)Security
DECRYPTTRAIL Decrypts data in a trail or extract file.
補充:
allownoopupdates 參數含義:允許執行無實際變化的update。例如,update a=a會紀錄一條update,但是沒有后影像,無法正確構築where語句。 ALLOWNOOPUPDATES --當生產端有某些列但是目標表卻沒有,或者復制進程中配置了COLSEXCEPT 參數 在這些情況下,當生產端對那些列進行更新,目標表將不發生任何變化
四、定義文件參數
DEFSFILE Identifies the name of the file to which DEFGEN writes the definitions SOURCEDB Specifies the data source as part of the login information. TABLE for DEFGEN Identifies a table for which you want to capture a definition. USERID Specifies database connection information.