A：故障：node未找到.
原因：node機器添加完成后，跳轉到機器列表頁面，獲取對應的機器序號nid,node節點對應的唯一標識
B: 故障：Unsupported BinlogFormat MIXED
show variables like '%log%';
看到 binlog_format值仍為MIXED
原因：my.cnf下有多余binlog_format賦值MIXED
有一點特別注意：目前canal支持mixed,row,statement多種日志協議的解析，但配合otter進行數據庫同步，目前僅支持row協議的同步，使用時需要注意.
https://github.com/alibaba/otter/wiki/QuickStart
C: 故障：channel掛起，從庫未同步主庫表數據。
(解決方案同DDL語句不能執行故障Exception: no support ddl for）
解決方案;在otter的管理界面，點擊同步管理，停用Channel,點擊進入Pipeline==>管理＝>最下面高級＝>跳過ddl異常。重啟就OK了
掛起時的報錯信息：
pid:9 nid:4 exception:setl:com.alibaba.otter.node.etl.load.exception.LoadException: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
Caused by: java.util.concurrent.ExecutionException: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader.load(DataBatchLoader.java:107)
at com.alibaba.otter.node.etl.load.loader.OtterLoaderFactory.load(OtterLoaderFactory.java:50)
at com.alibaba.otter.node.etl.load.LoadTask$1.run(LoadTask.java:85)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
Caused by: com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
Caused by: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:98)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.doDdl(DbLoadAction.java:357)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.load(DbLoadAction.java:135)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$$FastClassByCGLIB$$d932a4cb.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$$EnhancerByCGLIB$$80fd23c2.load()
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:198)
at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:189)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'CompanyName' doesn't exist in table
at sun.reflect.GeneratedConstructorAccessor91.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:839)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:739)
at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$1.doInStatement(DbLoadAction.java:369)
at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$1.doInStatement(DbLoadAction.java:357)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:396)
... 14 more
https://blog.csdn.net/weixin_36567485/java/article/details/78021636
1. canal和otter的關系？
答： 在回答這問題之前，首先來看一張canal&otter和mysql復制的類比圖. 
mysql的自帶復制技術可分成三步：
master將改變記錄到二進制日志(binary log)中（這些記錄叫做二進制日志事件，binary log events，可以通過show binlog events進行查看）；
slave將master的binary log events拷貝到它的中繼日志(relay log)，這里是I/O thread線程.
slave重做中繼日志中的事件，將改變反映它自己的數據，這里是SQL thread線程.
基於canal&otter的復制技術和mysql復制類似，具有類比性.
Canal對應於I/O thread，接收Master Binary Log.
Otter對應於SQL thread，通過Canal獲取Binary Log數據，執行同步插入數據庫.
兩者的區別在於：
otter目前嵌入式依賴canal，部署為同一個jvm，目前設計為不產生Relay Log，數據不落地.
otter目前允許自定義同步邏輯，解決各類需求.
a. ETL轉化. 比如Slave上目標表的表名，字段名，字段類型不同，字段個數不同等.
b. 異構數據庫. 比如Slave可以是oracle或者其他類型的存儲,nosql等.
c. M-M部署，解決數據一致性問題
d. 基於manager部署，方便監控同步狀態和管理同步任務.
2. canal目前支持的數據庫版本？
答： 支持mysql系列的5.1 ~ 5.6/5.7版本，mariadb 5/10版本. (全面支持ROW/STATEMENT/MIXED幾種binlog格式的解析)
3. otter目前支持的數據庫情況？
答：這里總結了一下
從問題1中的圖中看到，otter依賴canal解決數據庫增量日志，所以會收到canal的版本支持限制，僅支持mysql系列，不支持oracle做為master庫進行解析.
mysql做為master，otter只支持ROW模式的數據同步，其他兩種模式不支持. (只有ROW模式可保證數據的最終一致性)
目標庫，也就是slave，可支持mysql/oracle，也就是說可以將mysql的數據同步到oracle庫中，反過來不行.
4. otter目前存在的同步限制？
答：這里總結了一下
暫不支持無主鍵表同步. (同步的表必須要有主鍵，無主鍵表update會是一個全表掃描，效率比較差)
支持部分ddl同步 (支持create table / drop table / alter table / truncate table / rename table / create index / drop index，其他類型的暫不支持，比如grant,create user,trigger等等)，同時ddl語句不支持冪等性操作，所以出現重復同步時，會導致同步掛起，可通過配置高級參數:跳過ddl異常，來解決這個問題.
不支持帶外鍵的記錄同步. (數據載入算法會打散事務，進行並行處理，會導致外鍵約束無法滿足)
數據庫上trigger配置慎重. (比如源庫，有一張A表配置了trigger，將A表上的變化記錄到B表中，而B表也需要同步。如果目標庫也有這trigger，在同步時會插入一次A表，2次B表，因為A表的同步插入也會觸發trigger插入一次B表，所以有2次B表同步.)
5. otter同步相比於mysql的優勢？
答：
管理&運維方便. otter為純java開發的系統，提供web管理界面，一站式管理整個公司的數據庫同步任務.
同步效率提升. 在保證數據一致性的前提下，拆散原先Master的事務內容，基於pk hash的並發同步，可以有效提升5倍以上的同步效率.
自定義同步功能. 支持基於增量復制的前提下，定義ETL轉化邏輯，完成特殊功能.
異地機房同步. 相比於mysql異地長距離機房的復制效率，比如阿里巴巴杭州和美國機房，復制可以提升20倍以上. 長距離傳輸時，master向slave傳輸binary log可能會是一個瓶頸.
雙A機房同步. 目前mysql的M-M部署結構，不支持解決數據的一致性問題，基於otter的雙向復制+一致性算法，可完美解決這個問題，真正實現雙A機房.
特殊功能.
a. 支持圖片同步. 數據庫中的一條記錄，比如產品記錄，會在數據庫里存在一張圖片的path路徑，可定義規則，在同步數據的同時，將圖片同步到目標.
6. node jvm內存不夠用，如何解決？
node出現java.lang.OutOfMemoryError : Gc overhead limit exceeded.
答：
單node建議的同步任務，建議控制下1~2wtps以下，不然內存不夠用. 出現不夠用時，具體的解決方案:
調大node的-Xms,-Xmx內存設置，默認為3G,heap區大概是2GB
減少每個同步的任務內存配置.
a. canal配置里有個內存存儲buffer記錄數參數，默認是32768，代表32MB的binlog，解析后在內存中會占用100MB的樣子.
b. pipeline配置里有個批次大小設置，默認是6000，代表每次獲取6MB左右，解析后在內存占用=並行度*6MB*3，大概也是100MB的樣子.
所以默認參數，全速跑時，單個通道占用200MB的樣子，2GB能跑幾個大概能估算出來了
7. 源庫binlog不正確，如何重置同步使用新的位點開始同步？
場景：
源庫binlog被刪除，比如出現：Could not find first log file name in binary log index file
源庫binlog出現致命解析錯誤，比如運行過程使用了刪除性質的ddl，drop table操作，導致后續binlog在解析時無法獲取表結構.
答：
首先需要理解一下canal的位置管理，主要有兩個位點信息：起始位置 和 運行位置(記錄最后一次正常消費的位置).
優先加載運行位置，第一次啟動無運行位置，就會使用起始位置進行初始化，第一次客戶端反饋了ack信號后，就會記錄運行位置.
所以重置位置的幾步操作：
（1）刪除運行位置. (pipeline同步進度頁面配置)
（2）配置起始位置. (canal配置頁面)
（3）檢查是否生效. (pipeline對應的日志)
注意點：
如果日志中出現prepare to find start position just last position. 即代表是基於運行位置啟動，新設置的位點並沒有生效。 (此時需要檢查下位點刪除是否成功 或者 canal是否被多個pipeline引用，導致位點刪除后，被另一個pipeline重新寫入，導致新設置的位點沒有生效.)
otter中使用canal，不允許pipeline共享一個canal. otter中配置的canal即為一個instance，而otter就是為其一個client，目前canal不支持一個instance多個client的模式，會導致數據丟失，慎重.
點 啟用后的日志： 2020-05-22 18:54:34.094 [pipelineId = 1,taskName = SelectTask] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem is not run any in node 2020-05-22 18:54:34.153 [destination = crm_canal , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> begin to find start position, it will be long time for reset or first position 2020-05-22 18:54:34.153 [destination = crm_canal , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position mysql-uat-new-mysqlha-0-bin.002301:443951933:null 2020-05-22 18:54:34.153 [destination = crm_canal , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-uat-new-mysqlha-0-bin.002301,position=443951933,serverId=<null>,gtid=<null>,timestamp=<null>] cost : 0ms , the next step is binlog dump
2020-05-20 18:21:54.686 [pipelineId = 4,taskName = SelectTask] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem check is interrupt 2020-05-20 18:56:24.826 [pipelineId = 4,taskName = SelectTask] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem is not run any in node 2020-05-20 18:56:24.872 [pipelineId = 4,taskName = SelectTask] WARN c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Could not load properties from class path resource [canal.properties]: class path resource [canal.properties] cannot be opened because it does not exist 2020-05-20 18:56:24.872 [pipelineId = 4,taskName = SelectTask] WARN c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Could not load properties from class path resource [instance.properties]: class path resource [instance.properties] cannot be opened because it does not exist 2020-05-20 18:56:24.929 [destination = _channel_new , address = rds-test.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position 2020-05-20 18:56:24.933 [destination = _channel_new , address = rds-test.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position {"identity":{"slaveId":-1,"sourceAddress":{"address":"rds-test.mysql.rds.aliyuncs.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mysql-bin.005447","position":508370854,"serverId":2339357115,"timestamp":1589970112000}} 2020-05-20 18:56:24.944 [destination = _channel_new , address = rds-test.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.005447,position=508370854,serverId=2339357115,gtid=,timestamp=1589970112000] cost : 15ms , the next step is binlog dump
一個現象，需要排查，為什么沒有從 運行位置 啟動，而是從 起始位置 啟動：
運行位置:
{"journalName":"mysql-uat-new-mysqlha-0-bin.002534","position":314056714,"timestamp":1591076810000}
起始位置【配置在canal中】:
{"journalName":"mysql-uat-new-mysqlha-0-bin.002532","position":597087918,"timestamp":1591066862000};
在界面點停用，再點啟動后，otter拿的起始位點是 canal中配置的那個，而不是停止時正在運行的那個位點。
2020-06-02 13:51:03.063 [destination = crm_canal , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start positionmysql-uat-new-mysqlha-0-bin.002532:597087918:1591066862000
2020-06-02 13:51:03.064 [destination = crm_canal , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-uat-new-mysqlha-0-bin.002532,position=597087918,serverId=<null>,gtid=<null>,timestamp=1591066862000] cost : 0ms , the next step is binlog dump
com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser#findStartPositionInternal
8. 日志列表中出現POSITIONTIMEOUT后，數據進行重復同步？
答：首先需要理解下Position超時監控，該監控主要是監控當前同步進度中的位點的最后更新時間和當前時間的差值，簡單點說就是看你同步進度的位點多少時間沒有更新了，超過閥值后觸發該報警規則.
還有一點需要說明，同步進度中的位點只會記錄一個事務的BEGIN/COMMIT位置，保證事務處理的完整性，不會記錄事務中的中間位置.
幾種情況下會出現同步進度位點長時間無更新：
（1）源庫出現大事務，比如進行load data/ delete * from xxx，同時操作幾百萬/千萬的數據，同步該事務數據時，位點信息一直不會被更新，比如默認超過10分鍾后，就會觸發Position超時監控，此時就是一個誤判，觸發自動恢復，又進入重新同步，然后進入死循環。
（2）otter系統未知bug，導致系統的調度算法出現死鎖，無法正常的同步數據，導致同步進度無法更新，觸發該Position超時監控。
此時：自動恢復的 一次停用+啟用 同步任務，就可以恢復正常.
ps. 該Position超時監控，可以說是主要做為一種otter的系統保險機制，可以平衡一下，如果誤判的影響>系統bug觸發的概率，可以考慮關閉Position超時監控，一般超時監控也會發出報警.
9. 日志列表中出現miss data with keys異常，同步出現掛起后又自動恢復？
異常信息：
pid:2 nid:2 exception:setl:load miss data with keys:[MemoryPipeKey[identity=Identity[channelId=1,pipelineId=2,processId=4991953],time=1383190001987,dataType=DB_BATCH]]
答：要理解該異常，需要先了解一下otter調度模型，里面SEDA中多個stage之間通過pipe進行數據交互，比如T模塊完成后會將數據存到pipe中，然后通知SEDA中心，中心會通知L模塊起來工作，L模塊就會根據T傳給中心的pipeKey去提取數據，而該異常就是當L模塊根據pipeKey去提取數據時，發現數據沒了。 主要原因：pipe在設計時，如果是單機傳輸時，會使用softReference來存儲，所以當jvm內存不足時就會被GC掉，所以就會出現無數據的情況.
ps. 如果miss data with keys異常非常多的時候，你就得考慮是否當前node已經超負載運行，內存不夠，需要將上面的部分同步任務遷移出去。如果是偶爾的異常，那可以忽略，該異常會有自動恢復RESTART同步任務的處理。
10. 日志列表中出現manager異常？
異常信息：
pid:2 nid:null exception:channel:can't restart by no select live node
該異常代表pipelineId = 2，select模塊的node沒有可用的節點.
異常信息：
pid:-1 nid:null exception:cid:2 restart recovery successful for rid:-1
該異常代表channelId = 2，成功發起了一次restart同步任務的操作.
異常信息：
pid:-1 nid:null exception:nid:2 is dead and restart cids:[1,2]
該異常代表node id = 2，因為該node掛了，觸發了channelId = 1 / 2的兩個同步任務發起restart同步任務的操作. (一種failover的機制)
https://github.com/alibaba/otter/wiki/Faq
監控通道的延遲情況，可以從otter.delay_stat表中得到
https://blog.csdn.net/woson_wang/article/details/89011647
監控日志可在表 otter.log_record表中查
【todo】
pid:8 nid:4 exception:canal:espro:com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`retl`.`xdual` Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`retl`.`xdual` Caused by: java.net.SocketException: Connection timed out (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:143) at com.alibaba.otter.canal.parse.driver.mysql.socket.BioSocketChannel.write(BioSocketChannel.java:36) at com.alibaba.otter.canal.parse.driver.mysql.utils.PacketManager.writeBody0(PacketManager.java:42) at com.alibaba.otter.canal.parse.driver.mysql.utils.PacketManager.writeBody(PacketManager.java:35) at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.query(MysqlQueryExecutor.java:55) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.query(MysqlConnection.java:105) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.TableMetaCache.getTableMeta(TableMetaCache.java:170) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.getTableMeta(LogEventConvert.java:915) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:475) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:496) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:487) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:125) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:67) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser.parseAndProfilingIfNecessary(AbstractEventParser.java:409) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3$1.sink(AbstractEventParser.java:209) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:168) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:271) at java.lang.Thread.run(Thread.java:748)
沒有權限時，連續的報錯：
pid:10 nid:3 exception:canal:Canal名字:com.alibaba.otter.canal.parse.exception.CanalParseException: command : 'show master status' has an error! Caused by: java.io.IOException: ErrorPacket [errorNumber=1227, fieldCount=-1, message=Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation, sqlState=42000, sqlStateMarker=#] with command: show master status at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.query(MysqlQueryExecutor.java:61) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.query(MysqlConnection.java:105) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findEndPosition(MysqlEventParser.java:653) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findEndPositionWithMasterIdAndTimestamp(MysqlEventParser.java:389) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPositionInternal(MysqlEventParser.java:435) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPosition(MysqlEventParser.java:366) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:186) at java.lang.Thread.run(Thread.java:748)
解決辦法：
授權 canal 鏈接 MySQL 賬號具有作為 MySQL slave 的權限, 如果已有賬戶可直接 grant
CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; FLUSH PRIVILEGES;
https://github.com/alibaba/canal/wiki/QuickStart
在Mysql中，保存用戶權限有三個地方
（1）已經建立的數據庫連接中。grant，revoke或直接改db，不影響已經建立連接的權限情況
（2）Mysql服務器內存中的acl_users 數組
（3）db的mysql.user表中
flush privileges 命令會清空 acl_users 數組，然后從 mysql.user 表中讀取數據重新加載，重新構造一個 acl_users 數組。
也就是說，以數據表中的數據為准，會將全局權限內存數組重新加載一遍。
grant 語句會同時修改數據表和內存，判斷權限的時候使用的是內存數據。
因此，規范地使用 grant 和 revoke 語句，是不需要隨后加上 flush privileges 語句的。
flush privileges 語句本身會用數據表的數據重建一份內存權限數據，所以在權限數據可能存在不一致的情況下再使用。
而這種不一致往往是由於直接用 DML 語句操作系統權限表導致的，所以我們盡量不要使用這類語句。
表權限和列權限除了 db 級別的權限外，MySQL 支持更細粒度的表權限和列權限。
其中，表權限定義存放在表 mysql.tables_priv 中，列權限定義存放在表 mysql.columns_priv 中。
這兩類權限，組合起來存放在內存的 hash 結構 column_priv_hash 中。
https://time.geekbang.org/column/article/82231?utm_source=pinpaizhuanqu&utm_medium=geektime&utm_campaign=guanwang&utm_term=guanwang&utm_content=0511
pid:8 nid:4 exception:canal:espro:com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`retl`.`xdual` Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`retl`.`xdual` Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'admin'@'10.0.2.3' for table 'xdual', sqlState=42000, sqlStateMarker=#] with command: show create table `retl`.`xdual` at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.query(MysqlQueryExecutor.java:61) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.query(MysqlConnection.java:105) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.TableMetaCache.getTableMeta(TableMetaCache.java:170) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.getTableMeta(LogEventConvert.java:915) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:475) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:496) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:487) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:125) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:67) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser.parseAndProfilingIfNecessary(AbstractEventParser.java:409) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3$1.sink(AbstractEventParser.java:209) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:168) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:271) at java.lang.Thread.run(Thread.java:748)
上面報錯已解決。解決辦法，給用戶admin開啟 show create table權限，問題解決。
https://blog.csdn.net/u014355034/article/details/87974990
otter的總體架構
node模塊內嵌Canal,Canal監聽數據庫binlog中的變化傳送給node的SETL模塊
otter強依賴於canal，並對canal的配置有一定的約束。也正是因為強約束，在node中集成了canal，canal作為node的線程運行，使用otter搭建mysql同步環境不需要先手工搭建canal。在開始進入搭建環節之前，建議先看下術語，除非很清楚了，不然相信我，你還是要回過頭來看的。
Channel：同步通道，單向同步中一個Pipeline組成，在雙向同步中有兩個Pipeline組成
Pipeline：從源端到目標端的整個過程描述，主要由一些同步映射過程組成
DataMediaPair：根據業務表定義映射關系，比如源表和目標表，字段映射，字段組等
DataMedia : 抽象的數據介質概念，可以理解為數據表/mq隊列定義
DataMediaSource : 抽象的數據介質源信息，補充描述DateMedia
ColumnPair : 定義字段映射關系
ColumnGroup : 定義字段映射組
Node : 處理同步過程的工作節點，對應一個jvm
otter的S/E/T/L stage階段模型
說明：為了更好的支持系統的擴展性和靈活性，將整個同步流程抽象為Select/Extract/Transform/Load，這么4個階段.
Select階段: 為解決數據來源的差異性，比如接入canal獲取增量數據，也可以接入其他系統獲取其他數據等。
Extract/Transform/Load 階段：類似於數據倉庫的ETL模型，具體可為數據join，數據轉化，數據Load的
https://github.com/alibaba/otter/wiki/Introduction
他們之間的關系為：
otter一共包含兩個部分，manager（作為otter的配置中心和管理控制台應用）和node（作為otter的實際同步工作節點）。
網上和otter文檔均提及需要先安裝manager，我仔細看了下，是因為manager是被動連接的（很多應用的管理控制台是主動去連接服務的，otter則把所有的配置都存儲在了manager中），node啟動的時候會連接到manager獲取同步相關的信息。
生成nid這一步倒沒什么關系，事后不一致修改也可以。
manager配置
首先在計划保存otter配置信息的mysql數據庫執行otter-manager-schema.sql腳本。
manager的配置文件主要是manager/conf/otter.properties，如下所示（下面列出了建議和需要修改的）：
[root@v-03-01-00223 conf]# cat otter.properties ## otter manager domain name otter.domainName = 172.28.1.97 ## 建議改成所在服務器的ip，而不是默認的127.0.0.1，否則到時候啟動的時候所有的連接指向的目標都是localhost，因為通常otter跑在linux環境，很多linux環境是沒有圖形化界面的，感覺這是個bug ## otter manager http port otter.port = 8088 ## 如果非專用或者已經有了一些web應用在同一台服務器，建議改成其他的避免端口沖突，這里的端口號要和jetty.xml中的保持一致，這里也是，直接用個非8080端口就更友好了，比如weblogic 控制台7001,es控制台9200,rabbitmq控制台15672 ## jetty web config xml otter.jetty = jetty.xml ## otter manager database config otter.database.driver.class.name = com.mysql.jdbc.Driver otter.database.driver.url = jdbc:mysql://127.0.0.1:3308/otter ## otter配置信息維護的數據庫地址，庫名一般為otter/otter_manager/manager otter.database.driver.username = root otter.database.driver.password = 123456 ## otter communication port otter.communication.manager.port = 1099 ## node和manager通信的接口,一般不用修改 ## otter communication pool size otter.communication.pool.size = 10 ## default zookeeper address otter.zookeeper.cluster.default = 127.0.0.1:2181 ## zk地址 ## default zookeeper sesstion timeout = 60s otter.zookeeper.sessionTimeout = 60000 ## otter arbitrate connect manager config otter.manager.address = ${otter.domainName}:${otter.communication.manager.port} ## should run in product mode , true/false otter.manager.productionMode = true ## self-monitor enable or disable otter.manager.monitor.self.enable = true ## self-montir interval , default 120s otter.manager.monitor.self.interval = 120 ## auto-recovery paused enable or disable otter.manager.monitor.recovery.paused = true # manager email user config otter.manager.monitor.email.host = smtp.gmail.com otter.manager.monitor.email.username = otter.manager.monitor.email.password = otter.manager.monitor.email.stmp.port = 465
上述配置修改之后，就可以啟動manager了。
[root@v-03-01-00223 bin]# pwd
/usr/local/app/manager/bin
./startup.sh
查看日志
tail -fn 100 ../logs/manager.log
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
2018-07-03 14:59:49.002 [] INFO com.alibaba.otter.manager.deployer.OtterManagerLauncher - ## start the manager server.
2018-07-03 14:59:57.420 [] INFO com.alibaba.otter.manager.deployer.JettyEmbedServer - ##Jetty Embed Server is startup!
2018-07-03 14:59:57.420 [] INFO com.alibaba.otter.manager.deployer.OtterManagerLauncher - ## the manager server is running now ......
接下去就可以驗證manager了。
用瀏覽器打開http://172.18.1.97:8088/
端口建議不要修改。
機器添加完成以后，機器管理的列表中第一列就是nid（這個就是到時候要保存到node/conf/nix文件中的值），如下：
https://github.com/alibaba/otter/wiki/Node_Quickstart
上述三種類型的節點配置完成后，manager前期的配置就完成了。
manager配置完成之后，需要先啟動相應的node節點，node節點啟動之后，就可以配置真正的同步任務了。
node配置
首先
cd NODE_HOME/conf echo 1 > nid #這個1 是在manager中添加node時，生成的序號
node配置文件otter.properties（除otter.manager.address外，可以使用默認值）
otter arbitrate & node connect manager config ， 修改為正確的manager服務地址
otter.manager.address = 127.0.0.1:1099
啟動node
cd NODE_HOME/bin
./startup.sh
[root@v-03-01-00223 node]# more node.log
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
2018-07-03 15:13:09.364 [main] INFO com.alibaba.otter.node.deployer.OtterLauncher - INFO ## the otter server is running now ......
此時再查看manager控制台的機器管理，可以發現機器狀態為已啟動，如下：
https://github.com/alibaba/otter/wiki/Node_Quickstart
點位可以通過在主庫執行show master status和select unix_timestamp()得到。
otter高可用
對外開源部分HA這一塊基本上沒有比較完善的。對於canal連接到db主從切換，可以參考：https://www.cnblogs.com/f-zhao/p/7681960.html，已經講到位了。
如果是半同步模式或者基於GTID的話，沒有必要回退60s。
在otter中配置canal的主從切換依賴於groupKey，目前還沒有測，后面測了再補充。
https://www.cnblogs.com/zhjh256/p/9261725.html
node解壓修改配置啟動
機器名稱：可以隨意定義，方便自己記憶即可
機器ip：對應node節點將要部署的機器ip，如果有多ip時，可選擇其中一個ip進行暴露. (此ip是整個集群通訊的入口，實際情況千萬別使用127.0.0.1，否則多個機器的node節點會無法識別)
機器端口：對應node節點將要部署時啟動的數據通訊端口，建議值：2088
下載端口：對應node節點將要部署時啟動的數據下載端口，建議值：9090
外部ip ：對應node節點將要部署的機器ip，存在的一個外部ip，允許通訊的時候走公網處理。
zookeeper集群：為提升通訊效率，不同機房的機器可選擇就近的zookeeper集群.
2020-12-08 10:19:59.881 [] WARN com.alibaba.dubbo.remoting.transport.AbstractClient - [DUBBO] client reconnect to 172.26.9.117:2088 find error . url: dubbo://172.26.9.117:208 8/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&thread s=50, dubbo version: 2.5.3, current host: 10.10.30.121 com.alibaba.dubbo.remoting.RemotingException: Failed connect to server /172.26.9.117:2088 from NettyClient 10.10.30.121 using dubbo version 2.5.3, cause: Connect wait timeout: 1000ms. at com.alibaba.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:282) ~[dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.AbstractClient$1.run(AbstractClient.java:145) ~[dubbo-2.5.3.jar:2.5.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] 2020-12-08 10:20:03.880 [] WARN com.alibaba.dubbo.remoting.transport.AbstractClient - [DUBBO] client reconnect to 172.26.9.117:2088 find error . url: dubbo://172.26.9.117:208 8/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&thread s=50, dubbo version: 2.5.3, current host: 10.10.30.121
node這種設計，是為解決單機部署多實例而設計的，允許單機多node指定不同的端口
# nid配置 (將環境准備中添加機器后獲取到的序號，保存到conf目錄下的nid文件，比如我添加的機器對應序號為1)echo 1 > conf/nid
manager說明及參數
otter系統自帶了manager，所以簡化了一些admin管理上的操作成本，比如可以通過manager發布同步任務配置，接收同步任務反饋的狀態信息等。
同步配置管理
- 添加數據源
- canal解析配置
- 添加數據表
- 同步任務
同步狀態查詢
- 查詢延遲
- 查詢吞吐量
- 查詢同步進度
- 查詢報警&異常日志
用戶權限：
- ADMIN : 超級管理員
- OPERATOR : 普通用戶，管理某個同步任務下的同步配置，添加數據表，修改canal配置等
- ANONYMOUS : 匿名用戶，只能進行同步狀態查詢的操作.
https://github.com/alibaba/otter/wiki/Adminguide
具體配置參數
channel參數
-
同步一致性. ==>
基於數據庫反查 (簡單點說，就是強制反查數據庫，從binlog中拿到pk，直接反查對應數據庫記錄進行同步，回退到幾天前binlog進行消費時避免同步老版本的數據時可采用)
基於當前日志變更 (基於binlog/redolog解析出來的字段變更值進行同步，不做數據庫反查，推薦使用)小結：
基於數據庫反查(根據binlog反查數據庫)，基於當前日志變更(binlog數據)。針對數據庫反查，在延遲比較大時比較有效，可將最新的版本快速同步到目標，但會對源庫有壓力. -
同步模式. ==> 行記錄模式，列記錄模式。
行模式 (兼容otter3的處理方案，改變記錄中的任何一個字段，觸發整行記錄的數據同步，在目標庫執行merge sql)
列模式 (基於log中的具體變更字段，按需同步)小結：
行記錄模式特點：如果目標庫不存在記錄時，執行插入。
列記錄模式主要是變更哪個字段，只會單獨修改該字段。在雙Ａ同步時，為減少數據沖突，建議選擇列記錄模式。
雙A：雙主 且 會同時更改 一條記錄。 -
是否開啟數據一致性. ==> 請查看數據一致性文檔：Otter數據一致性
選擇 是，則會有下面兩個選項：
a. 一致性算法：單向回環補救【當前的選項】 ， 時間交集補救【目前開源版本暫未實現】
b. 一致性反查數據庫延遲閥值（s）
https://github.com/alibaba/otter/wiki/Manager%E9%85%8D%E7%BD%AE%E4%BB%8B%E7%BB%8D
補充：
需要處理一致性的業務場景：
（1）多地修改 (雙A機房)
（2）同一記錄，同時變更
同一記錄定義：具體到某一張表，某一條pk，某一字段
同時變更定義：A地寫入的數據在B地還未可見的一段時間范圍
使用 單向回環補救 策略 來解決 數據一致性，設置 一致性反查數據庫延遲閥值（s） 的原因：
解決方案：
反查數據庫同步 (以數據庫最新版本同步，解決交替性，
比如設置一致性反查數據庫延遲閥值為60秒，即當同步過程中發現數據延遲超過了60秒，就會基於PK反查一次數據庫，拿到當前最新值進行同步，減少交替性的問題)
https://github.com/alibaba/otter/wiki/Otter%E6%95%B0%E6%8D%AE%E4%B8%80%E8%87%B4%E6%80%A7
同步簡要信息 (點擊同步列表上的通道Channel名字的鏈接)
說明：
延遲時間 = 數據庫同步到目標庫成功時間 - 數據庫源庫產生變更時間， 單位秒. (由對應node節點定時推送配置)
最后同步時間 = 數據庫同步到目標庫最近一次的成功時間 (當前同步關注的相關表，同步到目標庫的最后一次成功時間)
最后位點時間 = 數據binlog消費最后一次更新位點的時間 (和同步時間區別：一個數據庫可能存在別的表的變更，不會觸發同步時間變更，但會觸發位點時間變更)
正常情況下，最后位點時間>=最后同步時間
pipeline參數
-
並行度. ==> 查看文檔：Otter調度模型，主要是並行化調度參數.(滑動窗口大小)
-
數據反查線程數. ==> 如果選擇了同步一致性為反查數據庫，在反查數據庫時的並發線程數大小
-
數據載入線程數. ==> 在目標庫執行並行載入算法時並發線程數大小
-
文件載入線程數. ==> 數據帶文件同步時處理的並發線程數大小
-
主站點. ==> 雙Ａ同步中的主站點設置
-
消費批次大小. ==> 獲取canal數據的batchSize參數
-
獲取批次超時時間. ==> 獲取canal數據的timeout參數
pipeline 高級設置
-
【開啟中】使用batch. ==> 是否使用jdbc batch提升效率，部分分布式數據庫系統不一定支持batch協議
- 【未開啟】跳過Select異常
-
【未開啟】跳過load異常. ==> 比如同步時出現目標庫主鍵沖突，開啟該參數后，可跳過數據庫執行異常
-
【ZOOKEEPER】仲裁器調度模式. ==> 查看文檔：Otter調度模型
-
【Stick】負載均衡算法. ==> 查看文檔：Otter調度模型
-
【自動選擇】傳輸模式. ==> 多個node節點之間的傳輸方式，RPC或HTTP. HTTP主要就是使用aria2c，如果測試環境不裝aria2c，可強制選擇為RPC
-
【開啟中】記錄selector日志. ==> 是否記錄簡單的canal抓取binlog的情況
-
【開啟中】記錄selector詳細日志. ==> 是否記錄canal抓取binlog的數據詳細內容
-
【開啟中】記錄load日志. ==> 是否記錄otter同步數據詳細內容
-
【未開啟】dryRun模式. ==> 只記錄load日志，不執行真實同步到數據庫的操作
-
【未開啟】支持ddl同步. ==> 是否同步ddl語句
-
【開啟中】是否跳過ddl異常. ==> 同步ddl出錯時，是否自動跳過
-
【未開啟】文件重復同步對比 ==> 數據帶文件同步時，是否需要對比源和目標庫的文件信息，如果文件無變化，則不同步，減少網絡傳輸量.
-
【未開啟】文件傳輸加密 ==> 基於HTTP協議傳輸時，對應文件數據是否需要做加密處理
-
【未開啟】啟用公網同步 ==> 每個node節點都會定義一個外部ip信息，如果啟用公網同步，同步時數據傳遞會依賴外部ip.
-
【開啟中】
==> 自定義數據同步的內容 -
【未開啟】跳過反查無記錄數據 ==> 反查記錄不存在時，是否需要進行忽略處理，不建議開啟.
-
【開啟中】啟用數據表類型轉化 ==> 源庫和目標庫的字段類型不匹配時，開啟此功能，可自動進行字段類型轉化
-
【開啟中】兼容字段新增同步 ==> 同步過程中，源庫新增了一個字段(必須無默認值)，而目標庫還未增加，是否需要兼容處理
-
自定義同步標記 ==> 級聯同步中屏蔽同步的功能.
Canal參數【說明：主要是管理canal鏈接到mysql/oracle獲取日志的相關參數等，業務方可不重點關注】
-
數據源信息
單庫配置： 10.20.144.34:3306;
多庫合並配置： 10.20.144.34:3306,10.20.144.35:3306; (逗號分隔)
主備庫配置：10.20.144.34:3306;10.20.144.34:3307; (分號分隔) -
connectionCharset ==> 獲取binlog時指定的編碼【與數據庫的字符集不同，譬如數據庫的字符集是UTF8MB4，但此處仍為UTF-8】
-
位點自定義設置 ==> 格式：{“journalName”:"",“position”:0,“timestamp”:0};
指定位置：{“journalName”:"",“position”:0};
指定時間：{“timestamp”:0}; -
內存存儲batch獲取模式 ==> MEMSIZE/ITEMSIZE，前者為內存控制，后者為數量控制. 針對MEMSIZE模式的內存大小計算 = 記錄數 * 記錄單元大小
內存存儲buffer記錄數
內存存儲buffer記錄單元大小 -
心跳SQL配置 ==> 可配置對應心跳SQL，如果配置 是否啟用心跳HA，當心跳SQL檢測失敗后，canal就會自動進行主備切換.
Node參數
-
機器名稱 ==> 自定義名稱，方便記憶
-
機器ip ==> 機器外部可訪問的ip，不能選擇127.0.0.1
-
機器端口 ==> 和manager/node之間RPC通訊的端口
-
下載端口 ==> 和node之間HTTP通訊的端口
-
外部Ip ==> node機器可以指定多IP，通過pipeline配置決定是否啟用
-
zookeeper集群 ==> 就近選擇zookeeper集群
Zookeeper集群參數
-
集群名字 ==> 自定義名稱，方便記憶
-
zookeeper集群 ==> zookeeper集群機器列表，逗號分隔，最后以分號結束
https://github.com/alibaba/otter/wiki/Manager%E9%85%8D%E7%BD%AE%E4%BB%8B%E7%BB%8D
映射關系列表【權重低的表先執行同步】
點擊查看打開映射關系信息頁面：
說明：
定義同步的源和目標的表信息 (注意:表名可以不同，可以定義數據庫分庫/分表)
Push權重 (對應的數字越大，同步會越后面得到同步，優先同步權重小的數據)
FileResolver (數據關聯文件的解析類，目前支持動態源碼推送,在目標jvm里動態編譯生效，不再需要起停同步任務)
EventProcessor (業務自定義的數據處理類，比如可以定義不需要同步status='ENABLE'的記錄或者根據業務改變同步的字段信息 簡單業務擴展，otter4新特性)
字段同步 (定義源和目標的字段映射，字段名和字段類型均可進行映射定義，類似於數據庫視圖定義功能 otter4新特性)
組合同步 (字段組的概念，字段組中的一個字段發生變更，會確保字段組中的3個字段一起同步到目標庫 otter4新特性)
多個字段決定一個圖片地址，變更文件字段中的任何一個字段，就會觸發FileResolver類解析，從而可以確保基於字段同步模式，也可以保證FileResolver能夠正常解析出文件 otter4重要的優化)
視圖映射
如何進入視圖編輯：
點擊下一步后，進入視圖編輯頁面：
說明：
映射規則配置頁面，可以選擇視圖模式為：include或exclude，代表正向匹配或者逆向排除.
視圖配置頁面，只支持存在的數據表(因為要獲取數據表結構，所以.*等正則的數據表不支持配置該功能)
視圖配置列表，左右選中列表會按照順序進行對應，做映射時需按照順序進行選擇.
舉個例子：
如果要排除表字段A的同步，則只需要選擇為exclude模式，然后視圖編輯頁面選擇左右皆選擇A字段即可，點擊保存.
字段組映射
首先解釋一下，需要字段組同步的需求.
（1）文件同步. 一條記錄對應的圖片，可能會有一個或者多個字段，比如會有image_path,image_version來決定圖片，所以我們可以定義這兩個字段為一組，只要滿足組內任意一個字段的變更，就會認為需要文件同步.
（2）數據上的組同步，比如國家，省份，城市，可能在數據庫為三個字段. 如果是雙A同步，兩地同時修改這些字段，但業務上可能在A地修改了國家為美國，在B地修改為省份為浙江，然后一同步，最終就會變成美國，浙江這樣的情況. 這種情況可以通過group來解決，將國家，省份，城市做一個group，組內任何一個字段發生了變更，其余字段會做為整體一起變更.
再來看一下配置：(點擊視圖編輯頁面的下一步，即可進入)
說明：
也可不配置視圖，單獨配置字段組，此時可選擇的字段即為當前所有字段(映射規則按照同名映射).
https://github.com/alibaba/otter/wiki/%E6%98%A0%E5%B0%84%E8%A7%84%E5%88%99%E9%85%8D%E7%BD%AE
吞吐量【1個點代表1分鍾的數據】
說明：
數據記錄統計 (insert/update/delete的變更總和，不區分具體的表，按表緯度的數據統計，可查看映射關系列表->每個映射關系右邊的行為曲線鏈接)
文件記錄統計
延遲時間
說明：
延遲時間的統計 = 數據庫同步到目標庫成功時間 - 數據庫源庫產生變更時間， 單位秒. (由對應node節點定時推送配置)
同步進度
說明：
mainstem狀態： 代表canal模塊當前的運行節點(也即是binlog解析的運行節點，解析會相對耗jvm內存)
position狀態： 當前同步成功的最后binlog位點信息 (包含鏈接的是數據庫ip/port，對應binlog的位置，對應binlog的變更時間此時間即是計算延遲時間的源庫變更時間)
同步進度： 每個同步批次會有一個唯一標識，可根據該唯一標示進行數據定位，可以查看每個批次的運行時間，找出性能瓶頸點
監控管理
說明：
監控項目
同步延遲，position超時(位點超過多少時間沒有更新) ， 一般業務方關心這些即可
異常 (同步運行過程中出現的異常，比如oracle DBA關心oracle系統ORA-的異常信息，mysql DBA關心mysql數據庫相關異常)
process超時(一個批次數據執行超過多少時間)，同步時間超時(數據超過多少時間沒有同步成功過)
閥值設置
1800@09:00-18:00 , 這例子是指定了早上9點到下午6點，報警閥值為1800.
發送對象
otterteam為otter團隊的標識，阿里內部使用了dragoon系統監控報警通知，如果外部系統可實現自己的報警通知機制
日志記錄
說明：
日志標題即為對應的監控規則定義的名字，可根據監控規則檢索對應的日志記錄
日志內容即為發送報警的信息
注意： otter4采用主動推送報警的模式，可以保證報警的及時性以及日志完整性(相比於日志文件掃描機制來說)
https://github.com/alibaba/otter/wiki/Manager%E4%BD%BF%E7%94%A8%E4%BB%8B%E7%BB%8D
數據合並
- insert + insert -> insert (數據遷移+數據增量場景)
- insert + update -> insert (update字段合並到insert)
- insert + delete -> delete
- update + insert -> insert (數據遷移+數據增量場景)
- update + update -> update
- update + delete -> delete
- delete + insert -> insert
- delete + update -> update (數據遷移+數據增量場景)
- delete + delete -> delete
數據入庫算法
入庫算法采取了按pk hash並行載入+batch合並的優化
-
打散原始數據庫事務，預處理數據，合並insert/update/delete數據(參見合並算法)，然后按照table + pk進行並行(相同table的數據，先執行delete,后執行insert/update，串行保證，解決唯一性約束數據變更問題)，相同table的sql會進行batch合並處理
-
提供table權重定義，根據權重定義不同支持"業務上類事務功能"，並行中同時有串行權重控制.
業務類事務描述：比如用戶的一次交易付款的流程，先產生一筆交易記錄，然后修改訂單狀態為已付款. 用戶對這事件的感知，是通過訂單狀態的已付款，然后進行查詢交易記錄。
所以，可以對同步進行一次編排： 先同步完交易記錄，再同步訂單狀態。 (給同步表定義權重，權重越高的表相對重要，放在后面同步，最后達到的效果可以保證業務事務可見性的功能，快的等慢的. )
https://github.com/alibaba/otter/wiki/Otter%E6%95%B0%E6%8D%AE%E5%85%A5%E5%BA%93%E7%AE%97%E6%B3%95
數據on-Fly，盡可能不落地，更快的進行數據同步. (開啟node loadBalancer算法，如果Node節點S+ETL落在不同的Node上，數據會有個網絡傳輸過程)；
node節點可以有failover / loadBalancer。
https://github.com/alibaba/otter/wiki/Introduction
otter學習（九）——常見報錯處理
一、binlog文件被清理：Could not find first log file name in binary log index file
1.報錯日志
2020-11-30 15:23:07.663 [destination = pathfinderpro , address = rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.R dsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position 2020-11-30 15:23:07.665 [destination = pathfinderpro , address = rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.R dsBinlogEventParserProxy - prepare to find start position just last position {"identity":{"slaveId":-1,"sourceAddress":{"address":"rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mysql-bin .009472","position":95437745,"serverId":2339357115,"timestamp":1606133836000}} 2020-11-30 15:23:07.712 [destination = pathfinderpro , address = rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.R dsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.009472,position=95437745,serverId=2339357115,gtid=,timestam p=1606133836000] cost : 49ms , the next step is binlog dump 2020-11-30 15:23:07.736 [destination = pathfinderpro , address = rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] ERROR c.a.o.canal.parse.inbound.mys ql.dbsync.DirectLogFetcher - I/O error while reading from client socket java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102) ~[canal.parse-1.1.3.jar:na] at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235) [canal.parse-1.1.3.jar:na] at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257) [canal.parse-1.1.3.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] 2020-11-30 15:23:07.736 [destination = pathfinderpro , address = rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.R dsBinlogEventParserProxy - dump address rm-bp1lc206u47icpvi5.mysql.rds.aliyuncs.com/10.10.0.238:3306 has an error, retrying. caused by java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
2.現象
查看 同步進度Tab頁，可以看到 pipeline的mainstem狀態 一直處於定位中狀態
3.問題排查
一般出現這個報錯，都是由於運維同學誤清空數據庫binlog文件導致，我們按下述步驟確定是否由於binlog文件被清理：
首先，查看當前同步的binlog位點：
其次：登錄數據庫查看binlog文件信息（查詢sql為：show master logs;）：
通過上述比對，發現數據庫的binlog被清理掉了，被清理前，otter標記的位點為mysql-bin.000245這個文件，被清理后，mysql重新從1開始生成binlog文件導致otter同步失敗
4.處理辦法：
1.清空掉otter的同步信息
2.檢查canal的同步位點配置
3.重新啟動otter同步
二、mysql大事務造成otter假死
1.報錯日志
無報錯日志
2.現象
channel狀態正常，mainstem狀態也是工作中，但是position信息里，position的信息一直不更新（超過半小時以上）
3.確認是否為大事務的方法
首先，登錄對應倉庫的數據庫，先查詢當前數據庫的binlog文件跑到哪里了
查詢sql為 show master logs;
然后查詢position信息里卡住的binlog文件信息，判斷是否產生了大事務
查詢sql為show binlog events in '你要查詢的binlog名稱';（例如：show binlog events in 'mysql-bin.000235';）
（下圖為查詢mysql-bin.000235這個binlog文件，發現由於源庫做切表處理，產生了大事務）
4.解決方法
從我們卡住的位點，依次往后面查詢binlog,找到這個大事務終結的位點。然后手動更新canal的位點信息，刪除同步記錄，然后重啟channel
a.清空掉otter的同步信息
b.檢查canal的同步位點配置
c.重新啟動otter同步
https://blog.csdn.net/u014355034/article/details/87974990
otter異常——zookeeper重新初始化
#跟運維的同事確認了下 測試環境的zookeeper前一天的確是出問題了（異常重啟過），而且進入zookeeper查找異常的路徑，的確不存在
至此，想到的是能否重新配置otter復制的mysqlbinlog文件和位置，讓其接着同步呢？
但是web管理頁面無法打開，無法配置。最后在github開源項目 otter的提問里面找到類似的問題：
https://github.com/alibaba/otter/issues/88
otter開發者回答到
otter會在zookeeper存儲一些節點信息，更換zookeeper后，需要復制節點數據，或者刪除數據庫中的channel、pipeline等表的數據內容
或者訪問 http://域名/system_reduction.htm，點擊一鍵修復
嘗試使用下面的連接 修復
訪問 http://otter_manager_ip:port/system_reduction.htm,在頁面上 會出現一個“ 一鍵補全 ”按鈕，點擊此按鈕即可
再將所有canal的journalName和position 換成出問題之前的相對較近的一個位置即可
至此 實現了 更換zookeeper或者重新初始化zookeeper的目的
http://blog.itpub.net/27000195/viewspace-2099256/
webx問題集
https://www.iteye.com/blog/welision-998812
我把zookeeper換成集群，就啟動不了了，請幫看下 #88
https://github.com/alibaba/otter/issues/88
【todo】manager日志報錯：
2020-05-18 21:47:13.098 [] WARN org.eclipse.jetty.servlet.ServletHandler - /URL=channelList.htm com.alibaba.citrus.service.pipeline.PipelineException: Failed to invoke Valve[#3/3, level 3]: com.alibaba.citrus.turbine.pipeline.valve.RenderTemplateValve#747f6c5a:RenderTemplateValve at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:161) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.invoke(PerformScreenValve.java:80) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PerformActionValve.invoke(PerformActionValve.java:73) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.ChooseValve.invoke(ChooseValve.java:98) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invokeBody(LoopValve.java:105) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invoke(LoopValve.java:83) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.CheckCsrfTokenValve.invoke(CheckCsrfTokenValve.java:123) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.otter.manager.web.webx.valve.AuthContextValve.invoke(AuthContextValve.java:117) ~[manager.web-4.2.18-SNAPSHOT.jar:na] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.AnalyzeURLValve.invoke(AnalyzeURLValve.java:126) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.SetLoggingContextValve.invoke(SetLoggingContextValve.java:66) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PrepareForTurbineValve.invoke(PrepareForTurbineValve.java:52) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.impl.WebxControllerImpl.service(WebxControllerImpl.java:43) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.impl.WebxRootControllerImpl.handleRequest(WebxRootControllerImpl.java:53) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.support.AbstractWebxRootController.service(AbstractWebxRootController.java:165) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.WebxFrameworkFilter.doFilter(WebxFrameworkFilter.java:152) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147) ~[citrus-webx-all-3.2.0.jar:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) ~[jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at com.alibaba.citrus.webx.servlet.SetLoggingContextFilter.doFilter(SetLoggingContextFilter.java:61) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147) ~[citrus-webx-all-3.2.0.jar:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) ~[jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) [jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) [jetty-security-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) [jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.Server.handle(Server.java:365) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) [jetty-http-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) [jetty-io-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) [jetty-io-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.7.v20120910.jar:8.1.7.v20120910] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: com.alibaba.citrus.service.template.TemplateNotFoundException: Could not find template "/screen/url=channelList" at com.alibaba.citrus.service.template.impl.TemplateServiceImpl.findTemplate(TemplateServiceImpl.java:279) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.template.impl.TemplateServiceImpl.writeTo(TemplateServiceImpl.java:224) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.RenderTemplateValve.renderTemplate(RenderTemplateValve.java:104) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.RenderTemplateValve.invoke(RenderTemplateValve.java:67) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] ... 53 common frames omitted
奇怪的報警：pid:321 delay_time:56792 seconds, but delay 1815 seconds no update #816
對於這種更新不頻繁的庫，開啟了canal的心跳之后就沒再出現過這個問題了。那么 開啟canal的心跳，需要什么條件呢？
https://github.com/alibaba/otter/issues/816
[todo]
2020-05-21 13:28:07.501 [] ERROR c.a.o.m.biz.config.pipeline.impl.PipelineServiceImpl - ERROR ## query pipelines has an exception! 2020-05-21 14:14:01.734 [] WARN c.a.o.s.a.i.setl.zookeeper.termin.WarningTerminProcess - nid:null[1:channel:java.util.concurrent.ExecutionException: java.lang.StackOverflowError at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.alibaba.otter.shared.arbitrate.impl.manage.ChannelArbitrateEvent.termin(ChannelArbitrateEvent.java:273) at com.alibaba.otter.shared.arbitrate.impl.manage.ChannelArbitrateEvent.stop(ChannelArbitrateEvent.java:147) at com.alibaba.otter.shared.arbitrate.impl.manage.ChannelArbitrateEvent.stop(ChannelArbitrateEvent.java:133) at com.alibaba.otter.manager.biz.config.channel.impl.ChannelServiceImpl$3.doInTransactionWithoutResult(ChannelServiceImpl.java:433) at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at com.alibaba.otter.manager.biz.config.channel.impl.ChannelServiceImpl.switchChannelStatus(ChannelServiceImpl.java:378) at com.alibaba.otter.manager.biz.config.channel.impl.ChannelServiceImpl.stopChannel(ChannelServiceImpl.java:458) at com.alibaba.otter.manager.biz.monitor.impl.RestartAlarmRecovery.processRecovery(RestartAlarmRecovery.java:102) at com.alibaba.otter.manager.biz.monitor.impl.RestartAlarmRecovery.access$100(RestartAlarmRecovery.java:44) at com.alibaba.otter.manager.biz.monitor.impl.RestartAlarmRecovery$1.run(RestartAlarmRecovery.java:137) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.StackOverflowError at com.alibaba.fastjson.JSONException.<init>(JSONException.java:33) at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:646) at com.alibaba.fastjson.JSON.parseObject(JSON.java:350) at com.alibaba.fastjson.JSON.parseObject(JSON.java:318) at com.alibaba.fastjson.JSON.parseObject(JSON.java:281) at com.alibaba.fastjson.JSON.parseObject(JSON.java:381) at com.alibaba.fastjson.JSON.parseObject(JSON.java:361) at com.alibaba.otter.shared.common.utils.JsonUtils.unmarshalFromByte(JsonUtils.java:55) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.processDelete(NormalTerminProcess.java:153) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.doProcess(NormalTerminProcess.java:135) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.process(NormalTerminProcess.java:63) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:86) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.ErrorTerminProcess.processChain(ErrorTerminProcess.java:96)
【todo】
2020-05-22 09:07:54.358 [DubboClientReconnectTimer-thread-1] WARN com.alibaba.dubbo.remoting.transport.AbstractClient - [DUBBO] client reconnect to 192.168.105.4:1099 find error . url: dubbo://192.168.105.4:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&threads=50, dubbo version: 2.5.3, current host: 172.16.2.103 com.alibaba.dubbo.remoting.RemotingException: client(url: dubbo://192.168.105.4:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&threads=50) failed to connect to server /192.168.105.4:1099, error message is:Connection refused at com.alibaba.dubbo.remoting.transport.netty.NettyClient.doConnect(NettyClient.java:124) ~[dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:280) ~[dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.AbstractClient$1.run(AbstractClient.java:145) ~[dubbo-2.5.3.jar:2.5.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_181] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_181] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) ~[netty-3.2.2.Final.jar:na] ... 3 common frames omitted 2020-05-22 09:07:54.358 [DubboClientReconnectTimer-thread-2] WARN com.alibaba.dubbo.remoting.transport.AbstractClient - [DUBBO] client reconnect to 192.168.105.4:1099 find error . url: dubbo://192.168.105.4:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&threads=50, dubbo version: 2.5.3, current host: 172.16.2.103 com.alibaba.dubbo.remoting.RemotingException: client(url: dubbo://192.168.105.4:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=8388608&send.reconnect=true&serialization=java&threads=50) failed to connect to server /192.168.105.4:1099, error message is:Connection refused at com.alibaba.dubbo.remoting.transport.netty.NettyClient.doConnect(NettyClient.java:124) ~[dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:280) ~[dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.AbstractClient$1.run(AbstractClient.java:145) ~[dubbo-2.5.3.jar:2.5.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_181] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_181] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) ~[netty-3.2.2.Final.jar:na] at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) ~[netty-3.2.2.Final.jar:na] ... 3 common frames omitted 2020-05-22 09:30:28.339 [DubboServerHandler-127.0.0.1:2088-thread-48] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem is not run any in node 2020-05-22 10:26:55.574 [DubboServerHandler-127.0.0.1:2088-thread-48] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem is not run any in node 2020-05-22 10:55:00.020 [DubboServerHandler-127.0.0.1:2088-thread-48] WARN c.a.o.shared.arbitrate.impl.setl.monitor.MainstemMonitor - mainstem is not run any in node
【todo】
2020-05-22 08:47:39.147 [] WARN org.eclipse.jetty.servlet.ServletHandler - /log_record_list.htm com.alibaba.citrus.service.pipeline.PipelineException: Failed to invoke Valve[#2/3, level 3]: com.alibaba.citrus.turbine.pipeline.valve.PerformTemplateScreenValve#686cf8ad:PerformTemplateScreenValve at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:161) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PerformActionValve.invoke(PerformActionValve.java:73) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.ChooseValve.invoke(ChooseValve.java:98) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invokeBody(LoopValve.java:105) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invoke(LoopValve.java:83) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.CheckCsrfTokenValve.invoke(CheckCsrfTokenValve.java:123) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.otter.manager.web.webx.valve.AuthContextValve.invoke(AuthContextValve.java:117) ~[manager.web-4.2.18-SNAPSHOT.jar:na] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.AnalyzeURLValve.invoke(AnalyzeURLValve.java:126) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.SetLoggingContextValve.invoke(SetLoggingContextValve.java:66) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PrepareForTurbineValve.invoke(PrepareForTurbineValve.java:52) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.impl.WebxControllerImpl.service(WebxControllerImpl.java:43) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.impl.WebxRootControllerImpl.handleRequest(WebxRootControllerImpl.java:53) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.support.AbstractWebxRootController.service(AbstractWebxRootController.java:165) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.WebxFrameworkFilter.doFilter(WebxFrameworkFilter.java:152) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147) ~[citrus-webx-all-3.2.0.jar:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) ~[jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at com.alibaba.citrus.webx.servlet.SetLoggingContextFilter.doFilter(SetLoggingContextFilter.java:61) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147) ~[citrus-webx-all-3.2.0.jar:3.2.0] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) ~[jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) [jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) [jetty-security-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) [jetty-servlet-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.Server.handle(Server.java:365) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) [jetty-http-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) [jetty-io-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) [jetty-io-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.7.v20120910.jar:8.1.7.v20120910] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.7.v20120910.jar:8.1.7.v20120910] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: com.alibaba.citrus.webx.WebxException: Failed to execute screen: LogRecordList at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.performScreenModule(PerformScreenValve.java:126) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.invoke(PerformScreenValve.java:74) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157) ~[citrus-webx-all-3.2.0.jar:3.2.0] ... 51 common frames omitted Caused by: org.springframework.dao.DataAccessResourceFailureException: SqlMapClient operation; SQL []; --- The error occurred while applying a parameter map. --- Check the getLogRecordCountWithPIdAndSearchKey-InlineParameterMap. --- Check the statement (query failed). --- Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 32,759 milliseconds ago. The last packet sent successfully to the server was 4,486 milliseconds ago.; nested exception is com.ibatis.common.jdbc.exception.NestedSQLException: --- The error occurred while applying a parameter map. --- Check the getLogRecordCountWithPIdAndSearchKey-InlineParameterMap. --- Check the statement (query failed). --- Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 32,759 milliseconds ago. The last packet sent successfully to the server was 4,486 milliseconds ago. at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:104) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:203) ~[spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:268) ~[spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.alibaba.otter.manager.biz.config.record.dal.ibatis.IbatisLogRecordDAO.getCount(IbatisLogRecordDAO.java:81) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.biz.config.record.impl.LogRecordServiceImpl.getCount(LogRecordServiceImpl.java:125) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.web.home.module.screen.LogRecordList.execute(LogRecordList.java:53) ~[manager.web-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.web.home.module.screen.LogRecordList$$FastClassByCGLIB$$288abc70.invoke(<generated>) ~[cglib-nodep-2.2.jar:na] at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53) ~[cglib-nodep-2.2.jar:na] at com.alibaba.citrus.service.moduleloader.impl.adapter.MethodInvoker.invoke(MethodInvoker.java:70) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.service.moduleloader.impl.adapter.DataBindingAdapter.executeAndReturn(DataBindingAdapter.java:41) ~[citrus-webx-all-3.2.0.jar:3.2.0] at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.performScreenModule(PerformScreenValve.java:111) ~[citrus-webx-all-3.2.0.jar:3.2.0] ... 53 common frames omitted Caused by: com.ibatis.common.jdbc.exception.NestedSQLException: --- The error occurred while applying a parameter map. --- Check the getLogRecordCountWithPIdAndSearchKey-InlineParameterMap. --- Check the statement (query failed). --- Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 32,759 milliseconds ago. The last packet sent successfully to the server was 4,486 milliseconds ago. at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:201) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForObject(MappedStatement.java:120) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:518) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:493) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:106) ~[ibatis-sqlmap-2.3.4.726.jar:na] at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:270) ~[spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:200) ~[spring-orm-3.1.2.RELEASE.jar:3.1.2.RELEASE] ... 62 common frames omitted Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 32,759 milliseconds ago. The last packet sent successfully to the server was 4,486 milliseconds ago. at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source) ~[na:na] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_181] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_181] at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3556) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3897) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.40.jar:5.1.40] at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) ~[commons-dbcp-1.4.jar:1.4] at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:185) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221) ~[ibatis-sqlmap-2.3.4.726.jar:na] at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189) ~[ibatis-sqlmap-2.3.4.726.jar:na] ... 68 common frames omitted Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008) ~[mysql-connector-java-5.1.40.jar:5.1.40] at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3466) ~[mysql-connector-java-5.1.40.jar:5.1.40] ... 80 common frames omitted
異常及解決辦法：
背景：啟動一個nid=5的node節點，但沒有在manger上配置這個node接口
報了下面的錯：
2020-05-28 15:54:24.851 [] ERROR c.a.otter.manager.biz.config.node.impl.NodeServiceImpl - ERROR ## couldn't query any node by nodeIds:[5] 2020-05-28 15:54:24.852 [] ERROR c.a.otter.manager.biz.config.node.impl.NodeServiceImpl - ERROR ## query nodes has an exception! 2020-05-28 15:54:24.853 [] ERROR c.a.o.m.b.r.interceptor.RemoteExceptionLoggerInterceptor - log exception message: com.alibaba.otter.manager.biz.common.exceptions.ManagerException: com.alibaba.otter.manager.biz.common.exceptions.ManagerException: couldn't query any node by nodeIds:[5] at com.alibaba.otter.manager.biz.config.node.impl.NodeServiceImpl.listByIds(NodeServiceImpl.java:189) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.biz.config.node.impl.NodeServiceImpl.findById(NodeServiceImpl.java:147) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.biz.config.node.impl.NodeServiceImpl.findById(NodeServiceImpl.java:47) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.biz.remote.impl.ConfigRemoteServiceImpl.onFindNode(ConfigRemoteServiceImpl.java:157) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] at com.alibaba.otter.manager.biz.remote.impl.ConfigRemoteServiceImpl$$FastClassByCGLIB$$3f77feba.invoke(<generated>) ~[cglib-nodep-2.2.jar:na] at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) ~[cglib-nodep-2.2.jar:na] at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) ~[spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.alibaba.otter.manager.biz.remote.impl.ConfigRemoteServiceImpl$$EnhancerByCGLIB$$88ecacc9.onFindNode(<generated>) ~[cglib-nodep-2.2.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181] at com.alibaba.otter.shared.communication.core.impl.AbstractCommunicationEndpoint.acceptEvent(AbstractCommunicationEndpoint.java:72) ~[shared.communication-4.2.18-SNAPSHOT.jar:na] at com.alibaba.dubbo.common.bytecode.Wrapper0.invokeMethod(Wrapper0.java) [na:2.5.3] at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) [dubbo-2.5.3.jar:2.5.3] at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] Caused by: com.alibaba.otter.manager.biz.common.exceptions.ManagerException: couldn't query any node by nodeIds:[5] at com.alibaba.otter.manager.biz.config.node.impl.NodeServiceImpl.listByIds(NodeServiceImpl.java:173) ~[manager.biz-4.2.18-SNAPSHOT.jar:na] ... 27 common frames omitted
解決辦法：
在manger配置頁中配置一個序號為5的node 。
代碼：
public List<Node> listByIds(Long... identities) { List<Node> nodes = new ArrayList<Node>(); try { List<NodeDO> nodeDos = null; if (identities.length < 1) { nodeDos = nodeDao.listAll(); if (nodeDos.isEmpty()) { logger.debug("DEBUG ## couldn't query any node, maybe hasn't create any channel."); return nodes; } } else { nodeDos = nodeDao.listByMultiId(identities); if (nodeDos.isEmpty()) { String exceptionCause = "couldn't query any node by nodeIds:" + Arrays.toString(identities); logger.error("ERROR ## " + exceptionCause); throw new ManagerException(exceptionCause); } } // 驗證zk的node信息 List<Long> nodeIds = arbitrateManageService.nodeEvent().liveNodes(); for (NodeDO nodeDo : nodeDos) { if (nodeIds.contains(nodeDo.getId())) { nodeDo.setStatus(NodeStatus.START); } else { nodeDo.setStatus(NodeStatus.STOP); } } nodes = doToModel(nodeDos); } catch (Exception e) { logger.error("ERROR ## query nodes has an exception!"); throw new ManagerException(e); } return nodes; }
com.alibaba.otter.manager.biz.config.node.impl.NodeServiceImpl#listByIds
同時node在啟動時，也會報錯：
Exception in thread "main" java.lang.ExceptionInInitializerError at com.alibaba.otter.node.deployer.OtterLauncher.main(OtterLauncher.java:39) Caused by: com.alibaba.otter.shared.common.model.config.ConfigException: ERROR ## Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'otterController' defined in URL [jar:file:/home/admin/node/lib/node.etl-4.2.17.jar!/spring/otter-node-common.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'arbitrateEventService' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Cannotresolve reference to bean 'extractRpcEvent' while setting bean property 'delegate' with key [TypedStringValue: value [RPC], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractRpcEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent]: Constructor threw exception; nested exception is com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:527) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:609) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:469) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83) at com.alibaba.otter.node.etl.OtterContextLocator$1.<init>(OtterContextLocator.java:39) at com.alibaba.otter.node.etl.OtterContextLocator.<clinit>(OtterContextLocator.java:39) at com.alibaba.otter.node.deployer.OtterLauncher.main(OtterLauncher.java:39) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'arbitrateEventService' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Cannot resolve reference to bean 'extractRpcEvent' while setting bean property 'delegate' with key [TypedStringValue: value [RPC], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractRpcEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent]: Constructor threw exception; nested exception is com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:527) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByName(AbstractAutowireCapableBeanFactory.java:1136) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1086) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) ... 13 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Cannot resolve reference to bean 'extractRpcEvent' while setting bean property 'delegate' with key [TypedStringValue: value [RPC], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractRpcEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent]: Constructor threw exception; nested exception is com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedMap(BeanDefinitionValueResolver.java:378) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:161) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByName(AbstractAutowireCapableBeanFactory.java:1136) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1086) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) ... 21 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'extractRpcEvent' defined in URL [jar:file:/home/admin/node/lib/shared.arbitrate-4.2.17.jar!/spring/otter-arbitrate-event.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent]: Constructor threw exception; nested exception is com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:997) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ... 35 more Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent]: Constructor threw exception; nested exception is com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:162) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:76) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:990) ... 43 more Caused by: com.google.common.collect.ComputationException: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found! at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:889) at com.alibaba.otter.shared.arbitrate.impl.zookeeper.ZooKeeperClient.getInstance(ZooKeeperClient.java:61) at com.alibaba.otter.shared.arbitrate.impl.zookeeper.ZooKeeperClient.getInstance(ZooKeeperClient.java:54) at com.alibaba.otter.shared.arbitrate.impl.setl.rpc.ExtractRpcArbitrateEvent.<init>(ExtractRpcArbitrateEvent.java:45) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147) ... 45 more Caused by: com.alibaba.otter.shared.common.model.config.ConfigException: nid:5 in manager[otter-manager.zkh-uat.svc.cluster.local:1099]is not found!
2020-05-29 09:05:07.002 [DbLoadAction] ERROR com.alibaba.otter.node.etl.load.loader.db.DbLoadAction - ##load phase two failed! com.alibaba.otter.node.etl.load.exception.LoadException: org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException forSQL [insert into `zaf-security-admin2`.`t_customer_sales_force`(`f_partner` , `f_customer_id` , `f_vkorg` , `f_vtweg` , `f_spart` , `f_customer_service_staff_id` , `f_saler_id` , `f_state_code` , `f_csf_id`) values (? , ? , ? , ? , ? , ? , ? , ? , ?) on duplicate key update `f_partner`=values(`f_partner`) , `f_customer_id`=values(`f_customer_id`) , `f_vkorg`=values(`f_vkorg`) , `f_vtweg`=values(`f_vtweg`) , `f_spart`=values(`f_spart`) , `f_customer_service_staff_id`=values(`f_customer_service_staff_id`) , `f_saler_id`=values(`f_saler_id`) , `f_state_code`=values(`f_state_code`) , `f_csf_id`=values(`f_csf_id`)]; SQL state [HY000]; error code [1364]; Field 'f_cso_id' doesn't have a default value; nested exception is java.sql.SQLException: Field 'f_cso_id' doesn't have a default value at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:603) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:812) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:868) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$DbLoadWorker$2.doInTransaction(DbLoadAction.java:625) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$DbLoadWorker.doCall(DbLoadAction.java:617) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$DbLoadWorker.call(DbLoadAction.java:545) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.doTwoPhase(DbLoadAction.java:462) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.doLoad(DbLoadAction.java:275) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction.load(DbLoadAction.java:161) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$$FastClassByCGLIB$$d932a4cb.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) at com.alibaba.otter.node.etl.load.loader.db.DbLoadAction$$EnhancerByCGLIB$$80fd23c2.load(<generated>) at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:198) at com.alibaba.otter.node.etl.load.loader.db.DataBatchLoader$2.call(DataBatchLoader.java:189) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: Field 'f_cso_id' doesn't have a default value at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2073) at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2009) at com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5098) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1994) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:818) at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:1) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:587) ... 21 more :----------------- - PairId: 104 , TableId: 125 , EventType : I , Time : 1590667984000 - Consistency : , Mode : ----------------- ---Pks EventColumn[index=5,columnType=-5,columnName=f_csf_id,columnValue=244012,isNull=false,isKey=true,isUpdate=true] ---oldPks ---Columns EventColumn[index=7,columnType=12,columnName=f_partner,columnValue=A79347,isNull=false,isKey=false,isUpdate=true] EventColumn[index=8,columnType=-5,columnName=f_customer_id,columnValue=2138596,isNull=false,isKey=false,isUpdate=true] EventColumn[index=9,columnType=12,columnName=f_vkorg,columnValue=1000,isNull=false,isKey=false,isUpdate=true] EventColumn[index=10,columnType=12,columnName=f_vtweg,columnValue=01,isNull=false,isKey=false,isUpdate=true] EventColumn[index=11,columnType=12,columnName=f_spart,columnValue=02,isNull=false,isKey=false,isUpdate=true] EventColumn[index=12,columnType=12,columnName=f_customer_service_staff_id,columnValue=8168,isNull=false,isKey=false,isUpdate=true] EventColumn[index=13,columnType=12,columnName=f_saler_id,columnValue=2533,isNull=false,isKey=false,isUpdate=true] EventColumn[index=17,columnType=4,columnName=f_state_code,columnValue=1,isNull=false,isKey=false,isUpdate=true] ---Sql insert into `zaf-security-admin2`.`t_customer_sales_force`(`f_partner` , `f_customer_id` , `f_vkorg` , `f_vtweg` , `f_spart` , `f_customer_service_staff_id` ,`f_saler_id` , `f_state_code` , `f_csf_id`) values (? , ? , ? , ? , ? , ? , ? , ? , ?) on duplicate key update `f_partner`=values(`f_partner`) , `f_customer_id`=values(`f_customer_id`) , `f_vkorg`=values(`f_vkorg`) , `f_vtweg`=values(`f_vtweg`) , `f_spart`=values(`f_spart`) , `f_customer_service_staff_id`=values(`f_customer_service_staff_id`) , `f_saler_id`=values(`f_saler_id`) , `f_state_code`=values(`f_state_code`) , `f_csf_id`=values(`f_csf_id`)
Channel管理 > Pipeline管理 > 映射關系列表
一個映射關系中，不能出現出現將相同數據源的一張表，分別映射到大於1個數據源的的表中。否則會出現數據異常：數據不同正常同步，但manager和node節點 都沒有報錯信息
2020-07-31 15:59:51.251 [pipelineId = 4,taskName = ExtractWorker] ERROR com.alibaba.otter.node.etl.extract.ExtractTask - [4] extractwork executor is error! data:EtlEventData[currNid=1,nextNid=1,desc=[MemoryPipeKey[identity=Identity[channelId=4,pipelineId=4,processId=1],time=1596182391249,dataType=DB_BATCH]],processId=1,startTime=1596182391249,endTime=<null>,firstTime=1596182391000,batchId=56,number=1,size=<null>,exts=<null>,pipelineId=4] com.alibaba.otter.node.etl.extract.exceptions.ExtractException: eventData after viewExtractor has no pks , pls check! identity:Identity[channelId=4,pipelineId=4,processId=1], new eventData:EventData[tableId=29,tableName=product,schemaName=pathfinderdb,eventType=UPDATE,executeTime=1596182391000,oldKeys=[],keys=[],columns=[EventColumn[index=118,columnType=4,columnName=isNationalSales,columnValue=1,isNull=false,isKey=false,isUpdate=true]],size=1636,pairId=-1,sql=<null>,ddlSchemaName=<null>,syncMode=<null>,syncConsistency=<null>,remedy=false,hint=<null>,withoutSchema=false] 2020-07-31 15:59:51.255 [pipelineId = 4,taskName = ExtractWorker] WARN c.a.o.s.a.i.setl.zookeeper.termin.WarningTerminProcess - nid:1[4:setl:com.alibaba.otter.node.etl.extract.exceptions.ExtractException: eventDataafter viewExtractor has no pks , pls check! identity:Identity[channelId=4,pipelineId=4,processId=1], new eventData:EventData[tableId=29,tableName=product,schemaName=pathfinderdb,eventType=UPDATE,executeTime=1596182391000,oldKeys=[],keys=[],columns=[EventColumn[index=118,columnType=4,columnName=isNationalSales,columnValue=1,isNull=false,isKey=false,isUpdate=true]],size=1636,pairId=-1,sql=<null>,ddlSchemaName=<null>,syncMode=<null>,syncConsistency=<null>,remedy=false,hint=<null>,withoutSchema=false] ]
Otter表同步時，無主鍵時會報錯
if (CollectionUtils.isEmpty(eventData.getKeys())) { // 無主鍵，報錯 throw new ExtractException( String.format("eventData after viewExtractor has no pks , pls check! identity:%s, new eventData:%s", dbBatch.getRowBatch().getIdentity().toString(), eventData.toString())); } com.alibaba.otter.node.etl.extract.extractor.ViewExtractor#extract
otter在zk上的存儲路徑
com.alibaba.otter.shared.arbitrate.impl.ArbitrateConstants
{ /** * otter的根節點 */ public String NODE_OTTER_ROOT = "/otter"; /** * otter的node機器的根節點 */ public String NODE_NID_ROOT = NODE_OTTER_ROOT + "/node"; /** * otter中node節點的format格式,接受nid做為參數 */ public String NODE_NID_FORMAT = NODE_NID_ROOT + "/{0}"; /** * otter的channel的根節點 */ public String NODE_CHANNEL_ROOT = NODE_OTTER_ROOT + "/channel"; /** * otter中channel節點的format格式,接受channelId做為參數 */ public String NODE_CHANNEL_FORMAT = NODE_CHANNEL_ROOT + "/{0}"; /** * otter中pipeline節點的format格式,接受channelId,pipelineId做為參數 */ public String NODE_PIPELINE_FORMAT = NODE_CHANNEL_FORMAT + "/{1}"; /** * otter的remedy的根節點 */ public String NODE_REMEDY_ROOT = NODE_PIPELINE_FORMAT + "/remedy"; /** * otter的process的根節點 */ public String NODE_PROCESS_ROOT = NODE_PIPELINE_FORMAT + "/process"; /** * otter中process節點的format格式,接受channelId,pipelineId,processId做為參數 */ public String NODE_PROCESS_FORMAT = NODE_PROCESS_ROOT + "/{2}"; /** * otter的termin信號的根節點 */ public String NODE_TERMIN_ROOT = NODE_PIPELINE_FORMAT + "/termin"; /** * otter中termin節點的format格式,接受channelId,pipelineId,processId做為參數 */ public String NODE_TERMIN_FORMAT = NODE_TERMIN_ROOT + "/{2}"; /** * otter的lock根節點 */ public String NODE_LOCK_ROOT = NODE_PIPELINE_FORMAT + "/lock"; /** * otter的load的lock節點 */ public String NODE_LOCK_LOAD = "load"; /** * 主導線程的狀態節點,為pipeline的子節點 */ public String NODE_MAINSTEM = "mainstem"; /** * select完成狀態的節點,為process的子節點 */ public String NODE_SELECTED = "selected"; /** * extract完成狀態的節點,為process的子節點 */ public String NODE_EXTRACTED = "extracted"; /** * transform完成狀態的節點,為process的子節點 */ public String NODE_TRANSFORMED = "transformed"; /** * load完成狀態的節點,為process的子節點 */ public String NODE_LOADED = "loaded"; /** * 在logback的配置文件中定義好的按照各個pipeline進行日志文件輸出的鍵值. */ public String splitPipelineLogFileKey = "otter"; }
Otter遠程調試
https://www.cnblogs.com/yanshaoshuai/p/12060938.html
克隆並編譯otter
https://www.cnblogs.com/yanshaoshuai/p/12060468.html
Canal和Otter介紹和使用
https://www.cnblogs.com/yanshaoshuai/p/11987253.html
Canal和Otter討論二(原理與實踐)
https://www.cnblogs.com/yanshaoshuai/p/11987281.html