canal常見問題排查(三)

數據庫發生改變收不到訂閱消息

查看canal是否有權限

select * from mysql.user where user='{canal-服務連接數據庫的賬戶名字}' 

 

 

 

如果沒有或者Select_priv不是Y

grant select, replication slave,replication client on *.* to '{用戶名}'@'%';

刷新權限

flush pricileges;

查看當前canal讀取的binlog日志文件和位置

cat canal安裝目錄/conf/example/meta.dat 

cat canal安裝目錄/conf/example/meta.dat 

注:這個是可以編輯的 查看是否有這個binlog日志文件 我之前就是測試環境測試過 記錄的是測試環境讀取到的然后在新的庫就沒有這個日志文件導致監聽不到

position為當前讀取的日志指針

 

 

 數據庫執行

show binary logs可以查看當前庫所有binlog日志文件

 

 show master status可以看到當前寫入日志的文件以及寫入指針位置

 

DDL報錯

Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:canal.classes,3 vs 2

因為canal默認是通過h2數據庫存儲的表結構變動，比如我們當前binlog日志position 10-100  這個時候classes的字段是2個 id,name  但是101之后就加了個字段sex

我們因為某些原因導致我們canalService阻塞讀取位置一直10 雖然表結構變動 及時刷入到了到h2數據庫,但是重啟啟動canalService 刷入的是最新的表結構 id,name,sex 然后我們繼續從10開始去讀取表結構DDL對不上報錯

解決方式就是將內存數據庫改為物理持久化 canal支持數據庫配置

編輯/conf/canal.properties 

1.注釋掉默認的h2配置

#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#tsdb走mysql
canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

#默認h2
#canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;

2.建表

{canal-server目錄}/conf/spring/tsdb/sql/create_table.sql 有執行腳本

2.打開mysql數據庫配置

url改為mysql的

canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
canal.instance.tsdb.dbUsername = root
canal.instance.tsdb.dbPassword = 868978

 后面我們再修改結構，canal-server收到binlog日志就會存到數據庫里面

無權限異常

在example下的install.properties 加上黑名單

同理 其他不相關表sql異常可以過濾 多個，號隔開

# table black regex 黑名單
canal.instance.filter.black.regex=mysql\..*,.*\\.__drds__systable__leadership__

 

2021-04-13 16:29:44.504 [destination = kuaihe_db_test , address = rm-2zeqc826391dt0lk2.mysql.rds.aliyuncs.com/192.168.10.4:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:kuaihe_db_test[com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`ucenter_test_bkuz_0000`.`__drds__systable__leadership__`
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`ucenter_test_bkuz_0000`.`__drds__systable__leadership__`
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`ucenter_test_bkuz_0000`.`__drds__systable__leadership__`
Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'kuaihe'@'192.168.20.5' for table '__drds__systable__leadership__', sqlState=42000, sqlStateMarker=#]
 with command: show create table `ucenter_test_bkuz_0000`.`__drds__systable__leadership__`
        at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.query(MysqlQueryExecutor.java:61)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.query(MysqlConnection.java:106)
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.TableMetaCache.getTableMeta(TableMetaCache.java:177)
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.getTableMeta(LogEventConvert.java:950)
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:479)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:280)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:246)
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 

021-04-13 16:24:35.234 [destination = kuaihe_db_test , address = rm-2zeqc826391dt0lk2.mysql.rds.aliyuncs.com/192.168.10.4:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:kuaihe_db_test[com.alibaba.otter.canal.parse.exception.CanalParseException: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'kuaihe'@'192.168.20.5' for table 'slow_log', sqlState=42000, sqlStateMarker=#]
 with command: show create table `mysql`.`event`;show create table `mysql`.`func`;show create table `mysql`.`general_log`;show create table `mysql`.`help_category`;show create table `mysql`.`help_keyword`;show create table `mysql`.`help_relation`;show create table `mysql`.`help_topic`;show create table `mysql`.`proc`;show create table `mysql`.`slow_log`;show create table `mysql`.`time_zone`;show create table `mysql`.`time_zone_leap_second`;show create table `mysql`.`time_zone_name`;show create table `mysql`.`time_zone_transition`;show create table `mysql`.`time_zone_transition_type`;
Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'kuaihe'@'192.168.20.5' for table 'slow_log', sqlState=42000, sqlStateMarker=#]
 with command: show create table `mysql`.`event`;show create table `mysql`.`func`;show create table `mysql`.`general_log`;show create table `mysql`.`help_category`;show create table `mysql`.`help_keyword`;show create table `mysql`.`help_relation`;show create table `mysql`.`help_topic`;show create table `mysql`.`proc`;show create table `mysql`.`slow_log`;show create table `mysql`.`time_zone`;show create table `mysql`.`time_zone_leap_second`;show create table `mysql`.`time_zone_name`;show create table `mysql`.`time_zone_transition`;show create table `mysql`.`time_zone_transition_type`;
        at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.queryMulti(MysqlQueryExecutor.java:109)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.queryMulti(MysqlConnection.java:111)
        at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.dumpTableMeta(DatabaseTableMeta.java:232)
        at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.rollback(DatabaseTableMeta.java:182)
        at com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.processTableMeta(AbstractMysqlEventParser.java:137)
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:200)
        at java.lang.Thread.run(Thread.java:748)

使用admin，啟動不了服務和instance都是停止

因為instance是基於配置，啟動服務忘記加local ./startup.sh local