Canal1.1.4同步數據至Es時間格式問題

環境描述: 

　　centos7、mysql5.6、canal1.1.4、es6.6

參考博客:https://blog.csdn.net/weixin_41546244/article/details/108381219

1、下載canal源碼

 

2、打開idea修改源碼（注意修改：ESSyncUtil.class 中的帶有Util.timeZone的時間格式），如下圖，共有8處，都需修改。

 3、找到編譯后的包，將其傳入/data/soft/canal1.4/canal/adapter/plugin下面

client-adapter.elasticsearch-1.1.4-jar-with-dependencies.jar

4、重啟adapter,就可以了....

-------------------心累，整這個整了好久--------------

FAQ:

錯誤一：

2020-11-07 22:57:18.662 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es.service.ESSyncService - sync error, es index: t_ad_account_rebate, DML : Dml{destination='instance_t_ad_account_rebate', database='skynetods', table='t_ad_account_rebate', type='INSERT', es=1604761038000, ts=1604761038584, sql='', data=[{id=409, exten_interface=微信朋友圈2, exten_account="", third_company=null, open_account_company=null, rebate=1.13, is_delete=0, create_time=2020-01-01 00:00:00.0, update_time=null, device=2}], old=null}
2020-11-07 22:57:18.664 [pool-2-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
        at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:110)
        at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:58)
        at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:169)
        at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:148)
        at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:201)
        at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
        at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.insert(ESTemplate.java:74)
        at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.singleTableSimpleFiledInsert(ESSyncService.java:442)
        at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:133)
        at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:93)
        ... 11 common frames omitted
2020-11-07 22:57:18.665 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - Outer adapter sync failed!  Error sync but ACK!

解決方案：這種就是你對應的canal目錄下es的配置文件有問題。我監察很久才發現

錯誤二：

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse [cardtime_date]]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Invalid format: "2019-10-21T08:44:31-05:00" is malformed at "T08:44:31-05:00"]];
        at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:65)
        at com.alibaba.otter.canal.client.adapter.es.service.ESEtlService.executeSqlImport(ESEtlService.java:80)
        at com.alibaba.otter.canal.client.adapter.support.AbstractEtlService.lambda$importData$1(AbstractEtlService.java:91)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

解決方案：這個就是時間格式問題了。。。可以按照上面的方法修改

錯誤三："data":null

{"data":null,"database":"","es":1606317332000,"id":1,"isDdl":false,"mysqlType":null,"old":null,"pkNames":null,"sql":"update XXXXXXX","sqlType":null,"table":"TABLE","ts":1606318913663,"type":"QUERY"}

解決方案: binlog_format 沒有指定為ROW

錯誤四：

錯誤描述：mysql到es的時間格式多14小時

解決方案：設置mysql的時區為東八區　　

set global time_zone = '+8:00' ;
set time_zone = '+8:00' ;
flush privileges ;

 錯誤五：

錯誤描述: 使用canal同步mysql數據至es，left join的sql語句不能包含子查詢語句，否則啟動adapter失敗....

 錯誤六：

錯誤描述：
    canal同步mysql數據至es規則：
    1、單表同步可以
    2、同庫/異庫兩張表可以
    3、同庫三張表可以(但是不能包含子查詢)