CentOS 7中使用canal實現MySQL 8同步數據到ElasticSearch 7.8中

canal簡介

canal是阿里巴巴開源的MySQL binlog 增量訂閱&消費組件。

canal [kə'næl]，譯意為水道/管道/溝渠，主要用途是基於 MySQL 數據庫增量日志解析，提供增量數據訂閱和消費。

canal的GitHub地址： GitHub - alibaba/canal: 阿里巴巴 MySQL binlog 增量訂閱&消費組件

canal下載與使用

使用canal前請先確保安裝了mysql、elasticsearch、jdk

本文章使用的各軟件版本為MySQL 8.0.25，ElasticSearch 7.8.0，java-1.8.0-openjdk.x86_64

到canal github項目頁面下載1.1.5版本的文件：Releases · alibaba/canal · GitHub

建議下載以下四個。其中下載源碼是為了解決canal.adapter中的一個依賴沖突，要保存在自己開發用的電腦里，后面會講到它。

canal的各個組件的用途各不相同，下面分別介紹下：
canal-deploy：用於監聽MySQL的binlog，是一個偽裝的MySQL從庫，只負責從MySQL主庫接收數據，不做處理。
canal-adapter：canal的客戶端，從canal-deploy中獲取數據，然后同步數據到目標數據源，我們用它將數據存儲到ElasticSearch中。
canal-admin：為canal提供整體配置管理、節點運維等面向運維的功能，提供相對友好的WebUI操作界面，方便更多用戶快速和安全的操作。

mysql的配置

先配置mysql數據庫，打開binlog寫入功能，設置binlog-format為ROW。我的電腦上，配置文件路徑為 /etc/my.cnf

vi /etc/my.cnf

 

 

 增加或者修改以下內容

[mysqld]
## 設置server_id，同一局域網中需要唯一
server_id=101
## 指定不需要同步的數據庫名稱
binlog-ignore-db=mysql
## 開啟二進制日志功能
log-bin=mall-mysql-bin
## 設置二進制日志使用內存大小（事務）
binlog_cache_size=1M
## 設置使用的二進制日志格式（mixed,statement,row）
binlog_format=row
## 二進制日志過期清理時間。默認值為0，表示不自動清理。
expire_logs_days=7
## 跳過主從復制中遇到的所有錯誤或指定類型的錯誤，避免slave端復制中斷。
## 如：1062錯誤是指一些主鍵重復，1032錯誤是因為主從數據庫數據不一致
slave_skip_errors=1062

配置完成后重新啟動MySQL，重啟后執行下面的SQL語句查看binlog是否啟用：

show variables like '%log_bin%';

 

 

 再查看binlog模式：

 

 

接下來需要創建一個擁有從庫權限的賬號，用於訂閱binlog，這里創建的賬號為canal:canal；

CREATE USER canal IDENTIFIED BY 'canal';  

GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';

FLUSH PRIVILEGES;

現在創建一個數據庫：

create database economic_enterprise_dev;

 

 

創建一張student表：

CREATE TABLE `economic_enterprise_dev`.`student` (  
  `id` VARCHAR(32) NOT NULL,
  `name` VARCHAR(100),
  `grade` TINYINT(1) DEFAULT 1,
  `class` TINYINT(1) DEFAULT 0,
  `birthday` DATE,
  PRIMARY KEY (`id`) 
);

可以隨便插入幾條數據，也可以不添加。mysql 到此配置完了。

canal-deployer的配置與使用

解壓deployer：

mkdir /usr/local/canal_deployer

tar -zxvf ./canal.deployer-1.1.5.tar.gz -C /usr/local/canal_deployer

解壓后目錄結構為

.
├── bin
│   ├── restart.sh
│   ├── startup.bat
│   ├── startup.sh
│   └── stop.sh
├── conf
│   ├── canal_local.properties
│   ├── canal.properties
│   ├── example
│   │   └── instance.properties
│   ├── logback.xml
│   ├── metrics
│   │   └── Canal_instances_tmpl.json
│   └── spring
├── lib
├── logs
└── plugin

修改配置文件conf/example/instance.properties

vi /usr/local/canal_deployer/conf/example/instance.properties

修改下面幾項配置：

# 需要同步數據的MySQL地址
canal.instance.master.address=127.0.0.1:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=
# 用於同步數據的數據庫賬號
canal.instance.dbUsername=canal
# 用於同步數據的數據庫密碼
canal.instance.dbPassword=canal
# 數據庫連接編碼
canal.instance.connectionCharset = UTF-8
# 需要訂閱binlog的表過濾正則表達式
canal.instance.filter.regex=.*\\..*

啟動deployer：

/usr/local/canal_deployer/bin/startup.sh

啟動成功后查看deployer的日志：

tail -f /usr/local/canal_deployer/logs/canal/canal.log

 

 

canal-adaptor的使用和配置

 解壓deployer：

mkdir /usr/local/canal_adapter

tar -zxvf ./canal.adapter-1.1.5.tar.gz -C /usr/local/canal_adapter

解壓后目錄結構為

.
├── bin
│   ├── restart.sh
│   ├── startup.bat
│   ├── startup.sh
│   └── stop.sh
├── conf
│   ├── application.yml
│   ├── bootstrap.yml
│   ├── es6
│   │   ├── biz_order.yml
│   │   ├── customer.yml
│   │   └── mytest_user.yml
│   ├── es7
│   │   ├── biz_order.yml
│   │   ├── customer.yml
│   │   └── mytest_user.yml
│   ├── hbase
│   │   └── mytest_person2.yml
│   ├── kudu
│   │   └── kudutest_user.yml
│   ├── logback.xml
│   ├── META-INF
│   │   └── spring.factories
│   └── rdb
│       └── mytest_user.yml
├── lib
├── logs
└── plugin
    ├── client-adapter.es6x-1.1.5-jar-with-dependencies.jar
    ├── client-adapter.es7x-1.1.5-jar-with-dependencies.jar
    ├── client-adapter.hbase-1.1.5-jar-with-dependencies.jar
    ├── client-adapter.logger-1.1.5-jar-with-dependencies.jar
    ├── client-adapter.rdb-1.1.5-jar-with-dependencies.jar
    ├── connector.kafka-1.1.5-jar-with-dependencies.jar
    ├── connector.rabbitmq-1.1.5-jar-with-dependencies.jar
    ├── connector.rocketmq-1.1.5-jar-with-dependencies.jar
    └── connector.tcp-1.1.5-jar-with-dependencies.jar

修改配置文件conf/application.yml

vi /usr/local/canal_adapter/conf/application.yml

修改下面幾項配置

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
    
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/economic_enterprise_dev?useUnicode=true
      username: root
      password: 123456
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: es7
        hosts: http://127.0.0.1:9200
        properties:
          mode: rest
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: elasticsearch

在canal-adapter/conf/es7/目錄下新建一個配置文件

vi /usr/local/canal_adapter/conf/es7/student.yml

寫入以下內容

dataSourceKey: defaultDS
destination: student
groupId: g1
esMapping:
  _index: student
  _type: _doc
  _id: _id
  sql: "SELECT 
        id as _id,
        name,
        grade,
        class,
        birthday
        from student"
  etlCondition: "where birthday>={}"
  commitBatch: 3000

啟動adapter：

/usr/local/canal_adapter/bin/startup.sh

查看adapter日志：

tail -100f  /usr/local/canal_adapter/logs/adapter/adapter.log

看到有以下報錯：

2021-06-11 18:11:29.223 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2021-06-11 18:11:29.223 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2021-06-11 18:11:29.224 [main] INFO  c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/canal_adapter/plugin
2021-06-11 18:11:29.245 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2021-06-11 18:11:29.441 [main] INFO  c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## Start loading es mapping config ... 
2021-06-11 18:11:29.493 [main] INFO  c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## ES mapping config loaded
2021-06-11 18:11:29.747 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
    at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
    at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.loadAdapter(CanalAdapterLoader.java:225) [client-adapter.launcher-1.1.5.jar:na]
    at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:56) [client-adapter.launcher-1.1.5.jar:na]
    at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) [client-adapter.launcher-1.1.5.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:126) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
    at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:264) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:182) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:144) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:400) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:354) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:888) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:19) ~[client-adapter.launcher-1.1.5.jar:na]
Caused by: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
    at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:83) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
    at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:52) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
    ... 42 common frames omitted
Caused by: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
    at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.addSyncConfigToCache(ESAdapter.java:146) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
    at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:75) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
    ... 43 common frames omitted
2021-06-11 18:11:29.753 [main] INFO  c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /usr/local/canal_adapter/plugin
2021-06-11 18:11:29.772 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-06-11 18:11:29.772 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2021-06-11 18:11:29.777 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-06-11 18:11:29.781 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-06-11 18:11:29.785 [Thread-3] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2021-06-11 18:11:29.907 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-06-11 18:11:29.909 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.1 seconds (JVM running for 3.499)
2021-06-11 18:11:30.045 [Thread-3] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============

在百度上查了一下，找到了這篇文章 canal本地運行異常：class com.alibaba.druid.pool.DruidDataSource  cannot be cast to_進擊的小白-CSDN博客

確認是jar包沖突導致的bug，於是下載了源碼壓縮包canal-canal-1.1.5.zip 下載地址 https://github.com/alibaba/canal/archive/refs/tags/canal-1.1.5.zip

解壓縮后用自己的開發工具打開項目，修改maven依賴

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid</artifactId>
            <scope>provided</scope>
        </dependency>

 

項目根目錄下執行mvn package，打包，然后到canal-canal-1.1.5\client-adapter\es7x\target\目錄下找到client-adapter.es7x-1.1.5-jar-with-dependencies.jar

將這個文件復制到/usr/local/canal_adapter/plugin下，重啟adapter，查看日志

/usr/local/canal_adapter/bin/restart.sh

tail -100f  /usr/local/canal_adapter/logs/adapter/adapter.log 

成功啟動，不再出現報錯。

同步到ElasticSearch

 在Elasticsearch中創建索引，和MySQL中的student表相對應，可以直接使用postman或者curl命令，也可以安裝kibana或者其它Elasticsearch管理插件

這里使用的是curl命令

curl --location --request PUT 'http://localhost:9200/student' \
--header 'Content-Type: application/json' \
--data '{"mappings":{"properties": {"name": {"type": "text"},"grade":{"type": "short"},"class":{"type": "short"},"birthday": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||date_time_no_millis||strict_date_optional_time||epoch_millis"}}}}'

執行結果顯示創建成功。

[root@localhost canal_adapter]# curl --location --request PUT 'http://localhost:9200/student' \
> --header 'Content-Type: application/json' \
> --data '{"mappings":{"properties": {"name": {"type": "text"},"grade":{"type": "short"},"class":{"type": "short"},"birthday": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||date_time_no_millis||strict_date_optional_time||epoch_millis"}}}}'

{"acknowledged":true,"shards_acknowledged":true,"index":"student"}
[root@localhost canal_adapter]# 

執行全量同步url

curl -X POST  http://localhost:8081/etl/es7/student.yml

返回結果：{"succeeded":true,"resultMessage":"導入ES 數據：1 條"}