概述
canal [kə'næl],譯意為水道/管道/溝渠,主要用途是基於 MySQL 數據庫增量日志解析,提供增量數據訂閱和消費
早期阿里巴巴因為杭州和美國雙機房部署,存在跨機房同步的業務需求,實現方式主要是基於業務 trigger 獲取增量變更。從 2010 年開始,業務逐步嘗試數據庫日志解析獲取增量變更進行同步,由此衍生出了大量的數據庫增量訂閱和消費業務。
基於日志增量訂閱和消費的業務包括
- 數據庫鏡像
- 數據庫實時備份
- 索引構建和實時維護(拆分異構索引、倒排索引等)
- 業務 cache 刷新
- 帶業務邏輯的增量數據處理
當前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
官方GitHub:
https://github.com/alibaba/canal
工作原理
MySQL主備復制原理
- MySQL master 將數據變更寫入二進制日志( binary log, 其中記錄叫做二進制日志事件binary log events,可以通過 show binlog events 進行查看)
- MySQL slave 將 master 的 binary log events 拷貝到它的中繼日志(relay log)
- MySQL slave 重放 relay log 中事件,將數據變更反映它自己的數據
canal 工作原理
- canal 模擬 MySQL slave 的交互協議,偽裝自己為 MySQL slave ,向 MySQL master 發送dump 協議
- MySQL master 收到 dump 請求,開始推送 binary log 給 slave (即 canal )
- canal 解析 binary log 對象(原始為 byte 流)
單機模式安裝
安裝前提環境
所需環境 | 本文環境 |
---|---|
linux | centos 7.0 |
jdk | jdk1.8 |
mysql | mysql5.7 |
安裝步驟
1.1下載地址
下載tar.gz結尾的壓縮包,主要是admin和deployer包,本文演示版本1.1.4;
canal.admin-1.1.4.tar.gz 此為canal的web管理頁面,非必須,為了可視化管理
canal.deployer-1.1.4.tar.gz 此為canal的服務器端,核心組件
更多版本參考以下網址:
https://github.com/alibaba/canal/releases
1.2上傳linux
利用FTP工具或者linux自帶RZ命令,上傳壓縮包到常用目錄(本文上傳到/opt/soft目錄)
[root@bigdata112 soft]# rz
1.3解壓包
在/opt/module目錄下,新建canal-admin和canal-server目錄,用於存放解壓的文件;
[root@bigdata112 module]# mkdir canal-admin
[root@bigdata112 module]# mkdir cana-server
切換到soft目錄,然后分別解壓兩個壓縮包到新建canal-admin和canal-server的目錄
[root@bigdata112 module]# cd ../soft
[root@bigdata112 soft]# tar -zvxf canal.admin-1.1.4.tar.gz -C /opt/module/canal-admin/
[root@bigdata112 soft]# tar -zvxf canal.deployer-1.1.4.tar.gz -C /opt/module/canal-server/
1.4配置canal-admin環境
釋義:canal-admin 為web管理頁面,可以通過圖形化進行查看和修改,方便快捷。
修改canal-admin/conf/application.yml相關配置(添加注釋部分按需要修改)
server:
# 默認端口,可以修改為自己的
port: 8089
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
spring.datasource:
# 若mysql在本機則無需修改,其他機器則寫對應ip
address: 127.0.0.1:3306
# canal數據庫默認名字,無需修改
database: canal_manager
# 數據庫用戶名和密碼修改
username: root
# 注意密碼添加單引號,尤其是有特殊字符時,容易報鏈接出錯
password: 'root'
# 若mysql版本是8以上,則需要改為 com.mysql.cj.jdbc.Driver
driver-class-name: com.mysql.jdbc.Driver
url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
hikari:
maximum-pool-size: 30
minimum-idle: 1
canal:
adminUser: admin
adminPasswd: admin
初始化canal數據庫
[root@bigdata112 conf]# cd /opt/module/canal-admin/conf
[root@bigdata112 conf]# ll
總用量 24
-rwxrwxrwx. 1 root root 485 3月 12 19:26 application.yml
-rwxrwxrwx. 1 root root 3898 9月 2 2019 canal_manager.sql
-rwxrwxrwx. 1 root root 5123 9月 2 2019 canal-template.properties
-rwxrwxrwx. 1 root root 2036 9月 2 2019 instance-template.properties
-rwxrwxrwx. 1 root root 1568 9月 2 2019 logback.xml
drwxrwxrwx. 3 root root 68 3月 12 19:08 public
[root@bigdata112 conf]# mysql -uroot -proot
mysql> source canal_manager.sql
1.5啟動canal-admin服務
[root@bigdata112 canal-admin]# bin/startup.sh
可以查看一下啟動日志,如下就代表啟動成功:
[root@bigdata112 canal-admin]# tail -f logs/admin.log
2020-03-12 19:38:15.342 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-03-12 19:38:15.560 [main] INFO o.s.w.s.m.m.annotation.ExceptionHandlerExceptionResolver - Detected @ExceptionHandler methods in customExceptionHandler
2020-03-12 19:38:15.639 [main] INFO o.s.b.a.web.servlet.WelcomePageHandlerMapping - Adding welcome page: class path resource [public/index.html]
2020-03-12 19:38:15.834 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2020-03-12 19:38:15.835 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'dataSource' has been autodetected for JMX exposure
2020-03-12 19:38:15.842 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2020-03-12 19:38:15.854 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8089"]
2020-03-12 19:38:15.895 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2020-03-12 19:38:15.919 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8089 (http) with context path ''
2020-03-12 19:38:15.923 [main] INFO com.alibaba.otter.canal.admin.CanalAdminApplication - Started CanalAdminApplication in 8.543 seconds (JVM running for 13.991)
或者jps命令查看,出現CanalAdminApplication代表服務啟動成功
[root@bigdata112 canal-admin]# jps
2445 CanalAdminApplication
2509 Jps
1.6訪問canal-admin測試
由於本機做了hosts映射,192.168.1.112 對應 bigdata112,所以可以在瀏覽器使用 bigdata112:8089訪問:
賬號:admin
密碼:123456
若想要修改密碼,管理頁面可以操作修改密碼。
1.7開啟mysql的binlog
1.7.1 關閉canal-admin服務
[root@bigdata112 canal-admin]# bin/stop.sh
bigdata112: stopping canal 2445 ...
Oook! cost:1
1.7.2 修改mysql的my.cnf文件
查找mysql的my.cnf配置文件位置,並增加開啟binlog的配置;
[root@bigdata112 canal-admin]# whereis my.cnf
my: /etc/my.cnf
[root@bigdata112 canal-admin]# vi /etc/my.cnf
[mysqld]
server_id=1
log-bin=mysql-bin
binlog_format=row
1.7.3重啟mysql服務並查看配置是否生效
重啟並查看配置是否生效,如下顯示則代表成功生效;
[root@bigdata112 canal-admin]# systemctl restart mysqld.service
[root@bigdata112 canal-admin]# mysql -uroot -proot
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.19-log MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW |
+---------------+-------+
1 row in set (0.03 sec)
mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+
1 row in set (0.00 sec)
1.7.4 配置canal賬戶和賦權限
由於本文為mysql5.7,密碼策略比較嚴格,所以進行了相關密碼策略的修改,為了創建canal的賬戶需要,如果mysql版本為5.7以下的,可以直接進行創建用戶和賦權操作。
mysql> set GLOBAL validate_password_length=5;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_number_count=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_mixed_case_count=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_special_char_count=0;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE USER canal IDENTIFIED BY 'canal';
Query OK, 0 rows affected (0.08 sec)
mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> show grants for 'canal' ;
+---------------------------------------------------------------------------+
| Grants for canal@% |
+---------------------------------------------------------------------------+
| GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' |
+---------------------------------------------------------------------------+
1 row in set (0.00 sec)
1.8 啟動canal-server服務
開啟命令:
[root@bigdata112 canal-admin]# cd ../canal-server/
[root@bigdata112 canal-server]# ll
總用量 8
drwxr-xr-x. 2 root root 72 3月 12 19:10 bin
drwxr-xr-x. 5 root root 4096 3月 12 19:10 conf
drwxr-xr-x. 2 root root 4096 3月 12 19:10 lib
drwxrwxrwx. 2 root root 6 9月 2 2019 logs
[root@bigdata112 canal-server]# bin/startup.sh
關閉命令:
[root@bigdata112 canal-server]# bin/stop.sh
查看是否成功:
[root@bigdata112 canal-server]# jps
2743 CanalLauncher
2777 Jps
出現CanalLauncher進程就代表啟動成功了;
OK,單機搭建就到此為止了;
HA高可用模式搭建
三台機器分別為bigdata111,bigdata112,bigdata113,相關軟件環境;
bigdata111 | bigdata112 | bigdata113 |
---|---|---|
centos7.0 | centos7.0 | centos7.0 |
jdk1.8 | jdk1.8 | jdk1.8 |
zookeeper3.4 | zookeeper3.4 | zookeeper3.4 |
mysql5.7 | ||
canal-server | canal-server | canal-server |
canal-admin |
由於上面演示了bigdata112的單機安裝,則HA的安裝在如上基礎上進行。
1.分發解壓目錄
在bigdata112執行scp命令,將bigdata112的canal-server發送到bigdata111,bigdata113服務器。
[root@bigdata112 ~]# scp -r /opt/module/canal-server/ root@bigdata111:/opt/module/
[root@bigdata112 ~]# scp -r /opt/module/canal-server/ root@bigdata113:/opt/module/
2.修改配置
bigdata111配置
在/opt/module/canal-server/conf目錄下操作:
[root@bigdata111 canal-server]# cd conf/
[root@bigdata111 conf]# ll
總用量 20
-rwxr-xr-x. 1 root root 291 3月 16 04:43 canal_local.properties
-rwxr-xr-x. 1 root root 5134 3月 16 04:43 canal.properties
drwxrwxrwx. 2 root root 47 3月 16 04:43 example
-rwxr-xr-x. 1 root root 3119 3月 16 04:43 logback.xml
drwxrwxrwx. 2 root root 38 3月 16 04:43 metrics
drwxrwxrwx. 3 root root 4096 3月 16 04:43 spring
[root@bigdata111 conf]# vi canal.properties
#修改zkServers地址
canal.zkServers =bigdata111:2181,bigdata112:2181,bigdata113:2181
#注釋file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#開啟default-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
修改完畢保存,然后進入example目錄,修改instance
[root@bigdata111 conf]# cd example/
[root@bigdata111 example]# ll
總用量 196
-rw-r--r--. 1 root root 196608 3月 16 04:43 h2.mv.db
-rwxr-xr-x. 1 root root 2036 3月 16 04:43 instance.properties
[root@bigdata111 example]# vi instance.properties
# 解開注釋,並修改數字,只要保證三台機器這個數字不重復即可
canal.instance.mysql.slaveId=111
# 填寫mysql的地址,由於我們裝在bigdata112上,所以改為了bigdata112
canal.instance.master.address=bigdata112:3306
bigdata112配置
重復bigdata111的操作步驟,注意將/opt/module/canal-server/conf/example/instance.properties的數字修改一下,同時數據庫地址寫成主機名或內網IP地址;
canal.instance.mysql.slaveId=112
canal.instance.master.address=bigdata112:3306
bigdata113配置
重復bigdata111的操作步驟,注意將/opt/module/canal-server/conf/example/instance.properties的數字修改一下,同時數據庫地址寫成主機名或內網IP地址;
canal.instance.mysql.slaveId=113
canal.instance.master.address=bigdata112:3306
3.啟動canal-server服務
分別在三台機器啟動server服務(若使用的xshell工具,則可以使用右鍵“發送鍵輸入到所有窗口”一鍵操作)
[root@bigdata111 canal-server]# bin/startup.sh
[root@bigdata112 canal-server]# bin/startup.sh
[root@bigdata113 canal-server]# bin/startup.sh
jps查看是否啟動成功,都出現CanalLauncher代表成功;
[root@bigdata111 canal-server]# jps
3137 Jps
2949 QuorumPeerMain
3115 CanalLauncher
[root@bigdata112 canal-server]# jps
3232 Jps
3025 QuorumPeerMain
3181 CanalLauncher
[root@bigdata113 canal-server]# jps
3193 CanalLauncher
3227 Jps
3036 QuorumPeerMain
在zookeeper上查看節點是否工作,如下則代表成功。
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, yarn-leader-election, hadoop-ha, rmstore, mylock, hbase, otter]
[zk: localhost:2181(CONNECTED) 1] get /otter/canal/destinations/example/running
{"active":true,"address":"192.168.1.112:11111"}
cZxid = 0x1800000023
ctime = Mon Mar 16 05:13:15 CST 2020
mZxid = 0x1800000023
mtime = Mon Mar 16 05:13:15 CST 2020
pZxid = 0x1800000023
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x370dfea0c140001
dataLength = 47
numChildren = 0
查看三台canal機器是否在zookeeper中注冊:
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, mylock, log_dir_event_notification, otter, controller_epoch, rmstore, consumers, latest_producer_id_block, config, hbase]
[zk: localhost:2181(CONNECTED) 1] ls /otter/canal/
Command failed: java.lang.IllegalArgumentException: Path must not end with / character
[zk: localhost:2181(CONNECTED) 2] ls /otter
[canal]
[zk: localhost:2181(CONNECTED) 3] ls /otter/canal
[cluster, destinations]
[zk: localhost:2181(CONNECTED) 4] ls /otter/canal/cluster
4.關閉canal
[root@bigdata111 canal-server]# bin/stop.sh
[root@bigdata112 canal-server]# bin/stop.sh
[root@bigdata113 canal-server]# bin/stop.sh
以上則為HA的搭建過程;