canal和maxwell的安裝配置及區別

Canal地址:https://github.com/alibaba/canal

Maxwell地址:https://github.com/zendesk/maxwell

Maxwell官網地址:http://maxwells-daemon.io/

0、mysql開啟binglog

在mysql的配置文件(Linux: /etc/my.cnf ,  Windows: \my.ini)下,修改配置

在[mysqld] 區塊（沒有就創建）

設置/添加

（1）log-bin=mysql-bin

這個表示binlog日志的前綴是mysql-bin  ，以后生成的日志文件就是 mysql-bin.123456 的文件后面的數字按順序生成。 每次mysql重啟或者到達單個文件大小的閾值時，新生一個文件，按順序編號。

（2）binlog_format=row

mysql binlog的格式，那就是有三種，分別是STATEMENT,MIXED,ROW

1） statement
語句級，binlog會記錄每次一執行寫操作的語句。
相對row模式節省空間，但是可能產生不一致性，比如
update tt set create_date=now()
如果用binlog日志進行恢復，由於執行時間不同可能產生的數據就不同。
優點： 節省空間
缺點： 有可能造成數據不一致。
2） row
行級， binlog會記錄每次操作后每行記錄的變化。
優點：保持數據的絕對一致性。因為不管sql是什么，引用了什么函數，他只記錄執行后的效果。
缺點：占用較大空間。

3） mixed
statement的升級版，一定程度上解決了，因為一些情況而造成的statement模式不一致問題
在某些情況下譬如：
1. 當函數中包含 UUID() 時；
2. 包含 AUTO_INCREMENT 字段的表被更新時；
3. 執行 INSERT DELAYED 語句時；
4. 用 UDF 時；
會按照 ROW的方式進行處理
優點：節省空間，同時兼顧了一定的一致性。
缺點：還有些極個別情況依舊會造成不一致，另外statement和mixed對於需要對binlog的監控的情況都不方便。

（3）binlog-do-db=test_db

要監聽的數據庫

（4）server-id=1

唯一ID

（5）重啟mysql

（6）查看binlog

mysql> show variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW   |
+---------------+-------+
mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin       | ON    |
+---------------+-------+

 

1、canal的單機配置

1.1 基本配置

canal.properties

conf/example/instance.properties

1.2 mysql創建用戶賦予權限

mysql> CREATE USER canal IDENTIFIED BY 'canal'; 
mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
mysql> FLUSH PRIVILEGES; 
mysql> show grants for 'canal' ;
+----------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for canal@%                                                                                                                           |
+----------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' IDENTIFIED BY PASSWORD '*E3619321C1A937C46A0D8BD1DAC39F93B27D4458' |
+----------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

 

給的權限只有，查，slave，和客戶端，不允許增刪等操作

1.3 多分區配置

所有庫，所有表，按主鍵分區

2、maxwell的配置

2.1 基本配置

復制config.properties.example為maxwell.properties

2.2 mysql創建用戶賦予權限

mysql> CREATE DATABASE maxwell ;
mysql> GRANT ALL   ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY 'maxwell';
mysql> GRANT  SELECT ,REPLICATION SLAVE , REPLICATION CLIENT  ON *.* TO maxwell@'%'
mysql> FLUSH PRIVILEGES;

2.3 多分區配置

2.4 maxwell-bootstrap

針對數據初始化的問題，Maxwell 提供了一個命令工具 maxwell-bootstrap 幫助我們完成數據初始化，maxwell-bootstrap 是基於 SELECT * FROM table 的方式進行全量數據初始化，不會產生多余的binlog！

參數	說明
–log_level LOG_LEVEL	日志級別（DEBUG, INFO, WARN or ERROR）
–user USER	mysql 用戶名
–password PASSWORD	mysql 密碼
–host HOST	mysql 地址
–port PORT	mysql 端口
–database DATABASE	要bootstrap的表所在的數據庫
–table TABLE	要引導的表
–where WHERE_CLAUSE	設置過濾條件
–client_id CLIENT_ID	指定執行引導操作的Maxwell實例

bin/maxwell-bootstrap --user maxwell --password 123456 --host 127.0.0.1 --database test --table test --client_id maxwell

其中client_id 是指另一個已啟動的maxwell監控進程的client_id

注意：--bootstrapper=sync 時，在處理bootstrap時，會阻塞正常的binlog解析；--bootstrapper=async 時，不會阻塞。

3、canal和maxwell對比

3.1 執行insert操作

insert into student values(4,"張三",100),(5,"李思",80),(6,"王五",50)

canal

{"data":null,"database":"gmall","es":1598770894000,"id":6439,"isDdl":true,"mysqlType":null,"old":null,"pkNames":null,"sql":"ALTER TABLE `student`\r\nMODIFY COLUMN `id`  int(11) UNSIGNED NOT NULL AUTO_INCREMENT FIRST","sqlType":null,"table":"student","ts":1598770895098,"type":"ALTER"}
{"data":[{"id":"4","name":"張三","age":"100"}],"database":"gmall","es":1598770932000,"id":6440,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598770932288,"type":"INSERT"}
{"data":[{"id":"5","name":"李思","age":"80"}],"database":"gmall","es":1598770932000,"id":6440,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598770932288,"type":"INSERT"}
{"data":[{"id":"6","name":"王五","age":"50"}],"database":"gmall","es":1598770932000,"id":6440,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598770932288,"type":"INSERT"}

 

maxwell更簡潔

{"database":"gmall","table":"student","type":"insert","ts":1598770932,"xid":261875,"commit":true,"data":{"id":6,"name":"王五","age":50}}
{"database":"gmall","table":"student","type":"insert","ts":1598770932,"xid":261875,"xoffset":1,"data":{"id":5,"name":"李思","age":80}}
{"database":"gmall","table":"student","type":"insert","ts":1598770932,"xid":261875,"xoffset":0,"data":{"id":4,"name":"張三","age":100}}

 

3.2 執行update操作

UPDATE student SET age='88' WHERE id IN(5,4)

maxwell

{"database":"gmall","table":"student","type":"update","ts":1598771585,"xid":270573,"xoffset":0,"data":{"id":4,"name":"張三","age":88},"old":{"age":100}}
{"database":"gmall","table":"student","type":"update","ts":1598771585,"xid":270573,"commit":true,"data":{"id":5,"name":"李思","age":88},"old":{"age":80}}

 

canal

{"data":[{"id":"4","name":"張三","age":"88"}],"database":"gmall","es":1598771585000,"id":6444,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":[{"age":"100"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598771585168,"type":"UPDATE"}
{"data":[{"id":"5","name":"李思","age":"88"}],"database":"gmall","es":1598771585000,"id":6444,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":[{"age":"80"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598771585168,"type":"UPDATE"}

 

3.3 執行delete操作

maxwell

{"database":"gmall","table":"student","type":"delete","ts":1598771670,"xid":271637,"xoffset":0,"data":{"id":4,"name":"張三","age":88}}
{"database":"gmall","table":"student","type":"delete","ts":1598771670,"xid":271637,"commit":true,"data":{"id":5,"name":"李思","age":88}}

 

canal

{"data":[{"id":"5","name":"李思","age":"88"}],"database":"gmall","es":1598771670000,"id":6445,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598771670306,"type":"DELETE"}
{"data":[{"id":"4","name":"張三","age":"88"}],"database":"gmall","es":1598771670000,"id":6445,"isDdl":false,"mysqlType":{"id":"int(11) unsigned","name":"varchar(20)","age":"int"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"name":12,"age":4},"table":"student","ts":1598771670306,"type":"DELETE"}

 

3.4 對比

• Maxwell 沒有 Canal那種server+client模式，只有一個server把數據發送到消息隊列或redis。

• Maxwell 有一個亮點功能，就是Canal只能抓取最新數據，對已存在的歷史數據沒有辦法處理。而Maxwell有一個bootstrap功能，可以直接引導出完整的歷史數據用於初始化，非常好用。

• Maxwell不能直接支持HA，但是它支持斷點還原，即錯誤解決后重啟繼續上次點兒讀取數據。

• Maxwell只支持json格式，而Canal如果用Server+client模式的話，可以自定義格式。

• Maxwell比Canal更加輕量級。

3.5 總結

• 日志結構

canal 每一條SQL會產生一條日志，如果該條Sql影響了多行數據，則已經會通過集合的方式歸集在這條日志中。（即使是一條數據也會是數組結構）

maxwell 以影響的數據為單位產生日志，即每影響一條數據就會產生一條日志。如果想知道這些日志是否是通過某一條sql產生的可以通過xid進行判斷，相同的xid的日志來自同一sql。

• 數字類型

   當原始數據是數字類型時,maxwell會尊重原始數據的類型不增加雙引，變為字符串。

   canal一律轉換為字符串。

• 帶原始數據字段定義

canal數據中會帶入表結構。

maxwell更簡潔。