MySQL數據源接入DBus
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
一.cannal概述
1>.cannal簡介
canal [kə'næl]，譯意為水道/管道/溝渠，主要用途是基於 MySQL 數據庫增量日志解析，提供增量數據訂閱和消費 早期阿里巴巴因為杭州和美國雙機房部署，存在跨機房同步的業務需求，實現方式主要是基於業務 trigger 獲取增量變更。從 2010 年開始，業務逐步嘗試數據庫日志解析獲取增量變更進行同步，由此衍生出了大量的數據庫增量訂閱和消費業務。 基於日志增量訂閱和消費的業務包括 數據庫鏡像 數據庫實時備份 索引構建和實時維護(拆分異構索引、倒排索引等) 業務 cache 刷新 帶業務邏輯的增量數據處理 當前的canal支持源端MySQL版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x 重要版本更新說明 canal 1.1.x 版本（release_note）,性能與功能層面有較大的突破,重要提升包括: 整體性能測試&優化,提升了150%. 原生支持prometheus監控 原生支持kafka消息投遞 原生支持aliyun rds的binlog訂閱 (解決自動主備切換/oss binlog離線解析) 原生支持docker鏡像 canal 1.1.4版本，迎來最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal動態管理能力，支持配置、任務、日志等在線白屏運維能力。 GitHub地址: https://github.com/alibaba/canal
2>.MySQL主備復制原理概述
如下圖所示:
MySQL master 將數據變更寫入二進制日志( binary log, 其中記錄叫做二進制日志事件binary log events，可以通過 show binlog events 進行查看)
MySQL slave 將 master 的 binary log events 拷貝到它的中繼日志(relay log)
MySQL slave 重放 relay log 中事件，將數據變更反映它自己的數據
3>.canal 工作原理
canal模擬MySQL slave的交互協議，偽裝自己為MySQL slave，向MySQL master發送dump協議 MySQL master收到dump請求，開始推送binary log給MySQL slave(即canal) canal解析binary log對象(原始為 byte 流) 博主推薦閱讀: https://github.com/alibaba/canal/blob/master/README.md
二.數據庫源端創建心跳庫
數據庫源端配置只在第一次配置環境時需要，在以后的加表流程中不需要再配置。
此步驟中需要在mysql數據源的mysql_instance實例上創建一個名字為dbus的庫，並在此庫下創建表db_heartbeat_monitor和db_full_pull_requests兩張表，用於心跳檢測和全量拉取。
在數據源端新建的dbus庫，可以實現無侵入方式接入多種數據源，業務系統無需任何修改，以無侵入性讀取數據庫系統的日志獲得增量數據實時變化。
1>.自行配置MySQL主從復制
博主推薦閱讀: https://www.cnblogs.com/yinzhengjie/p/11816066.html
2>.創建dbus庫和dbus用戶及相應權限
[root@hdp103.yinzhengjie.org.cn ~]# mysql -uroot -p -S /yinzhengjie/softwares/mysql/data/mysql.sock Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 26 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> CREATE DATABASE dbus; Query OK, 1 row affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> CREATE USER 'dbus'@'%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> GRANT ALL ON dbus.* TO dbus@'%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected, 1 warning (0.01 sec) MySQL [(none)]> MySQL [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]#
[root@hdp101.yinzhengjie.org.cn ~]# mysql -udbus -pyinzhengjie -h hdp103.yinzhengjie.org.cn mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 39 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | dbus | | yinzhengjie_bigdata | +---------------------+ 3 rows in set (0.01 sec) mysql> mysql> QUIT Bye [root@hdp101.yinzhengjie.org.cn ~]# [root@hdp101.yinzhengjie.org.cn ~]#
3>.創建dbus庫中需要包含的2張表
use dbus; -- ---------------------------- -- Table structure for db_full_pull_requests -- ---------------------------- DROP TABLE IF EXISTS `db_full_pull_requests`; CREATE TABLE `db_full_pull_requests` ( `seqno` bigint(19) NOT NULL AUTO_INCREMENT, `schema_name` varchar(64) DEFAULT NULL, `table_name` varchar(64) NOT NULL, `physical_tables` varchar(10240) DEFAULT NULL, `scn_no` int(11) DEFAULT NULL, `split_col` varchar(50) DEFAULT NULL, `split_bounding_query` varchar(512) DEFAULT NULL, `pull_target_cols` varchar(512) DEFAULT NULL, `pull_req_create_time` timestamp(6) NOT NULL, `pull_start_time` timestamp(6) NULL DEFAULT NULL, `pull_end_time` timestamp(6) NULL DEFAULT NULL, `pull_status` varchar(16) DEFAULT NULL, `pull_remark` varchar(1024) DEFAULT NULL, PRIMARY KEY (`seqno`) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; SET FOREIGN_KEY_CHECKS=0; -- ---------------------------- -- Table structure for db_heartbeat_monitor -- ---------------------------- DROP TABLE IF EXISTS `db_heartbeat_monitor`; CREATE TABLE `db_heartbeat_monitor` ( `ID` bigint(19) NOT NULL AUTO_INCREMENT COMMENT '心跳表主鍵', `DS_NAME` varchar(64) NOT NULL COMMENT '數據源名稱', `SCHEMA_NAME` varchar(64) NOT NULL COMMENT '用戶名', `TABLE_NAME` varchar(64) NOT NULL COMMENT '表名', `PACKET` varchar(256) NOT NULL COMMENT '數據包', `CREATE_TIME` datetime NOT NULL COMMENT '創建時間', `UPDATE_TIME` datetime NOT NULL COMMENT '修改時間', PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
MySQL [dbus]> SHOW TABLES; +-----------------------+ | Tables_in_dbus | +-----------------------+ | db_full_pull_requests | | db_heartbeat_monitor | +-----------------------+ 2 rows in set (0.00 sec) MySQL [dbus]> MySQL [dbus]> DESC db_full_pull_requests; +----------------------+----------------+------+-----+----------------------+--------------------------------+ | Field | Type | Null | Key | Default | Extra | +----------------------+----------------+------+-----+----------------------+--------------------------------+ | seqno | bigint(19) | NO | PRI | NULL | auto_increment | | schema_name | varchar(64) | YES | | NULL | | | table_name | varchar(64) | NO | | NULL | | | physical_tables | varchar(10240) | YES | | NULL | | | scn_no | int(11) | YES | | NULL | | | split_col | varchar(50) | YES | | NULL | | | split_bounding_query | varchar(512) | YES | | NULL | | | pull_target_cols | varchar(512) | YES | | NULL | | | pull_req_create_time | timestamp(6) | NO | | CURRENT_TIMESTAMP(6) | on update CURRENT_TIMESTAMP(6) | | pull_start_time | timestamp(6) | YES | | NULL | | | pull_end_time | timestamp(6) | YES | | NULL | | | pull_status | varchar(16) | YES | | NULL | | | pull_remark | varchar(1024) | YES | | NULL | | +----------------------+----------------+------+-----+----------------------+--------------------------------+ 13 rows in set (0.00 sec) MySQL [dbus]>
MySQL [dbus]> DESC db_heartbeat_monitor; +-------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+----------------+ | ID | bigint(19) | NO | PRI | NULL | auto_increment | | DS_NAME | varchar(64) | NO | | NULL | | | SCHEMA_NAME | varchar(64) | NO | | NULL | | | TABLE_NAME | varchar(64) | NO | | NULL | | | PACKET | varchar(256) | NO | | NULL | | | CREATE_TIME | datetime | NO | | NULL | | | UPDATE_TIME | datetime | NO | | NULL | | +-------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec) MySQL [dbus]>
三.准備業務庫表
1>.創建業務庫和表(這一步你可以自行定義測試的庫和表)
MySQL [(none)]> CREATE DATABASE yinzhengjie_bigdata; Query OK, 1 row affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> USE yinzhengjie_bigdata Database changed MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> CREATE TABLE student(id int,name varchar(50)); Query OK, 0 rows affected (0.06 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> SHOW TABLES; +-------------------------------+ | Tables_in_yinzhengjie_bigdata | +-------------------------------+ | student | +-------------------------------+ 1 row in set (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> DESC student; +-------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+-------------+------+-----+---------+-------+ | id | int(11) | YES | | NULL | | | name | varchar(50) | YES | | NULL | | +-------+-------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> CREATE USER 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> GRANT ALL ON yinzhengjie_bigdata.* TO 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected, 1 warning (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
[root@hdp103.yinzhengjie.org.cn ~]# mysql -ujason -pyinzhengjie -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 38 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | yinzhengjie_bigdata | +---------------------+ 2 rows in set (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
2>.為dbus用戶分配咱們上一步驟創建的業務數據庫權限
MySQL [yinzhengjie_bigdata]> GRANT SELECT ON yinzhengjie_bigdata.* TO 'dbus'@'%'; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]>
四.部署cannal
1>.在數據庫中創建canal用戶
MySQL [yinzhengjie_bigdata]> CREATE USER 'canal'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> GRANT SELECT,REPLICATION SLAVE,REPLICATION CLIENT ON *.* TO 'canal'@'172.200.1.%'; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]>
[root@hdp103.yinzhengjie.org.cn ~]# mysql -ucanal -pyinzhengjie -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 32 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | dbus | | dbusmgr | | mysql | | performance_schema | | sys | | yinzhengjie_bigdata | +---------------------+ 7 rows in set (0.01 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]#
2>.cannel自動化部署
在DBus-keeper頁面添加數據線時可以選擇自動部署cannal，當然我們也可以一鍵部署，只需要把dbus官方提供的依賴canal的安裝包放到指定目錄並解壓即可，具體操作如下所示。 [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll total 417572 -rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz drwxr-xr-x 5 root root 222 Mar 13 22:06 dbuskeeper_web -rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# tar -zxf dbus-canal-auto-0.5.0.tar.gz [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll total 417572 drwxr-xr-x 5 500 500 95 Feb 20 2019 dbus-canal-auto-0.5.0 -rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz drwxr-xr-x 5 root root 222 Mar 13 22:06 dbuskeeper_web -rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0 total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Feb 20 2019 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]#
3>.如上圖所示，咱們需要修改conf目錄下的canal-auto.properties文件
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0 total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Feb 20 2019 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0/conf/ total 4 -rw-r--r-- 1 500 500 331 Oct 19 2018 canal-auto.properties [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# vim dbus-canal-auto-0.5.0/conf/canal-auto.properties [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# cat dbus-canal-auto-0.5.0/conf/canal-auto.properties #數據源名稱，需要與dbus keeper中添加的一致 dsname=yinzhengjie_bigdata #zk地址,替換成自己的信息 zk.path=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 #canal 用戶連接地址。即：要canal去同步的源端庫的備庫的地址 canal.address=hdp103.yinzhengjie.org.cn:3306 #canal用戶名 canal.user=canal #canal密碼，替換成自己配置的 canal.pwd=yinzhengjie [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]#
4>.執行deploy.sh腳本
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Mar 14 10:02 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# cat deploy.sh #!/bin/sh #獲得當前shell所在路徑 basepath=$(cd `dirname $0`; pwd) #echo $basepath echo "****************************************** starting *****************************************" #jvm啟動參數 #GC_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/data/dbus-canal-auto/logs/gc/gc.log" LOG_CONF="-Dlogs.base.path=$basepath -Duser.dir=$basepath" OOM_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$basepath/logs/oom" JVM_OPTS="-server -Xmx4096m -Xms100m -XX:NewRatio=1" CLASS_PATH="" MAIN="com.creditease.dbus.canal.auto.AutoDeployStart" if [ "x$1" = "xcheck" ] then MAIN="com.creditease.dbus.canal.auto.AutoCheckStart" fi #導入jar和config進入classpath for i in $basepath/lib/*.jar; do CLASS_PATH=$i:"$CLASS_PATH"; done export CLASS_PATH=.:$CLASS_PATH java $JVM_OPTS $LOG_CONF $OOM_OPTS -classpath $CLASS_PATH $MAIN sleep 1 cd reports filename=`ls -ltr |grep canal_| tail -n 1 | awk '{print $9}'` if [ "x$filename" != "x" ]; then cat $filename echo "report文件： $filename" sleep 1 fi [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Mar 14 10:02 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh ****************************************** starting ***************************************** ************************************ CANAL CONFIG CHECK BEGIN!******************************* 數據源名稱: yinzhengjie_bigdata zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 備庫地址: hdp103.yinzhengjie.org.cn:3306 canal 用戶名: canal canal 密碼: yinzhengjie *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 數據庫連接成功... 檢查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 2020-03-14 10:38:14,778 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting 2020-03-14 10:38:14,808 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_201 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log 4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_64 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0 2020-03-14 10:38:14,825 INFO - [org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c urator.ConnectionState@25359ed82020-03-14 10:38:14,840 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp102.yinzhengjie.org.cn/172.200.1.102:2181. Will not attempt to authenticate using SASL (unknown error) 2020-03-14 10:38:14,900 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp102.yinzhengjie.org.cn/172.200.1.102:2181, initiating session 2020-03-14 10:38:14,911 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp102.yinzhengjie.org.cn/172.200.1.102:2181, sessionid = 0x270d619e962000a, negotiated timeout = 60000 2020-03-14 10:38:14,915 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal 2020-03-14 10:38:14,977 INFO - [com.creditease.dbus.commons.ZkService.createNode(ZkService.java:159)] 節點創建成功, Path: /DBus/Canal/canal-yinzhengjie_bigdata create zk node: /DBus/Canal/canal-yinzhengjie_bigdata node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** 2020-03-14 10:38:15,007 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x270d619e962000a closed ************************************ CANAL CONFIG CHECK SUCCESS!***************************** **************************************** CANAL DEPLOY BEGIN!********************************* **************************************** COPY CANAL BEGIN *********************************** copy canal files: cp -r canal/. canal-yinzhengjie_bigdata 2020-03-14 10:38:15,013 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x270d619e962000a copy instance files: cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata **************************************** COPY CANAL SUCCESS ********************************* ************************************ EDIT CANAL.PROPERTIES BEGIN **************************** props: canal.port = 10000 props: canal.zkServers = hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata props: canal.destinations = yinzhengjie_bigdata props: canal.auto.scan = false props: canal.instance.filter.query.dcl = true props: canal.instance.filter.query.dml = true props: canal.instance.binlog.format = ROW props: canal.instance.binlog.image = FULL props: #canal.instance.global.spring.xml = classpath:spring/file-instance.xml props: canal.instance.global.spring.xml = classpath:spring/default-instance.xml ********************************** EDIT CANAL.PROPERTIES SUCCESS **************************** ****************************** UPDATE INSTANCE.PROPERTIES BEGIN ***************************** instance file path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties props: canal.instance.master.address = hdp103.yinzhengjie.org.cn:3306 props: canal.instance.dbUsername = canal props: canal.instance.dbPassword = yinzhengjie props: canal.instance.connectionCharset = UTF-8 ***************************** UPDATE INSTANCE.PROPERTIES SUCCESS **************************** starting canal..... exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.sh exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.sh exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log exec: ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}' canal 進程啟動成功， pid 80463 請手動檢查當前目錄下canal.log，和yinzhengjie_bigdata.log中有無錯誤信息。 ********************************* CANAL DEPLOY SCCESS! ************************************** ************************************ CANAL CONFIG CHECK BEGIN!******************************* 數據源名稱: yinzhengjie_bigdata zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 備庫地址: hdp103.yinzhengjie.org.cn:3306 canal 用戶名: canal canal 密碼: yinzhengjie *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 數據庫連接成功... 檢查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal create zk node: /DBus/Canal/canal-yinzhengjie_bigdata node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** ************************************ CANAL CONFIG CHECK SUCCESS!***************************** **************************************** CANAL DEPLOY BEGIN!********************************* **************************************** COPY CANAL BEGIN *********************************** copy canal files: cp -r canal/. canal-yinzhengjie_bigdata copy instance files: cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata **************************************** COPY CANAL SUCCESS ********************************* ************************************ EDIT CANAL.PROPERTIES BEGIN **************************** props: canal.port = 10000 props: canal.zkServers = hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata props: canal.destinations = yinzhengjie_bigdata props: canal.auto.scan = false props: canal.instance.filter.query.dcl = true props: canal.instance.filter.query.dml = true props: canal.instance.binlog.format = ROW props: canal.instance.binlog.image = FULL props: #canal.instance.global.spring.xml = classpath:spring/file-instance.xml props: canal.instance.global.spring.xml = classpath:spring/default-instance.xml ********************************** EDIT CANAL.PROPERTIES SUCCESS **************************** ****************************** UPDATE INSTANCE.PROPERTIES BEGIN ***************************** instance file path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties props: canal.instance.master.address = hdp103.yinzhengjie.org.cn:3306 props: canal.instance.dbUsername = canal props: canal.instance.dbPassword = yinzhengjie props: canal.instance.connectionCharset = UTF-8 ***************************** UPDATE INSTANCE.PROPERTIES SUCCESS **************************** starting canal..... exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.sh exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.sh exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log exec: ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}' canal 進程啟動成功， pid 80463 請手動檢查當前目錄下canal.log，和yinzhengjie_bigdata.log中有無錯誤信息。 ********************************* CANAL DEPLOY SCCESS! ************************************** report文件： canal_deploy_yinzhengjie_bigdata_20200314103813.txt [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
5>.檢查部署是否正常
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh check ****************************************** starting ***************************************** ************************************* CANAL CHECK BEGIN! ************************************ *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 數據庫連接成功... 檢查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 2020-03-14 10:41:30,246 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting 2020-03-14 10:41:30,261 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2020-03-14 10:41:30,263 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_201 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log 4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_64 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0 2020-03-14 10:41:30,270 INFO - [org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c urator.ConnectionState@25359ed82020-03-14 10:41:30,292 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp101.yinzhengjie.org.cn/172.200.1.101:2181. Will not attempt to authenticate using SASL (unknown error) 2020-03-14 10:41:30,373 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp101.yinzhengjie.org.cn/172.200.1.101:2181, initiating session 2020-03-14 10:41:30,385 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp101.yinzhengjie.org.cn/172.200.1.101:2181, sessionid = 0x170d61a5413000e, negotiated timeout = 60000 2020-03-14 10:41:30,393 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** 2020-03-14 10:41:30,452 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x170d61a5413000e closed 2020-03-14 10:41:30,453 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x170d61a5413000e exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log **************************************** CANAL CHECK SUCCESS ******************************** ************************************* CANAL CHECK BEGIN! ************************************ *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 數據庫連接成功... 檢查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log **************************************** CANAL CHECK SUCCESS ******************************** report文件： canal_check_yinzhengjie_bigdata_20200314104129.txt [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
6>.管理canal服務
[root@hdp103.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/ total 16 -rw-r--r-- 1 root root 5 Mar 14 16:11 canal.pid -rwxr-xr-x 1 root root 1145 Mar 14 10:38 startup.bat -rwxr-xr-x 1 root root 2956 Mar 14 10:38 startup.sh -rwxr-xr-x 1 root root 1356 Mar 14 10:38 stop.sh [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
五.DBus一鍵加線
1>.數據線的概念
所謂數據線是一種形象的說法，DBus要想采集某個數據庫(數據源)的數據，就得拉一根線連接數據庫和DBus，加數據線就是DBus-Keeper頁面配置一條數據線，即指定從哪個數據源的哪個庫采集哪些表的數據。
Dbus對每個DataSource數據源配置一條數據線，當要添加新的datasource時，需要新添加一條數據線。下面對通過dbus keeper頁面添加新數據線的步驟進行介紹
2>.刪除自動部署canal的配置
我們采用手工方式部署canal，所以把相關配置刪掉，避免加數據線時出現問題，具體操作如下圖所示。 依次點擊"配置中心","zk管理","DBus","Commons","auto-deploy-canal.conf"(需要鼠標右擊),"刪除節點"
刪除成功后，會有以下的提示信息，建議大家都刪除掉喲，否則你盡管下面再不勾選自動部署canal依舊會部署失敗，這算是官方的一個bug啦~
3>.管理員身份進入dbus keeper頁面，並依次點擊"數據源管理","新建數據線"
4>.填寫數據源基本信息 （master和slave jdbc連接串信息）
jdbc:MySQL://hdp103.yinzhengjie.org.cn:3306/dbus?useUnicode=true&characterEncoding=utf8&autoReconnect=true&failOverReadOnly=false&noAccessToProcedureBodies=true&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false
5>.下一步添加的schema，勾選要添加的表。Keeper支持一次添加多個schema下的多個table
6>.啟動Topology
我們從上往下依次點擊啟動按鈕，點擊按鈕后當前頁面就不可用被選擇啦，如下圖所示。
第一個dsName啟動成功后，它的狀態會變成running狀態喲~如下圖所示。
啟動成功后，我們通過Ambari的storm服務菜單可以找到打開storm WebUI的界面，可以看到的確是有Topology啟動。
接下來的工作就簡單了，按照上述的操作，依次啟動三個Topology，啟動成功后，狀態均變為running狀態，如下圖所示。
再次查看Storm WebUI，可以看到相應的狀態信息。你可以點擊每一個任務查看詳細喲~
7>.上一步執行成功后，再次回到"數據源管理"，我們看到數據線就加好了，如下圖所示。
