6、Canal實現MySQL到ES實時同步-1

1、准備一台虛擬機

1.1 配置靜態ip

可以參考 https://www.cnblogs.com/braveym/p/9096402.html 

 

1.2給hadoop用戶賦予root權限

切換到root用戶

並輸入命令 

visudo

 

這時會進入/etc/sudoers文件的編輯頁面，增加如下配置即可

hadoop ALL=(ALL) NOPASSWD: ALL
#以下這行一定要注釋掉以免被覆蓋(hadoop用戶屬於wheel組)
#%wheel ALL=(ALL) ALL

 

1.3 配置hostname

永久修改hostname執行如下命令

sudo hostnamectl set-hostname canal01

 

 

1.4 配置hosts文件

sudo vi /etc/hosts

 

添加以下內容

172.16.25.31 canal01

 

 

2、安裝Mysql

配置Mysql 8.0安裝源

 

sudo rpm -Uvh https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

 

 

 

 

安裝mysql

 

 sudo yum -y install mysql-community-server

 

 

 

出現以下錯誤

warning: /var/cache/yum/x86_64/7/mysql80-community/packages/mysql-community-client-8.0.28-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 3a79bd29: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

The GPG keys listed for the "MySQL 8.0 Community Server" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.

 Failing package is: mysql-community-client-8.0.28-1.el7.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

 

出現上述提示原因是Mysql的GPG升級了，需要重新獲取

執行以下命令解決：

rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022

再次進行服務安裝：sudo yum -y install mysql-server就可以了

 

設置開機啟動mysql:

sudo systemctl enable mysqld

 

 

啟動mysql:

sudo systemctl start mysqld

 

 

查看Mysql狀態：

sudo systemctl status mysqld

 

 

查看root的臨時密碼：

sudo grep 'temporary password' /var/log/mysqld.log

 

 我們看到隨機生成的root臨時密碼是(注意每次隨機生成密碼不一樣，不要死記)

修改root密碼，先進入mysql命令行:

mysql -u root -p

 

 這時會要求輸入密碼，我們輸入上述臨時密碼然后回車即可，然后按照如下命令把root用戶的密碼改為root%123

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root%123';
Query OK, 0 rows affected (0.01 sec)

mysql> SHOW VARIABLES LIKE 'validate_password%';
+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password.check_user_name    | ON     |
| validate_password.dictionary_file    |        |
| validate_password.length             | 8      |
| validate_password.mixed_case_count   | 1      |
| validate_password.number_count       | 1      |
| validate_password.policy             | MEDIUM |
| validate_password.special_char_count | 1      |
+--------------------------------------+--------+
7 rows in set (0.00 sec)

mysql> set global validate_password.policy=0;
Query OK, 0 rows affected (0.01 sec)

mysql> set global validate_password.length=1;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root%123';
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye

 

 在生產上建議把root密碼設置的更加復雜，我這里設置的root只能本機連接

 

 3、MySQL特殊處理

 

在生產上Canal-Admin自己用的MySQL和你要增量采集binlog的MySQL大概率不是同一個，甚至在不

同的機器上。為了簡單，我們這里采用同一個MySQL，生產上自己要注意。

 

 

為Canal-Server采集數據創建用戶

mysql -u root -p

 

set global validate_password.policy=0; 
set global validate_password.length=1; 
CREATE USER canal IDENTIFIED BY 'canal'; 
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; 
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; 
FLUSH PRIVILEGES; 
exit

 

 

為Canal-Admin創建用戶

mysql -u root -p

 

set global validate_password.policy=0; 
set global validate_password.length=1; 
CREATE USER canaladmin IDENTIFIED BY 'canaladmin'; 
GRANT ALL ON canal_manager.* TO 'canaladmin'@'%'; 

  revoke all privileges,grant option from 'canaladmin'@'%';

FLUSH PRIVILEGES; exit

 

 

配置源MySQL的Binlog格式

sudo vi /etc/my.cnf

 

增加以下配置：

server-id=1 
log-bin=mysql-bin
binlog-format=ROW 
binlog-ignore-db=information_schema 
binlog-ignore-db=mysql 
binlog-ignore-db=performance_schema
binlog-ignore-db=sys

log-bin用於指定binlog日志文件名前綴，默認存儲在/var/lib/mysql 目錄下。

server-id用於標識唯一的數據庫，不能和別的服務器重復，建議使用ip的最后一段，默認值也不可以。

binlog-ignore-db：表示同步的時候忽略的數據庫。

binlog-do-db：指定需要同步的數據庫（如果沒有此項，表示同步所有的庫）。

配置完保存退出，然后重啟MySQL。

 

 

登錄mysql

mysql -uroot -p

如果能看到上面的圖，則binlog 就成功開啟了。

 

禁用explicit_defaults_for_timestamp

mysql -uroot -p

 

SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%';
set persist explicit_defaults_for_timestamp=0;
SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%';

 

 

重啟MySQL使上面修改生效：

sudo systemctl status mysqld

 

 

安裝MySQL的Java驅動

sudo mkdir -p /usr/share/java

 

將MySQL的java驅動mysql-connector-java-8.0.18.jar上傳到該目錄下

 

如果本地沒有改驅動板也可以通過wget下載一個

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.18/mysql-connector-java-8.0.18.jar

 

4、安裝Canal

 

下載安裝包

 

 wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.admin-1.1.4.tar.gz

 wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz

 

安裝Canal-Admin

 解壓縮

mkdir -p app/canal-admin
tar -zxvf canal.admin-1.1.4.tar.gz -C app/canal-admin
cd app/canal-admin

 

配置環境變量

sudo vim /etc/profile

末尾添加：

#canal-admin
export CANAL_ADMIN_HOME=/home/hadoop/app/canal-admin
export PATH=${CANAL_ADMIN_HOME}/bin:$PATH

使環境變量生效：

source /etc/profile

 

修改配置

server:
  port: 8089
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: canal01:3306
  database: canal_manager
  username: canal
  password: canal
  driver-class-name: com.mysql.jdbc.Driver
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

 

 

替換MySQL驅動包

我們安裝的MySQL8，但是Canal-Admin默認的MySQL驅動不支持MySQL8，因此需要替換驅動包：

ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_ADMIN_HOME/lib/mysql-connector-java-8.0.18.jar
rm $CANAL_ADMIN_HOME/lib/mysql-connector-java-5.1.40.jar

 

初始化元數據庫

登錄mysql執行以下命令：

source /home/hadoop/app/canal-admin/conf/canal_manager.sql

[hadoop@canal01 lib]$ mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.28 MySQL Community Server - GPL

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> source /home/hadoop/app/canal-admin/conf/canal_manager.sql
Query OK, 1 row affected, 2 warnings (0.00 sec)

Database changed
Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.01 sec)

Query OK, 0 rows affected, 2 warnings (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 2 warnings (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 4 warnings (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 4 warnings (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 6 warnings (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 2 warnings (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

mysql>

 

啟動canal-admin

sh $CANAL_ADMIN_HOME/bin/startup.sh

 

查看日志：

tail -n 100 admin.log

 

報錯了！！！

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-03-28 15:21:00.496 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in class path resource [com/alibaba/otter/canal/admin/config/EbeanConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
        at com.alibaba.otter.canal.admin.CanalAdminApplication.main(CanalAdminApplication.java:19)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
        ... 16 common frames omitted
Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
        at io.ebeaninternal.server.core.DefaultContainer.checkDataSource(DefaultContainer.java:323)
        at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:106)
        at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:35)
        at io.ebean.EbeanServerFactory.createInternal(EbeanServerFactory.java:109)
        at io.ebean.EbeanServerFactory.create(EbeanServerFactory.java:70)
        at com.alibaba.otter.canal.admin.config.EbeanConfig.ebeanServer(EbeanConfig.java:38)
        at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f.CGLIB$ebeanServer$0(<generated>)
        at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f$$FastClassBySpringCGLIB$$d3e49a6c.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f.ebeanServer(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 17 common frames omitted
Caused by: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
        at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
        at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:365)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:194)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:460)
        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:534)
        at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
        at io.ebeaninternal.server.core.DefaultContainer.checkDataSource(DefaultContainer.java:316)
        ... 32 common frames omitted

 

 在mysql的jdbc連接參數添加allowPublicKeyRetrieval=true

修改配置文件的jdbc連接參數

 

  重新啟動，並查看日志

 

 

 通過瀏覽器打開頁面 http://canal01:8089/#/login?redirect=%2Fdashboard

 

默認賬號密碼：admin/123456

 

 其他操作：

sh $CANAL_ADMIN_HOME/bin/stop.sh  //停止
sh  $CANAL_ADMIN_HOME/bin/restart.sh  //重啟

 

安裝Canal-Server

引入了canal-admin之后，canal-server之前面向命令行的運維方式需要有一些變化，主要的變化在於

配置體系上，每個server節點上不應該再去維護復雜而且冗長的

canal.properties/instance.properties，應該選擇以最小化、無狀態的方式去啟動，因此在canal 1.1.4

上，對於配置做了一些重構來支持canal-admin，同時也兼容了原先的命令行運維模式。

 

解壓縮安裝包：

cd ~; 
mkdir -p app/canal-server; 
tar -zxvf canal.deployer-1.1.4.tar.gz -C app/canal-server;

 

配置環境變量：

sudo vim /etc/profile

末尾新增：

#canal-server 
export CANAL_SERVER_HOME=/home/hadoop/app/canal-server 
export PATH=${CANAL_SERVER_HOME}/bin:$PATH

使環境變量生效：

source /etc/profile

 

配置canal-server

采用canal-admin來管理canal-server之后，不再需要按照以前的方式修改大量配置了。

cd $CANAL_SERVER_HOME/conf/; 
mv canal.properties canal.properties.bak; 
mv canal_local.properties canal.properties; 
vim  $CANAL_SERVER_HOME/conf/canal.properties;

 

配置內容如下：

# register ip
canal.register.ip =

# canal admin config
canal.admin.manager = canal01:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster = 

 

替換MySQL驅動包

我們安裝的MySQL8，但是Canal-Admin默認的MySQL驅動不支持MySQL8，因此需要替換驅動包

ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_SERVER_HOME/lib/mysql-connector-java-8.0.18.jar 
rm $CANAL_SERVER_HOME/lib/mysql-connector-java-5.1.47.jar

 

啟停canal-server

$CANAL_SERVER_HOME/bin/startup.sh 
$CANAL_SERVER_HOME/bin/stop.sh 
$CANAL_SERVER_HOME/bin/restart.sh