通過kafka處理
- 下載；
重新進行ansible-playbook bootstrap.yml
兩個pump和drainer在下載的resource/bin/下面。拷貝到 deploy/bin/目錄下。
- 修改 inventory.ini
設置enable_binlog = True
設置zk地址，不然會報錯，一定需要zk
注釋原來的#[pump_servers:children]
增加
[pump_servers]
192.xxx.xxx.205
192.xxx.xxx.218
3.滾動升級tidb節點，部署pump
嘗試 ansible-playbook rolling_update.yml –tags=tidb
報錯如下：
TASK [stop pump by systemd] *******************************************************************************************************************************************************************
fatal: [192.xxx.xxx.205]: FAILED! => {"changed": false, "msg": "Could not find the requested service pump.service: host"}
測試自定義systemd 加入pump
[Unit]
Description=pd service
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
LimitNOFILE=1000000
User=tidb
ExecStart=/home/tidb/deploy2/bin/pump -config pump.toml
Restart=always
RestartSec=15s
[Install]
WantedBy=multi-user.target
配置使用到的pump.toml
# pump Configuration.
# pump 提供服務的 rpc 地址(默認 "127.0.0.1:8250")
addr = "127.0.0.1:8250"
# pump 對外提供服務的 rpc 地址(默認 "127.0.0.1:8250")
advertise-addr = ""
# binlog 最大保留天數 (默認 7)， 設置為 0 可永久保存
gc = 3
# pump 數據存儲位置路徑
data-dir = "data.pump"
# pump 向 pd 發送心跳間隔 (單位 秒)
heartbeat-interval = 3
# pd 集群節點的地址 (默認 "http://127.0.0.1:2379")
pd-urls = "http://192.xxx.xxx.218:2379"
# unix socket 模式服務監聽地址 (默認 unix:///tmp/pump.sock)
socket = "unix:///tmp/pump.sock"
#log-file
log-file = "/home/tidb/deploy2/log/pump.log"
發現systemd啟動不了，查看日志 ，路徑為
/var/log/messages
Nov 20 10:34:12 TEST-1807-V002 pump: 2018/11/20 10:34:12 main.go:23: #033[0;31m[fatal] verifying flags error, open pump.toml: no such file or directory. See 'pump --help'.#033[0m
Nov 20 10:34:12 TEST-1807-V002 systemd: pump.service: main process exited, code=exited, status=255/n/a
Nov 20 10:34:12 TEST-1807-V002 systemd: Unit pump.service entered failed state.
Nov 20 10:34:12 TEST-1807-V002 systemd: pump.service failed.
是路徑不對：修改后 systemctl daemon-reload
再次重啟
[root@TEST-1807-V002 ~]# ps -ef|grep pump
tidb 23063 1 0 10:37 ? 00:00:00 /home/tidb/tidb-binlog/bin/pump -configl /home/tidb/depoy/conf/pump.toml
root 23092 21306 0 10:38 pts/1 00:00:00 grep --color=auto pump
可以啟動
嘗試停止
[root@TEST-1807-V002 ~]# systemctl stop pump.service
[root@TEST-1807-V002 ~]# ps -ef|grep pump
root 23136 21306 0 10:40 pts/1 00:00:00 grep --color=auto pump
停止成功
其實上面的主要是升級程序能走通，后面升級程序會自己生成腳本，進行部署，deploy.
再次ansible-playbook rolling_update.yml –tags=tidb
PLAY RECAP ************************************************************************************************************************************************************************************
192.xxx.xxx.205 : ok=48 changed=6 unreachable=0 failed=0
192.xxx.xxx.218 : ok=49 changed=12 unreachable=0 failed=0
192.xxx.xxx.219 : ok=8 changed=0 unreachable=0 failed=0
192.xxx.xxx.220 : ok=8 changed=0 unreachable=0 failed=0
192.xxx.xxx.221 : ok=9 changed=0 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
Congrats! All goes well. :-)
審計tidb 節點成功升級，部署pump進程成功。
4.部署drainer
官方文檔上寫明需要手工部署。
a.配置drainer配置文件：
# drainer Configuration.
# drainer 提供服務的地址(默認 "127.0.0.1:8249")
addr = "192.xxx.xxx.218:8249"
# 向 pd 查詢在線 pump 的時間間隔 (默認 10，單位 秒)
detect-interval = 10
# drainer 數據存儲位置路徑 (默認 "data.drainer")
data-dir = "/home/tidb/deploy2/data.drainer"
#pd-urls = "http://192.xxx.xxx.205:2379,http://192.xxx.xxx.218:2379,http://192.xxx.xxx.219:2379"
# log 文件路徑
log-file = "/home/tidb/deploy2/log/drainer.log"
zookeeper-addrs="192.168.110.22:2181"
kafka-addrs="192.xxx.xxx.66:9092,192.xxx.xxx.67:9092"
kafka-version="0.10.2.1"
# Syncer Configuration.
[syncer]
## db 過濾列表 (默認 "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql,test"),
## 不支持對 ignore schemas 的 table 進行 rename DDL 操作
ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql"
# 輸出到下游數據庫一個事務的 sql 數量 (default 1)
txn-batch = 1
# 同步下游的並發數，該值設置越高同步的吞吐性能越好 (default 1)
worker-count = 1
# 是否禁用拆分單個 binlog 的 sqls 的功能，如果設置為 true，則按照每個 binlog
# 順序依次還原成單個事務進行同步( 下游服務類型為 mysql, 該項設置為 False )
disable-dispatch = false
# drainer 下游服務類型 (默認為 mysql)
# 參數有效值為 "mysql", "pb"
db-type = "mysql"
# replicate-do-db priority over replicate-do-table if have same db name
# and we support regex expression ,
# 以 '~' 開始聲明使用正則表達式
#replicate-do-db = ["~^b.*","s1"]
#[[syncer.replicate-do-table]]
#db-name ="test"
#tbl-name = "log"
#[[syncer.replicate-do-table]]
#db-name ="test"
#tbl-name = "~^a.*"
# db-type 設置為 mysql 時，下游數據庫服務器參數
[syncer.to]
host = "192.168.216.22"
user = "drainer"
password = "123456"
port = 3306
主要需要配置的上面標紅的內容。
b.配置systemd
[Unit]
Description=drainer service
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
LimitNOFILE=1000000
User=tidb
ExecStart=/home/tidb/deploy2/bin/drainer -config /home/tidb/deploy2/conf/drainer.toml
Restart=always
RestartSec=15s
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
為了測試效果：
在測試庫中新建數據庫
MySQL [antiml]> create database caoyf;
Query OK, 0 rows affected (1.04 sec)
MySQL [caoyf]> create table test(id int);
Query OK, 0 rows affected (2.04 sec)
MySQL [caoyf]> insert into test values(1);
Query OK, 1 row affected (0.14 sec)
MySQL [caoyf]> select * from test;
+------+
| id |
+------+
| 1 |
| 2 |
+------+
2 rows in set (0.01 sec)
c. 生成 drainer savepint 文件
/home/tidb/deploy2/bin/drainer -gen-savepoint --data-dir=/home/tidb/deploy2/data.drainer --pd-urls=http://192.xxx.xxx.205:2379,http://192.xxx.xxx.218:2379,http://192.xxx.xxx.219:2379
在drainer配置文件中配置同步的起點tso
查看
為了測試，先全量備份數據
./mydumper -h 192.xxx.xxx.205 -P 4000 -u root -p 'xxxx' -t 4 -F 64 -B caoyf --skip-tz-utc -o /tmp/caoyf
傳到對應的mysql機器
在新的庫進行恢復
./loader -u root -p xxxxx -P 3306 -d /tmp/caoyf
查看已恢復：
mysql> show tables;
+-----------------+
| Tables_in_caoyf |
+-----------------+
| test |
+-----------------+
1 row in set (0.00 sec)
mysql> select * from test;
+------+
| id |
+------+
| 1 |
| 2 |
+------+
2 rows in set (0.00 sec)
再在tidb插入幾條數據：
MySQL [(none)]> use caoyf
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MySQL [caoyf]> insert into test values(3);
Query OK, 1 row affected (0.02 sec)
MySQL [caoyf]> insert into test values(4);
Query OK, 1 row affected (0.02 sec)
d.啟動drainer （希望能成功！！！！！）
報錯，顯示連不到kafka
需要配置pump 進行指定kafka,特別注意各版本的參數不一樣，需要仔細查看對應的版本。-help查看參數內容。
后修改/home/tidb/deploy2/scripts/run_pump.sh
exec bin/pump \
--gc="5" \
--addr="0.0.0.0:8250" \
--advertise-addr="192.xxx.xxx.218:8250" \
--pd-urls="http://192.xxx.xxx.205:2379,http://192.xxx.xxx.218:2379,http://192.xxx.xxx.219:2379" \
--zookeeper-addrs="192.168.110.22:2181" \
--data-dir="/home/tidb/deploy2/data.pump" \
--socket="/home/tidb/deploy2/status/pump.sock" \
--log-file="/home/tidb/deploy2/log/pump.log" \
--kafka-addrs="192.xxx.xxx.67:9092,192.xxx.xxx.66:9092"\
--zookeeper-addrs="192.168.110.22:2181"\
--kafka-version="0.10.2.1"\
--config=conf/pump.toml
后啟動drainer
systemctl start drainer.service
查看日志：
2018/11/21 15:30:31 config.go:382: [sarama] ClientID is the default of 'sarama', you should consider setting it to something application-specific.
2018/11/21 15:30:31 broker.go:146: [sarama] Connected to broker at 192.xxx.xxx.66:9092 (registered as #1)
2018/11/21 15:30:31 consumer.go:712: [sarama] consumer/broker/1 added subscription to 6626154907404840978_TEST-1807-V003_8250/0
2018/11/21 15:34:11 syncer.go:453: [info] [ddl][start]create database test_caoyf;[commit ts]404431999019253762[pos]{0 908}
2018/11/21 15:34:11 syncer.go:255: [info] [write save point]404431999019253762[positions]map[TEST-1807-V002:8250:{0 908}]
2018/11/21 15:34:11 syncer.go:462: [info] [ddl][end]create database test_caoyf;[commit ts]404431999019253762[pos]{0 908}
2018/11/21 15:35:51 syncer.go:453: [info] [ddl][start]use `test_caoyf`; create table test1(id int);;[commit ts]404432026819624961[pos]{0 944}
2018/11/21 15:35:51 syncer.go:255: [info] [write save point]404432026819624961[positions]map[TEST-1807-V002:8250:{0 944}]
2018/11/21 15:35:51 syncer.go:462: [info] [ddl][end]use `test_caoyf`; create table test1(id int);;[commit ts]404432026819624961[pos]{0 944}
一些測試。日志顯示成功，在下游mysql 查看包括建庫，見表，簡單插入同步沒有問題。