利用Go實現MySQL binlog與 es數據同步

1.安裝go

go官網：https://golang.google.cn/dl/
注意！ 安裝go的同時也需要安裝git

wget https://golang.google.cn/dl/go1.15.2.linux-amd64.tar.gz
tar -C /usr/local -zxvf go1.15.2.linux-amd64.tar.gz

配置環境變量vim /etc/profile

export GOROOT=/usr/local/go
export GOPATH=/data/bogiang/go
export PATH=$PATH:/usr/local/go/bin

2.修改MySQL binlog日志模式

需要值得注意的是， go-mysql-elasticsearch 支持binlog的日志模式為row模式因此建議在配置MySQLbinlog的時候需指定該模式

log-bin=mysql-bin
binlog_format=row

3.go-mysql-elasticsearch安裝

go env -w GOPROXY=https://goproxy.cn
go get go get github.com/siddontang/go-mysql-elasticsearch
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
make

配置river.toml

#  MySQL 配置：地址，用戶名，密碼
my_addr = "xxxx:3306"
my_user = "xxxx"
my_pass = "xxxx"

# Elasticsearch地址
es_addr = "xxxxx"

# 存儲數據的位置
data_dir = "/var"

# Inner Http status address
stat_addr = "xxxxx:12800"
stat_path = "/es"
# pseudo server id like a slave
server_id = 1
# mysql or mariadb
flavor = "mysql"
# mysqldump execution path
# mysqldump = "mysqldump"
# minimal items to be inserted in one bulk
bulk_size = 128

# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time = "200ms"

# Ignore table without primary key
skip_no_pk_table = false

# elasticsearch 與 mysql 同步時對應的數據庫名稱
# mysql的數據源
[[source]]
schema = "lw"
tables = ["goods"]

# es 映射的mapping
[[rule]]
schema = "lw"
table = "goods"
# es的索引名
index = "lw_goods"
type = "link_info"

啟動服務

$GOPATH/src/github.com/siddontang/go-mysql-elasticsearch/bin/go-mysql-elasticsearch -config=/xxx/river.toml