從MySQL遷移到MongoDB

前言

       生產上有部分MySQL業務需要遷移到MongoDB，為確保數據的一致性故采用離線遷移的方式。MySQL端導出為CSV格式，然后MongoDB端通過mongoimport方式導入。

MySQL端-導出

導出表數據為CSV格式，列與列之間以“,”分隔。

 

編輯mysql_export.sh腳本

[root]# vi mysql_export.sh
#!/bin/bash
MYSQL_SCHEMA="china"
for TABLE_NAME in `mysql -se "select table_name from information_schema.tables where table_schema='$MYSQL_SCHEMA';"`
do
  mysql $MYSQL_SCHEMA -e "select * from $TABLE_NAME into outfile '/tmp/$TABLE_NAME.csv' fields terminated by ',';"
done

 

執行腳本導出數據

[root]# sh mysql_export.sh

 

查看導出生成的CSV文件

[root]# ls -rtl /tmp/*.csv
-rw-rw-rw- 1 mysql mysql 8363 Jan 13 14:52 /tmp/city.csv
-rw-rw-rw- 1 mysql mysql  714 Jan 13 14:52 /tmp/provincial.csv

 

在MySQL端生成mongoimport.sh然后傳輸到MongoDB端：

編輯gene_mongoimport.sh腳本

[root]# vi gene_mongoimport.sh
#!/bin/bash

MYSQL_SCHEMA="china"
for TABLE_NAME in `mysql -se "select table_name from information_schema.tables where table_schema='$MYSQL_SCHEMA';"`
do
  echo "echo '----- $TABLE_NAME ------'" >> mongoimport.sh
  mysql -se "SELECT concat('mongoimport -u mysql -p Gaoyu@029 -d mysql -c ',table_name,' --fields ',GROUP_CONCAT(COLUMN_NAME SEPARATOR ','),' --file=/tmp/',table_name,'.csv --type=csv') FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = '$MYSQL_SCHEMA' and TABLE_NAME='$TABLE_NAME';" >> mongoimport.sh
done

 

執行腳本生成mongoimport.sh腳本

[root]# sh gene_mongoimport.sh
[root]# cat mongoimport.sh
echo '----- city -----'
mongoimport -u mysql -p Gaoyu@029 -d mysql -c city --fields cid,city,postcode,pid --file=/tmp/city.csv --type=csv


echo '----- provincial -----'
mongoimport -u mysql -p Gaoyu@029 -d mysql -c provincial --fields pid,provincial,postcode --file=/tmp/provincial.csv --type=csv

 

將mongoimport.sh腳本和CSV文件傳輸到MongoDB端。

MongoDB端-導入

創建mysql庫，創建mysql用戶

[root]# mongo -u admin -p Passwd admin
> use mysql
> db.createUser(
   {
     user: "mysql",
     pwd: "Gaoyu@029",
     roles: [ { role: "readWrite", db: "mysql" } ]
   }
   )

 

執行mongoimport.sh腳本導入數據

[root]# sh mongoimport.sh
----- city -----
2022-01-13T15:01:56.351+0800	connected to: mongodb://localhost/
2022-01-13T15:01:56.359+0800	391 document(s) imported successfully. 0 document(s) failed to import.
----- provincial -----
2022-01-13T15:01:56.379+0800	connected to: mongodb://localhost/
2022-01-13T15:01:56.387+0800	34 document(s) imported successfully. 0 document(s) failed to import.

數據比對

MySQL端查看數據條數：

(root@localhost) [mysql]> select table_name,table_rows from information_schema.tables where table_schema='china';
+------------+------------+
| table_name | table_rows |
+------------+------------+
| city       |        391 |
| provincial |         34 |
+------------+------------+

 

MongoDB查看數據條數：

[root]# mongo -u mysql -p Gaoyu@029 mysql
> db.city.count()
391
> db.provincial.count()
34

 

對比MySQL端與MongoDB端數據是否有差異：

TABLE_NAME	MySQL	MongoDB	差異條數
city	391	391	0
provincial	34	34	0

 

數據遷移完成。