TiDB數據庫 mydumper與loader導入數據

從mysql導出數據最好的方法是使用tidb官方的工具mydumper。

導入tidb最好的方法是使用loader工具，大概19.4G每小時的速度。

詳細的步驟可以參考官網：https://pingcap.com/docs-cn/op-guide/migration-overview/

自己的總結：以前將mysq導出數據的時候，是一個大的sql文件，比如這次就是從mysql中導出一個41G大的文件，這是單個的。以為這樣子就能使用loader命令導入到tidb中，最后發現錯了。loader命令的導入是將一個目錄下的sql文件導入到數據庫中，那么問題就來了，像我們之前那樣只有一個大的sql文件是不能使用loader命令導入數據庫的。（這里的loader是tidb官方自己研發的）最后使用了source命令導入了這個大的sql文件。

所以說要使用loader命令導入數據，那么前提是就要用mydumper命令導出數據，這樣的數據才能被loader導入進去。

mydumper命令是將一個表導出到一個目錄下面，這個表比較大的時候，mydumper就會分成以64M為大小的一個個的小文件（比如這次的41G的文件分成了600多個小文件）。

 

當然使用導入導出命令也踩了一些坑

報錯：

1、導入數據出現10084錯誤

這是因為端口被占用，導致不能重新寫入，使用netstat -lnp| grep 10084命令找到該端口10084的對應的進程，然后使用kill命令殺死進程即可。

 

 2、tidb_loader數據庫下check_point表的作用

 

解釋：當我們導入tidb數據庫數據之后，這個check_point表都會產生一行數據，用來記錄導入數據的狀態信息以及導入時間等。

 

如果你導入一次數據之后，check_point以及記錄一次了，這個時候你把導入的那個表刪除了，但是沒有刪除check_point的導入記錄，那么現在你繼續重新導入之前的那個表，會發現導入失敗。

下面是講解check_point表的作用：

 

mysql> select * from haha
    -> ;
+------+------+
| id   | name |
+------+------+
|    1 | chen |
|    2 | chao |
|    3 | feng |
+------+------+
3 rows in set (0.00 sec)

mysql> drop table haha；
Query OK, 0 rows affected (0.25 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| INFORMATION_SCHEMA |
| PERFORMANCE_SCHEMA |
| mob                |
| mysql              |
| test               |
| tidb_loader        |
+--------------------+
6 rows in set (0.00 sec)

mysql> use tidb_loader;
Database changed
mysql> show tables;
+-----------------------+
| Tables_in_tidb_loader |
+-----------------------+
| checkpoint            |
+-----------------------+
1 row in set (0.00 sec)

mysql> select * from checkpoint;
+--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+
| id     | filename      | cp_schema | cp_table | offset | end_pos | create_time         | update_time         |
+--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+
| 32c716 | test.haha.sql | test      | haha     |     98 |      98 | 2018-11-16 12:00:08 | 2018-11-16 12:00:08 |
+--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+
1 row in set (0.00 sec)

mysql> exit
Bye
[tidb@:vg_adn_tidbCkhsTest:54.158.254.36:172.31.30.62 /usr/local/tidb-tools]$sudo bin/loader -h 127.0.0.1 -u root -P 4000 -t 1 -d /data/haha
2018/11/16 12:01:20 printer.go:52: [info] Welcome to loader
2018/11/16 12:01:20 printer.go:53: [info] Release Version: v1.0.0-47-gbaf98f6
2018/11/16 12:01:20 printer.go:54: [info] Git Commit Hash: baf98f63ae77b2d9a22e00bb3d1555452b21e4f9
2018/11/16 12:01:20 printer.go:55: [info] Git Branch: master
2018/11/16 12:01:20 printer.go:56: [info] UTC Build Time: 2018-11-12 01:34:40
2018/11/16 12:01:20 printer.go:57: [info] Go Version: go version go1.11 linux/amd64
2018/11/16 12:01:20 main.go:51: [info] config: {"log-level":"info","log-file":"","pprof-addr":":10084","pool-size":1,"dir":"/data/haha","db":{"host":"127.0.0.1","user":"root","port":4000},"checkpoint-schema":"tidb_loader","config-file":"","alternative-db":"","source-db":"","route-rules":[],"do-table":null,"do-db":null,"ignore-table":null,"ignore-db":null,"rm-checkpoint":false}
2018/11/16 12:01:20 checkpoint.go:82: [info] [loader] load checkpoint takes 0.005710 seconds
2018/11/16 12:01:20 loader.go:524: [info] [loader] prepare takes 0.000085 seconds
2018/11/16 12:01:20 checkpoint.go:207: [info] calc checkpoint finished. finished tables (map[test.haha:{}])
2018/11/16 12:01:20 loader.go:685: [info] [loader][run db schema]/data/haha/test-schema-create.sql[start]
2018/11/16 12:01:20 db.go:112: [warning] [exec][sql][CREATE DATABASE `test` /* !40100 DEFAULT CHARACTER SET utf8 */;][error]Error 1007: Can't create database 'test'; database exists
2018/11/16 12:01:20 loader.go:689: [info] [loader][database already exists, skip]/data/haha/test-schema-create.sql
2018/11/16 12:01:20 loader.go:694: [info] [loader][run db schema]/data/haha/test-schema-create.sql[finished]
2018/11/16 12:01:20 loader.go:704: [info] table (test.haha) has finished, skip.
2018/11/16 12:01:20 loader.go:750: [info] [loader] create tables takes 0.004609 seconds
2018/11/16 12:01:20 loader.go:757: [info] [loader] all data files have been dispatched, waiting for them finished 
2018/11/16 12:01:20 loader.go:760: [info] [loader] all data files has been finished, takes 0.004650 seconds
2018/11/16 12:01:20 main.go:88: [info] loader stopped and exits 
[tidb@:vg_adn_tidbCkhsTest:54.158.254.36:172.31.30.62 /usr/local/tidb-tools]$mysql -u root -P 4000 -h 127.0.0.1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2564
Server version: 5.7.10-TiDB-v2.0.8 MySQL Community Server (Apache License 2.0)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test
Database changed
mysql> show tables;
Empty set (0.00 sec)

mysql>