MyCat垂直分庫

一.什么是垂直分庫

將一類功能的表從一個實例切分到另一個實例，橫向擴展實例，增加寫負載

目標：將1個實例的4類表拆分多4個實例中

二.垂直切分步驟

2.1收集分析業務模塊間的關系，能分幾個庫

2.2全量復制需要的數據到其他實例

root@localhost 21:51:  [imooc_db]> show tables;
+-----------------------+
| Tables_in_imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.01 sec)

root@localhost 21:51:  [imooc_db]> system ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:a3:fe:0a brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.141/24 brd 172.16.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.10.139/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:fe0a/64 scope link 
       valid_lft forever preferred_lft forever

將上面一個實例中的一個庫拆成4個庫，分別是imooc_db/order_db/product_db/customer_db

• 1.備份原數據庫並記錄事務點

# master導出單個庫的備份
SET GLOBAL log_timestamps = SYSTEM;（
/usr/local/mysql/bin/mysqldump -uroot -p123456 -hlocalhost -P3306 --socket=/usr/local/mysql/mysql.sock --single-transaction --master-data=2  imooc_db --triggers --routines --events > /tmp/all.sql

• 2.原數據庫中建立復制用戶

grant replication slave on *.* to 'repl'@'%' identified by '123456';

• 3.新實例上恢復備份數據庫，並更改數據庫名

mysql -uroot -p123456 -e"create database order_db";
mysql -uroot -p123456 order_db < /tmp/all.sql
mysql -uroot -p123456 -e"create database product_db";
mysql -uroot -p123456 product_db < /tmp/all.sql
mysql -uroot -p123456 -e"create database customer_db";
mysql -uroot -p123456 customer_db < /tmp/all.sql
######################################
ERROR 1840 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty
Query OK, 0 rows affected (0.01 sec)

root@localhost 23:55:  [(none)]> show master status;
+---------------+----------+--------------+------------------+-------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000001 |      154 |              |                  |                   |
+---------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

root@localhost 23:55:  [(none)]> show master status;
+---------------+----------+--------------+------------------+-------------------------------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                         |
+---------------+----------+--------------+------------------+-------------------------------------------+
| binlog.000001 |      154 |              |                  | ad083f85-9264-11e8-9e91-005056a3fe0a:1-85 |
+---------------+----------+--------------+------------------+-------------------------------------------+
1 row in set (0.00 sec)

• 4.新實例上配置復制鏈路

\help change master to
CHANGE MASTER TO
  MASTER_HOST='172.16.10.141',
  MASTER_USER='repl',
  MASTER_PASSWORD='123456',
  MASTER_PORT=3306,
  master_auto_position=1;

\help change replication filter
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, order_db));
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, product_db));
CHANGE REPLICATION FILTER
  REPLICATE_REWRITE_DB = ((imooc_db, customer_db));
start slave;
# 主庫插入數據驗證
root@localhost 00:12:  [imooc_db]> insert into region_info(parent_id,region_name,region_level) values(1,'guangdong',2);
# 從庫查看數據
root@localhost 00:13:  [order_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
+-----------+-----------+-------------+--------------+
1 row in set (0.00 sec)
# 其他2個也這么同步

• 5.新實例上啟動復制追上主庫，結果如下：

IP	dbname
172.16.10.141	imooc_db
172.16.10.142	order_db
172.16.10.143	product_db
172.16.10.144	customer_db

2.3配置mycat垂直分庫

• 1.配置schema.xml，需要授權grant select,insert,update,delete on *.* to 'bm_mycat'@'%' identified by '123456';

<schema name="imooc_db" checkSQLschema="false" sqlMaxLimit="100">
        <table name="company" primaryKey="ID" dataNode="dn3,dn2,dn1" rule="mod-long"/>
        <table name="order_cart" primaryKey="cart_id" dataNode="dn_orderdb" />
        <table name="order_customer_addr" primaryKey="customer_addr_id" dataNode="dn_orderdb" />
        <table name="order_detail" primaryKey="order_detail_id" dataNode="dn_orderdb" />
        <table name="order_master" primaryKey="order_id" dataNode="dn_orderdb" />
        
        <table name="region_info" primaryKey="region_id" dataNode="dn_orderdb" />        
        <table name="serial" primaryKey="id" dataNode="dn_orderdb" />          
        <table name="shipping_info" primaryKey="ship_id" dataNode="dn_orderdb" />     
        <table name="warehouse_info" primaryKey="w_id" dataNode="dn_orderdb" />    
        <table name="warehouse_proudct" primaryKey="wp_id" dataNode="dn_orderdb" />
        
        <table name="product_brand_info" primaryKey="brand_id" dataNode="dn_productdb" />
        <table name="product_category" primaryKey="category_id" dataNode="dn_productdb" />
        <table name="product_comment" primaryKey="comment_id" dataNode="dn_productdb" />
        <table name="product_info" primaryKey="product_id" dataNode="dn_productdb" />
        <table name="product_pic_info" primaryKey="product_pic_id" dataNode="dn_productdb" />
        <table name="product_supplier_info" primaryKey="supplier_id" dataNode="dn_productdb" />
        
        <table name="customer_balance_log" primaryKey="balance_id" dataNode="dn_customerdb" />
        <table name="customer_inf" primaryKey="customer_inf_id" dataNode="dn_customerdb" />
        <table name="customer_level_inf" primaryKey="customer_level" dataNode="dn_customerdb" />
        <table name="customer_login" primaryKey="customer_id" dataNode="dn_customerdb" />
        <table name="customer_login_log" primaryKey="login_id" dataNode="dn_customerdb" />
        <table name="customer_point_log" primaryKey="point_id" dataNode="dn_customerdb" />
        
</schema>

<dataNode name="dn_orderdb" dataHost="mysql10142" database="order_db" />
<dataNode name="dn_productdb" dataHost="mysql10143" database="product_db" />
<dataNode name="dn_customerdb" dataHost="mysql10144" database="customer_db" />

<dataHost name="mysql10142" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.142" url="172.16.10.142:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>
<dataHost name="mysql10143" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.143" url="172.16.10.143:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>
<dataHost name="mysql10144" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>show slave status</heartbeat>
        <writeHost host="172.16.10.144" url="172.16.10.144:3306" user="bm_mycat" password="123456">
        </writeHost>
</dataHost>

• 2.配置server.xml，

<user name="app_imooc">
        <property name="password">123456</property>
        <property name="schemas">imooc_db</property>
</user>

2.4通過mycat訪問db，應用端做訪問更改

[root@mycat01 logs]# mysql -uapp_imooc -p123456 -P8066
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'app_imooc'@'localhost' (using password: YES)
[root@mycat01 logs]# mysql -uapp_imooc -p123456 -P8066 -h172.16.10.142
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.8-mycat-1.5.1-RELEASE-20160525110043 MyCat Server (OpenCloundDB)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

app_imooc@172.16.10.142 01:59:  [(none)]> show databases;
+----------+
| DATABASE |
+----------+
| imooc_db |
+----------+
1 row in set (0.00 sec)

app_imooc@172.16.10.142 01:59:  [(none)]> use imooc_db;
Database changed
app_imooc@172.16.10.142 01:59:  [imooc_db]> show tables;
+-----------------------+
| Tables in imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.01 sec)

app_imooc@172.16.10.142 01:59:  [imooc_db]> 

2.5刪除原庫中已遷移的表

• 172.16.10.141 | imooc_db
• 172.16.10.142 | order_db

drop table product_brand_info   ;
drop table product_category     ;
drop table product_comment      ;
drop table product_info         ;
drop table product_pic_info     ;
drop table product_supplier_info;

drop table customer_balance_log ;
drop table customer_inf         ;
drop table customer_level_inf   ;
drop table customer_login       ;
drop table customer_login_log   ;
drop table customer_point_log   ;

root@localhost 02:07:  [order_db]> show tables;
+---------------------+
| Tables_in_order_db  |
+---------------------+
| order_cart          |
| order_customer_addr |
| order_detail        |
| order_master        |
| region_info         |
| serial              |
| shipping_info       |
| warehouse_info      |
| warehouse_proudct   |
+---------------------+
9 rows in set (0.00 sec)

• 172.16.10.143 | product_db

drop table order_cart           ;
drop table order_customer_addr  ;
drop table order_detail         ;
drop table order_master         ;
drop table region_info          ;
drop table serial               ;
drop table shipping_info        ;
drop table warehouse_info       ;
drop table warehouse_proudct    ;

drop table customer_balance_log ;
drop table customer_inf         ;
drop table customer_level_inf   ;
drop table customer_login       ;
drop table customer_login_log   ;
drop table customer_point_log   ;

root@localhost 02:10:  [product_db]> show tables;
+-----------------------+
| Tables_in_product_db  |
+-----------------------+
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
+-----------------------+
6 rows in set (0.00 sec)

• 172.16.10.144 | customer_db

drop table product_brand_info   ;
drop table product_category     ;
drop table product_comment      ;
drop table product_info         ;
drop table product_pic_info     ;
drop table product_supplier_info;

drop table order_cart           ;
drop table order_customer_addr  ;
drop table order_detail         ;
drop table order_master         ;
drop table region_info          ;
drop table serial               ;
drop table shipping_info        ;
drop table warehouse_info       ;
drop table warehouse_proudct    ;
root@localhost 02:08:  [customer_db]> show t
ables;
+-----------------------+
| Tables_in_customer_db |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
+-----------------------+
6 rows in set (0.00 sec)

2.6在mycat邏輯庫中查看，和拆分之前的物理庫是一樣的

app_imooc@172.16.10.142 02:11:  [imooc_db]> show tables;
+-----------------------+
| Tables in imooc_db    |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| order_cart            |
| order_customer_addr   |
| order_detail          |
| order_master          |
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
| serial                |
| shipping_info         |
| warehouse_info        |
| warehouse_proudct     |
+-----------------------+
21 rows in set (0.00 sec)

2.7訪問測試

app_imooc@172.16.10.142 23:14:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
+-----------+-----------+-------------+--------------+
2 rows in set (0.01 sec)

# 跨分片查詢是不允許的
app_imooc@172.16.10.142 23:14:  [imooc_db]> select * from product_supplier_info a join region_info b on a.province=b.region_id;
ERROR 1064 (HY000): invalid route in sql, multi tables found but datanode has no intersection  sql:select * from product_supplier_info a join region_info on a.province=b.region_id

三.全局表

• 適用字典表/元數據表
• 每個物理庫中都存在的表
• 由mycat來進行多寫進行數據一致性保證

3.1配置全局表過程

• 1.在原物理庫中將數據備份導出

mysqldump -uroot -p123456 --set-gtid-purged=OFF order_db region_info > /tmp/region_info.sql
scp /tmp/region_info.sql 172.16.10.143:/tmp
scp /tmp/region_info.sql 172.16.10.144:/tmp

• 2.將備份導入另外兩個物理庫

mysql -uroot -p123456 product_db < /tmp/region_info.sql
mysql -uroot -p123456 customer_db < /tmp/region_info.sql

• 3.此時3個物理庫中都存在全局表region_info

root@localhost 23:28:  [order_db]> show tables;
+---------------------+
| Tables_in_order_db  |
+---------------------+
| order_cart          |
| order_customer_addr |
| order_detail        |
| order_master        |
| region_info         |
| serial              |
| shipping_info       |
| warehouse_info      |
| warehouse_proudct   |
+---------------------+
9 rows in set (0.00 sec)

root@localhost 23:28:  [customer_db]> show tables;
+-----------------------+
| Tables_in_customer_db |
+-----------------------+
| customer_balance_log  |
| customer_inf          |
| customer_level_inf    |
| customer_login        |
| customer_login_log    |
| customer_point_log    |
| region_info           |
+-----------------------+
7 rows in set (0.00 sec)

root@localhost 23:28:  [product_db]> show tables;
+-----------------------+
| Tables_in_product_db  |
+-----------------------+
| product_brand_info    |
| product_category      |
| product_comment       |
| product_info          |
| product_pic_info      |
| product_supplier_info |
| region_info           |
+-----------------------+
7 rows in set (0.00 sec)

• 4.在schema.xml配置全局表屬性，只需要在表屬性中增加type="global"，並增加對應的物理節點信息即可

<table name="region_info" primaryKey="region_id" dataNode="dn_orderdb,dn_productdb,dn_customerdb" type="global"/>

• 5.重啟生效驗證，此時可以在一個物理庫內關聯查詢

app_imooc@172.16.10.142 23:35:  [imooc_db]> select * from product_supplier_info a join region_info b on a.province=b.region_id;
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
| supplier_id | supplier_code | supplier_name | supplier_type | link_man | phone_number | bank_name    | bank_account   | province | city | district | address | supplier_status | modified_time       | region_id | parent_id | region_name | region_level |
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
|           3 | 20001         | 供應商-3      |             1 | 王五     | 13800138003  | 中國銀行     | 12345656785443 |        2 |   52 |      503 | 北京    |               1 | 2018-07-28 21:50:57 |         2 |         1 | shanghai    |            3 |
+-------------+---------------+---------------+---------------+----------+--------------+--------------+----------------+----------+------+----------+---------+-----------------+---------------------+-----------+-----------+-------------+--------------+
1 row in set (0.02 sec)

• 6.在mycat上維護全局表

app_imooc@172.16.10.142 23:38:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
+-----------+-----------+-------------+--------------+
2 rows in set (0.00 sec)

app_imooc@172.16.10.142 23:38:  [imooc_db]> insert into region_info values(3,3,'beijing',4);
Query OK, 1 row affected (0.05 sec)

app_imooc@172.16.10.142 23:39:  [imooc_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)
# 在其他庫上也可以同時更新該表
root@localhost 23:28:  [product_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)

root@localhost 23:28:  [customer_db]> select * from region_info;
+-----------+-----------+-------------+--------------+
| region_id | parent_id | region_name | region_level |
+-----------+-----------+-------------+--------------+
|         1 |         1 | guangdong   |            2 |
|         2 |         1 | shanghai    |            3 |
|         3 |         3 | beijing     |            4 |
+-----------+-----------+-------------+--------------+
3 rows in set (0.00 sec)

四.垂直切分的優缺點

4.1優點

• 1.拆分簡單明了，拆分規則明確
• 2.應用程序模塊清晰明確，整合容易
• 3.數據維護方便易行，容易定位

4.2缺點

• 1.部分表關聯無法在數據庫級別完成，需要在程序中完成
• 2.對於訪問機器頻繁且數據量超大的表任然存在性能瓶頸
• 3.切分達到一定程度之后，擴展性會遇到限制，單表性能依然存在瓶頸

4.3解決跨分片關聯的方式

• 1.適用mycat全局表，在每個庫中冗余數據
• 2.在各個表中冗余字段，增加關鍵數據
• 3.適用API方式獲取數據，再進行拼接