spring+mybatis的插件【shardbatis2.0】+mysql+java自定義注解實現分表

一、業務場景分析

只有大表才需要分表，而且這個大表還會有經常需要讀的需要，即使經過sql服務器優化和sql調優，查詢也會非常慢。例如共享汽車的定位數據表等。

二、實現步驟

1.准備pom依賴

<dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.30</version>
    </dependency>
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>druid</artifactId>
        <version>1.1.3</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-jdbc</artifactId>
        <version>4.3.11.RELEASE</version>
    </dependency>

    <dependency>
        <groupId>org.mybatis</groupId>
        <artifactId>mybatis-spring</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.mybatis</groupId>
        <artifactId>mybatis</artifactId>
        <version>3.2.3</version>
    </dependency>

這里關鍵是要額外引入 插件shardbatis 相關的依賴，主要有兩個：

<dependency>
        <groupId>org.shardbatis</groupId>
        <artifactId>shardbatis</artifactId>
        <version>2.0.0B</version>
    </dependency>
    <dependency>
        <groupId>net.sf.jsqlparser</groupId>
        <artifactId>jsqlparser</artifactId>
        <version>0.7.0</version>
    </dependency>

2.准備表

把原來的t_location單表拆分成t_location_01、t_location_02、t_location_03、t_location_04、t_location_05、t_location_06

3.准備好mybatis的mapper interface

public interface UserMapper {
        int deleteByPrimaryKey(Integer id);
        int insert(User record);
        int insertSelective(User record);
        User selectByPrimaryKey(Integer id);
        int updateByPrimaryKeySelective(User record);
        int updateByPrimaryKey(User record);
    }

對應的sql這里就省略了，shardbatis這個插件使用時也不需要去調整實際的sql，插件達到的效果就是替換掉實際sql中的表名。

4.新增一個shard_config.xml文件

<?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE shardingConfig PUBLIC "-//shardbatis.googlecode.com//DTD Shardbatis 2.0//EN"
            "http://shardbatis.googlecode.com/dtd/shardbatis-config.dtd">
    <shardingConfig>
        <!--
            parseList可選配置
            如果配置了parseList,只有在parseList范圍內並且不再ignoreList內的sql才會被解析和修改
        -->
        <ignoreList>
            <value>xxx.xxx</value>
        </ignoreList>
        <parseList>
            <value>xxx.dao.UserMapper.insertSelective</value>
            <value>xxx.dao.UserMapper.selectByPrimaryKey</value>
            <value>xxx.UserMapper.updateByPrimaryKeySelective</value>
        </parseList>
        <!--
            配置分表策略
            tableName指的是實際的表名，strategyClass對應的分表策略實現類
        -->
        <strategy tableName="location" strategyClass="xxx.DeviceShardStrategyImpl"/>
    </shardingConfig>

並在項目的mybatis-config.xml里聲明使用這個插件，比如

<?xml version="1.0" encoding="UTF-8" ?>
    <!DOCTYPE configuration PUBLIC "-//mybatis.org//DTD Config 3.0//EN" "http://mybatis.org/dtd/mybatis-3-config.dtd">

    <configuration>
        <plugins>
            <plugin interceptor="com.google.code.shardbatis.plugin.ShardPlugin">
                <property name="shardingConfig" value="shard-config.xml"/>
            </plugin>
        </plugins>
    </configuration>

5.實現分表策略

就是完成上面strategyClass對應的分表策略實現類，其實只需要實現ShardStrategy接口並實現其中的getTargetTableName方法即可，比如：

public class DeviceShardStrategyImpl implements ShardStrategy {

        private final static int tableCount = 5;
        /**
         * 得到實際表名
         * @param baseTableName 邏輯表名,一般是沒有前綴或者是后綴的表名
         * @param params mybatis執行某個statement時使用的參數
         * @param mapperId mybatis配置的statement id
         * @return
         */
        @Override
        public String getTargetTableName(String baseTableName, Object params, String mapperId) {
            // TODO: 需要根據實際的參數或其他（比如當前時間）計算出一個滿足要求的值
            int value = 2;
            try {
                int index = value % tableCount + 1;
                String strIndex = "0" + index;
                return baseTableName + "_" + strIndex;
            } catch (Exception e) {
                throw new RuntimeException(e.getMessage(), e);
            }
        }
    }

實際中實現需要根據實際的參數或其他（比如當前時間）計算出一個滿足要求的值，最后拼接成實際的表名就可以了。當然了，這個【滿足要求的值】有時要計算起來會特別麻煩。這里呢，說一說我自己在實際項目中計算value的一個設計和實現。

實際業務講解

假設有1000輛被客戶使用的共享汽車，假設每輛車每天跑4個小時，每3S一條定位數據，那樣一天下來定位數據在60*60*4*1000/3=480w這個量級，實際存儲的數據在350w~450w之間，這些數據都需要插入到數據庫中。我們知道對於一般的數據庫而言，單表達到百萬甚至千萬級別時，任何操作即使是select count(1)也會變得很慢，這時分表是必須的。

具體說一下分表的策略：假設我們要把原來的大表拆分成512張小表，以設備為維度進行水平拆分，每次對表執行插入時，找到對應的設備(設備表t_device，設備和車輛是一對一) 的 id，使用設備id%512作為表后綴。

舉個例子，車輛的定位數據存儲的表為t_location_000 ~ t_location_511（注意不是1~512）, 設備A在t_device表里的id為513，那么A對應的定位數據存儲表為：513%512=1 -> t_location_001, 設備B在t_device表里的id為1128，那么B對應的分時數據存儲表為：1128%512=104 -> t_location_104。相信這個不難理解，接下來的問題就是如何從 public String getTargetTableName(String baseTableName, Object params, String mapperId) 這個方法里取出我們說的t_device A和B了，根據代碼上的解釋，我們可以知道A和B要從 Object params 里解析出來。

注意，接下來是重點！！！

為了盡可能通用，我們自定義一個注解，@DeviceShard

@Retention(RetentionPolicy.RUNTIME)
    @Target({ElementType.PARAMETER})
    public @interface DeviceShard {
        String value() default "";
    }

分表策略使用的參數要求必須使用這個，比如：

 int insertSelective(@DeviceShard String instrumentId, @Param("timeTrend")TimeTrend record);

接下來增加一個根據Object params解析標識有@DeviceShard注解的參數的實際值，直接給出代碼：

public class DeviceShardValue {
        private DeviceShardValue() {}

        public static Object[] getShardValue(Object params, String mapperId) throws Exception {

            if(params != null && StringUtils.isNotBlank(mapperId)) {
                int lastPoint = mapperId.lastIndexOf(".");
                String clazzName = mapperId.substring(0,lastPoint);
                String methodName = mapperId.substring(lastPoint + 1);
                Class clazz = Class.forName(clazzName);
                Method[] methods = clazz.getMethods();

                if (methods.length <= 0){
                    throw new Exception("class has no method!");
                }

                List<Integer> shardFieldIndexes = new ArrayList<>();
                List<Class> shardFieldTypes = new ArrayList<>();
                List<SymbolShard> fieldAnnotations = new ArrayList<>();

                for (Method method : methods) {
                    if (methodName.equals(method.getName())) {
                        Annotation[][] annotations = method.getParameterAnnotations();
                        if(annotations == null || annotations.length <= 0){
                            throw new Exception("method has no shard field");
                        }
                        for (int i = 0; i < annotations.length; i++){
                            Annotation[] fieldAnno = annotations[i];
                            if (fieldAnno != null && fieldAnno.length > 0) {
                                for (Annotation annotation:fieldAnno) {
                                    if (annotation.annotationType() == SymbolShard.class){
                                        shardFieldIndexes.add(i);
                                        shardFieldTypes.add(method.getParameterTypes()[i]);
                                        fieldAnnotations.add((SymbolShard)annotation);
                                    }
                                }
                            }
                        }
                    }
                }
                if (shardFieldIndexes.size() <= 0){
                    throw new Exception("method has no shard field");
                }

                Object[] values = new Object[shardFieldIndexes.size()];
                for (int i = 0; i < shardFieldIndexes.size(); i++){
                    int shardFieldIndex = shardFieldIndexes.get(i);
                    Class shardFieldType = shardFieldTypes.get(i);
                    DeviceShard fieldAnnotation = fieldAnnotations.get(i);
                    if (params.getClass() == shardFieldType) {
                        values[i] = getFieldValue(fieldAnnotation,params);
                    } else {
                        String key = "param" + (shardFieldIndex+1);
                        HashMap<String,Object> map = (HashMap<String,Object>) params;
                        Object tmp = map.get(key);
                        values[i] = getFieldValue(fieldAnnotation, tmp);
                    }
                }

                return values;
            }

            return null;
        }

        private static Object getFieldValue(JmxShard fieldAnnotation,Object params) throws Exception {
            if(isBasicType(params)) {
                return params;
            } else {
                String shardFieldName = fieldAnnotation.value();
                if(StringUtils.isBlank(shardFieldName)) {
                    throw new Exception("the shardFieldName was not annotated");
                }
                Field field = null;
                try {
                    field = params.getClass().getDeclaredField(shardFieldName);
                } catch (NoSuchFieldException e){
                    field = params.getClass().getSuperclass().getDeclaredField(shardFieldName);
                }
                field.setAccessible(true);
                return field.get(params);
            }
        }

        private static boolean isBasicType(Object param){
            if (param == null){
                return false;
            }
            if (param instanceof String){
                return true;
            }
            if (param instanceof BigDecimal){
                return true;
            }
            if (param instanceof Integer){
                return true;
            }
            if (param instanceof Long){
                return true;
            }
            if (param instanceof Double){
                return true;
            }
            if (param instanceof Float){
                return true;
            }
            if (param instanceof Character){
                return true;
            }
            if (param instanceof Byte){
                return true;
            }
            if (param instanceof Short){
                return true;
            }
            if (param instanceof Boolean){
                return true;
            }
            return false;
        }
    }

接下來去實現ShardStrategy就很容易了（個別細節忽略）：

public class TimeTrendShardStrategyImpl implements ShardStrategy{

        private final static Integer tableCount = 512;

        @Override
        public String getTargetTableName(String baseTableName, Object params,String mapperId) {
            Object value;
            try {
                // 調用封裝的工具類獲取傳入標識有@SymbolShard注解的參數的值
                value = DeviceShardValue.getShardValue(params, mapperId)[0];

                // 連接數據庫，去symbols表查詢，注意這里不是使用自動注入@Autowired和@Resource，這種方式在權限課程里介紹過
                String device = value.toString();
                Device deviceInstance = SpringContextHolder.getBean(DeviceMapper.class).selectByDevice(device);
                if(symbolInstance == null) { // 如果查不到對應的symbol實例, 則返回一個普通表,否則這里會拋一個上層不知道的異常
                    return baseTableName + "_000";
                }

                // 根據id拼裝實際的分表后的表名
                Integer index = deviceInstance.getId() % tableCount;
                String strIndex = "";
                if(index < 10) {
                    strIndex = "00" + index;
                } else if(index < 100) {
                    strIndex = "0" + index;
                } else {
                    strIndex = "" + index;
                }
                return baseTableName + "_" + strIndex;
            } catch (Exception e) {
                throw new RuntimeException(e.getMessage(), e);
            }
        }
    }

寫到這里，再提醒一下，別忘了把分表的方法和分表策略寫入shard_config.xml

6.適用范圍

還要注意這個插件的適用范圍，我自己在上面踩過坑，就是在做數據庫批量操作時使用這個插件會沒有效果。

具體支持哪些sql呢，網上有人給了總結，我直接引用一下：

select * from test_table1
    select * from test_table1 where col_1='123'
    select * from test_table1 where col_1='123' and col_2=8
    select * from test_table1 where col_1=?
    select col_1,max(col_2) from test_table1 where col_4='t1' group by col_1
    select col_1,col_2,col_3 from test_table1 where col_4='t1' order by col_1
    select col_1,col_2,col_3 from test_table1 where id in (?,?,?,?,?,?,?,?,?) limit ?,?
    select a.*  from test_table1 a,test_table2 b where a.id=b.id and a.type='xxxx'
    select a.col_1,a.col_2,a.col_3 from test_table1 a where a.id in (select aid from test_table2 where col_1=1 and col_2=?) order by id desc
    select col_1,col_2 from test_table1 where type is not null and col_3 is null order by id
    select count(*),col_1 from test_table2 group by col_1 having count(*)>1
    select a.col_1,a.col_2,b.col_1 from test_table1 a,t_table b where a.id=b.id
    insert into test_table1 (col_1,col_2,col_3,col_4) values (?,?,?,?)
    SELECT EMPLOYEEIDNO FROM test_table1 WHERE POSITION = 'Manager' AND SALARY > 60000 OR BENEFITS > 12000
    SELECT EMPLOYEEIDNO FROM test_table1 WHERE POSITION = 'Manager' AND (SALARY > 50000 OR BENEFIT > 10000)
    SELECT EMPLOYEEIDNO FROM test_table1 WHERE LASTNAME LIKE 'L%'
    SELECT DISTINCT SELLERID, OWNERLASTNAME, OWNERFIRSTNAME FROM test_table1, test_table2 WHERE SELLERID = OWNERID ORDER BY OWNERLASTNAME, OWNERFIRSTNAME, OWNERID
    SELECT OWNERFIRSTNAME, OWNERLASTNAME FROM test_table1 WHERE EXISTS (SELECT * FROM test_table2 WHERE ITEM = ?)
    SELECT BUYERID, ITEM FROM test_table1 WHERE PRICE >= ALL (SELECT PRICE FROM test_table2)
    SELECT BUYERID FROM test_table1 UNION SELECT BUYERID FROM test_table2
    SELECT OWNERID, 'is in both Orders & Antiques' FROM test_table1 a, test_table2 b WHERE a.OWNERID = b.BUYERID and a.type in (?,?,?)
    SELECT DISTINCT SELLERID, OWNERLASTNAME, OWNERFIRSTNAME FROM test_table1, noconvert_table WHERE SELLERID = OWNERID ORDER BY OWNERLASTNAME, OWNERFIRSTNAME, OWNERID
    SELECT a.* FROM test_table1 a, noconvert_table b WHERE a.SELLERID = b.OWNERID 
    update test_table1 set col_1=123 ,col_2=?,col_3=? where col_4=?
    update test_table1 set col_1=?,col_2=col_2+1 where id in (?,?,?,?)
    delete from test_table2 where id in (?,?,?,?,?,?) and col_1 is not null
    INSERT INTO test_table1 VALUES (21, 01, 'Ottoman', ?,?)
    INSERT INTO test_table1 (BUYERID, SELLERID, ITEM) VALUES (01, 21, ?)

可能有些sql語句沒有出現在測試用例里，但是相信基本上常用的查詢sql shardbatis解析都沒有問題，因為shardbatis對sql的解析是基於jsqlparser的

另外需要注意的是：

• 2.0版本中insert update delete 語句中的子查詢語句中的表不支持sharding
• select語句中如果進行多表關聯，請務必為每個表名加上別名 例如原始sql語句：SELECT a. FROM ANTIQUES a,ANTIQUEOWNERS b, mytable c where a.id=b.id and b.id=c.id 經過轉換后的結果可能為：SELECT a. FROM ANTIQUES_0 AS a, ANTIQUEOWNERS_1 AS b, mytable_1 AS c WHERE a.id = b.id AND b.id = c.id