MyBatis(十)：Mybatis 幾種批量操作的對比

本章主要講解幾種批量處理的用法及對別，批量處理一般用法包含以下幾種：

1）普通foreach處理（沒循環一次執行一次與mysql服務器交互操作），實際上也是采用的ExecutorType.SIMPLE；

2）使用ExecutorType.BATCH批量處理方法；

3）拼接SQL，一次批量提交給Mysql多個插入語句到mysql服務器端，執行批量操作。

下邊針對這幾種方案分別進行示例展示用法，以及優缺點對比。

新建maven項目具體工作參考：《MyBatis(九)：Mybatis Java API批量操作（增、刪、改、查） 》
在spring-config.xml中添加sqlSessionTemplate bean:

   <bean id="sqlSessionTemplate" class="org.mybatis.spring.SqlSessionTemplate">
       <constructor-arg index="0" ref="sqlSessionFactory" />
       <!--<constructor-arg index="1" value="BATCH" />-->
   </bean>

新建測試類：com.dx.test.TestBatchInsert.java

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Random;

import org.apache.ibatis.session.ExecutorType;
import org.apache.ibatis.session.SqlSession;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mybatis.spring.SqlSessionTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.dx.test.mapper.ArticleCategoryMapper;
import com.dx.test.model.ArticleCategory;
import com.dx.test.model.enums.DataStatus;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({ "classpath:spring-config.xml" })
public class TestBatchInsert {
   @Autowired
   private SqlSessionTemplate sqlSessionTemplate; // 引入注入的sqlSessionTemplate bean實例
   @Autowired
   private ArticleCategoryMapper articleCategoryMapper; // 參考上篇文章該類的定義。

}

mybatis 批量處理測試：

方案1）采用mybatis foreach循環插入方案：

   @Test
   public void testWithSimple() {
      long start = System.currentTimeMillis();
      SqlSession sqlSession = sqlSessionTemplate.getSqlSessionFactory().openSession(ExecutorType.SIMPLE, false);
      ArticleCategoryMapper articleCategoryMapper=sqlSession.getMapper(ArticleCategoryMapper.class);
      
      List<ArticleCategory> list = new ArrayList<>();
      for (int i = 0; i < 10000; i++) {
         ArticleCategory articleCategory = new ArticleCategory();
         articleCategory.setTitle("title_" + new Random().nextInt(1000) + "_" + new Random().nextInt(1000));
         articleCategory.setDescription("category description");
         articleCategory.setImgSrc("http://www.test.com/img/category/img-" + new Random().nextInt(1000) + ".gif");
         articleCategory.setState(DataStatus.Living);
         articleCategory.setCreateTime(new Date());
         articleCategory.setCreateUser("create user");
         articleCategory.setCreateUserId("user-" + new Random().nextInt(1000));

         articleCategoryMapper.insert(articleCategory);
      }
      sqlSession.commit();
      sqlSession.clearCache();
      sqlSession.close();

      long stop = System.currentTimeMillis();
      System.out.println("testWithExecutorType.SIMPLE:" + (stop - start) + "ms");
   }

執行該測試方法：

==>  Preparing: INSERT INTO article_category (title, img_src, description, state, create_user, create_user_id, create_time, update_user, update_user_id, update_time, version) VALUES (?, ?, ?, ?, ?, ?, now(), ?, ?, now(), 0) 
==> Parameters: title_159_754(String), http://www.test.com/img/category/img-606.gif(String), category description(String), 0(Integer), create user(String), user-740(String), null, null
<==    Updates: 1
。。。
==>  Preparing: INSERT INTO article_category (title, img_src, description, state, create_user, create_user_id, create_time, update_user, update_user_id, update_time, version) VALUES (?, ?, ?, ?, ?, ?, now(), ?, ?, now(), 0) 
==> Parameters: title_416_561(String), http://www.test.com/img/category/img-72.gif(String), category description(String), 0(Integer), create user(String), user-98(String), null, null
<==    Updates: 1
==>  Preparing: INSERT INTO article_category (title, img_src, description, state, create_user, create_user_id, create_time, update_user, update_user_id, update_time, version) VALUES (?, ?, ?, ?, ?, ?, now(), ?, ?, now(), 0) 
==> Parameters: title_967_656(String), http://www.test.com/img/category/img-897.gif(String), category description(String), 0(Integer), create user(String), user-620(String), null, null
<==    Updates: 1
==>  Preparing: INSERT INTO article_category (title, img_src, description, state, create_user, create_user_id, create_time, update_user, update_user_id, update_time, version) VALUES (?, ?, ?, ?, ?, ?, now(), ?, ?, now(), 0) 
==> Parameters: title_399_676(String), http://www.test.com/img/category/img-819.gif(String), category description(String), 0(Integer), create user(String), user-889(String), null, null
<==    Updates: 1
testWithExecutorType.SIMPLE::8163ms

測試結果發現耗時為：8163ms，而且從打印日志，可以看出該類執行是一條一條的插入的，而且每次插入前都要Preparing插入sql，這個是比較耗時的。

方案2）采用mybatis batch方案（使用ExecutorType.BATCH）：

   @Test
   public void testWithBatch() {
      long start = System.currentTimeMillis();
      SqlSession sqlSession = sqlSessionTemplate.getSqlSessionFactory().openSession(ExecutorType.BATCH, false);
      ArticleCategoryMapper articleCategoryMapper=sqlSession.getMapper(ArticleCategoryMapper.class);
      List<ArticleCategory> list = new ArrayList<>();
      for (int i = 0; i < 10000; i++) {
         ArticleCategory articleCategory = new ArticleCategory();
         articleCategory.setTitle("title_" + new Random().nextInt(1000) + "_" + new Random().nextInt(1000));
         articleCategory.setDescription("category description");
         articleCategory.setImgSrc("http://www.test.com/img/category/img-" + new Random().nextInt(1000) + ".gif");
         articleCategory.setState(DataStatus.Living);
         articleCategory.setCreateTime(new Date());
         articleCategory.setCreateUser("create user");
         articleCategory.setCreateUserId("user-" + new Random().nextInt(1000));

         articleCategoryMapper.insert(articleCategory);
      }
      sqlSession.commit();
      sqlSession.clearCache();
      sqlSession.close();

      long stop = System.currentTimeMillis();
      System.out.println("testWithExecutorType.BATCH:" + (stop - start) + "ms");
   }

執行該測試方法：

==>  Preparing: INSERT INTO article_category (title, img_src, description, state, create_user, create_user_id, create_time, update_user, update_user_id, update_time, version) VALUES (?, ?, ?, ?, ?, ?, now(), ?, ?, now(), 0) 
==> Parameters: title_345_236(String), http://www.test.com/img/category/img-688.gif(String), category description(String), 0(Integer), create user(String), user-337(String), null, null
。。。
==> Parameters: title_534_211(String), http://www.test.com/img/category/img-572.gif(String), category description(String), 0(Integer), create user(String), user-139(String), null, null
==> Parameters: title_292_271(String), http://www.test.com/img/category/img-8.gif(String), category description(String), 0(Integer), create user(String), user-547(String), null, null
testWithExecutorType.BATCH:5283ms

測試結果發現耗時為：5283ms，而且從打印日志可以看出該方案只一致性一次預編譯，之后插入都不需要再次預編譯。

方案3）采用拼接SQL方案：

用法1：

   @Test
   public void testWithBatchScript() {
      long start = System.currentTimeMillis();
      
      List<ArticleCategory> list = new ArrayList<>();
      for (int i = 0; i < 10000; i++) {
         ArticleCategory articleCategory = new ArticleCategory();
         articleCategory.setTitle("title_" + new Random().nextInt(1000) + "_" + new Random().nextInt(1000));
         articleCategory.setDescription("category description");
         articleCategory.setImgSrc("http://www.test.com/img/category/img-" + new Random().nextInt(1000) + ".gif");
         articleCategory.setState(DataStatus.Living);
         articleCategory.setCreateTime(new Date());
         articleCategory.setCreateUser("create user");
         articleCategory.setCreateUserId("user-" + new Random().nextInt(1000));

         list.add(articleCategory);
      }
      articleCategoryMapper.batchInsertsWithScript(list);

      long stop = System.currentTimeMillis();
      System.out.println("testWithBatchScript:" + (stop - start) + "ms");
   }

該方法執行日志：

JDBC Connection [com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl@1b84f475] will not be managed by Spring
==>  Preparing: INSERT INTO `article_category` (`title`,`img_src`,`description`,`state`,`create_time`,`create_user`,`create_user_id`,`update_time`,`update_user`,`update_user_id`,`version`) 
VALUES (?,?,?,?,now(),?,?,now(),?,?,0) , 
。。。
(?,?,?,?,now(),?,?,now(),?,?,0) ,
(?,?,?,?,now(),?,?,now(),?,?,0) 
ON DUPLICATE KEY UPDATE `update_time` = now() 
==> Parameters: 
title_375_245(String), http://www.test.com/img/category/img-247.gif(String), category description(String), 0(Integer), create user(String), user-930(String), null, null, 
。。。 
title_275_39(String), http://www.test.com/img/category/img-875.gif(String), category description(String), 0(Integer), create user(String), user-323(String), null, null, 
title_743_735(String), http://www.test.com/img/category/img-220.gif(String), category description(String), 0(Integer), create user(String), user-917(String), null, null
<==    Updates: 10000
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@65b104b9]
testWithBatchScript:3312ms

測試2：

   @Test
   public void testWithBatchSQL() {
      long start = System.currentTimeMillis();

      List<ArticleCategory> list = new ArrayList<>();
      for (int i = 0; i < 10000; i++) {
         ArticleCategory articleCategory = new ArticleCategory();
         articleCategory.setTitle("title_" + new Random().nextInt(1000) + "_" + new Random().nextInt(1000));
         articleCategory.setDescription("category description");
         articleCategory.setImgSrc("http://www.test.com/img/category/img-" + new Random().nextInt(1000) + ".gif");
         articleCategory.setState(DataStatus.Living);
         articleCategory.setCreateTime(new Date());
         articleCategory.setCreateUser("create user");
         articleCategory.setCreateUserId("user-" + new Random().nextInt(1000));

         list.add(articleCategory);
      }
      articleCategoryMapper.batchInserts(list);
      long stop = System.currentTimeMillis();
      System.out.println("testWithBatchSQL:" + (stop - start) + "ms");
   }

執行該方法打印日志：

Creating a new SqlSession
SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@65b104b9] was not registered for synchronization because synchronization is not active
JDBC Connection [com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl@31ff43be] will not be managed by Spring
==>  Preparing: INSERT INTO `article_category`(`title`,`img_src`,`description`,`state`,`create_time`,`create_user`,`create_user_id`,`update_time`,`update_user`,`update_user_id`,`version`)
VALUES(?,?,?,?,now(),?,?,now(),?,?,0),
。。。
(?,?,?,?,now(),?,?,now(),?,?,0),
(?,?,?,?,now(),?,?,now(),?,?,0) 
ON DUPLICATE KEY UPDATE `update_time` = now(); 
==> Parameters: title_131_603(String), http://www.test.com/img/category/img-831.gif(String), category description(String), 0(Integer), create user(String), user-35(String), null, null, 
。。。
title_317_725(String), http://www.test.com/img/category/img-208.gif(String), category description(String), 0(Integer), create user(String), user-968(String), null, null, 
title_403_241(String), http://www.test.com/img/category/img-870.gif(String), category description(String), 0(Integer), create user(String), user-483(String), null, null
<==    Updates: 10000
Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@65b104b9]
testWithBatchSQL:2978ms

上邊這兩種方法實際上原理相同：拼接SQL，一次提交一批SQL方案。

拼接SQL一次提交方案缺點：當拼接sql超出’max_allowed_packet‘設置大小時，會拋出異常：

com.mysql.jdbc.PacketTooBigException: Packet for query is too large (8346602 > 4194304（4M）). You can change this value on the server by setting the max_allowed_packet’ variable.

備注：
SqlServer 對語句的條數和參數的數量都有限制，分別是 1000 和 2100。
Mysql 對語句的長度有限制，默認是 4M。
Mybatis 對動態語句沒有數量上的限制。

解決錯誤方案：

我安裝的mysql版本是：8.0.17 MySQL Community Server - GPL

dx:~ $ mysql -uroot -p123456 -h127.0.0.1
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 356
Server version: 8.0.17 MySQL Community Server - GPL

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

該版本的mysql允許的發送給服務器的最大數據包大小情況如下：

ysql> show variables like '%packet%';
+---------------------------+------------+
| Variable_name             | Value      |
+---------------------------+------------+
| max_allowed_packet        | 67108864   |
| mysqlx_max_allowed_packet | 67108864   |
| slave_max_allowed_packet  | 1073741824 |
+---------------------------+------------+
3 rows in set (0.00 sec)

ax_allowed_packet 設置以字節發送給服務器的最大數據包大小. (默認: 64MB)，
數據庫變更需要：大的插入和更新可能會被max_allowed_packet參數限制，導致失敗。
此時，解決方案：
1）在mysql中執行：set global max_allowed_paclet = 128*1024*1024；
2）修改mysql服務器的配置(my.ini 或者 my.cnf 文件)參數： max_allowed_packet = 128M 來解決，
3）需要重啟mysql服務器（缺陷）。

備注：

上邊‘mysqlx_max_allowed_packet’參數時mysqlx插件的配置參數，起作用針對mysqlx起作用，作用於 max_allowed_packet 相同。

mysql> show plugins;
+---------------------------------+----------+--------------------+---------+---------+
| Name                            | Status   | Type               | Library | License |
+---------------------------------+----------+--------------------+---------+---------+
| binlog                          | ACTIVE   | STORAGE ENGINE     | NULL    | GPL     |
| mysql_native_password           | ACTIVE   | AUTHENTICATION     | NULL    | GPL     |
| sha256_password                 | ACTIVE   | AUTHENTICATION     | NULL    | GPL     |
| caching_sha2_password           | ACTIVE   | AUTHENTICATION     | NULL    | GPL     |
| sha2_cache_cleaner              | ACTIVE   | AUDIT              | NULL    | GPL     |
| CSV                             | ACTIVE   | STORAGE ENGINE     | NULL    | GPL     |
| MEMORY                          | ACTIVE   | STORAGE ENGINE     | NULL    | GPL     |
| InnoDB                          | ACTIVE   | STORAGE ENGINE     | NULL    | GPL     |
...
| mysqlx                          | ACTIVE   | DAEMON             | NULL    | GPL     |
| mysqlx_cache_cleaner            | ACTIVE   | AUDIT              | NULL    | GPL     |
+---------------------------------+----------+--------------------+---------+---------+
44 rows in set (0.01 sec)

性能總結：

+----------------------------------------------+------------+
| 采用方案                                      | 耗時        |
+----------------------------------------------+------------+
| 普通處理方法(ExecutorType.SIMPLE)逐條插入處理    | 8s         |
| 使用ExecutorType.BATCH                        | 5s         |
| 使用拼接SQL,一致性提交一批SQL                    | 3s         |
+-----------------------------------------------+------------+