JDBC批量插入遇到的坑

 首先看代碼，這段代碼是測試插入多條數據耗時。

 測試環境：jdk1.8，mysql-5.7.27-winx64，mysql-connector-java-5.1.10.jar

public class Test
{
    /**
     * rewriteBatchedStatements=true    開啟批量更新
     */
    private static final String URL = "jdbc:mysql://localhost:3306/test?characterEncoding=utf-8";
    private static final String USERNAME = "root";
    private static final String PASSWORD = "root";
    private static Connection connection;
    private static PreparedStatement pstm;

    static
    {
        try
        {
            Class.forName("com.mysql.jdbc.Driver");
            connection = DriverManager.getConnection(URL, USERNAME, PASSWORD);
        } catch (ClassNotFoundException e)
        {
            e.printStackTrace();
        } catch (SQLException e)
        {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) throws SQLException
    {
        deleteAll();
        update();
        deleteAll();
        addBatch();
        connection.close();
    }
    
    public static void update() throws SQLException
    {
        long startTime = System.currentTimeMillis();
        String sql = "insert into `user` (`id`,`name` ) values (?, ?);";
        pstm = connection.prepareStatement(sql);
        for (int i = 1; i <= 10000; i++)
        {
            pstm.setInt(1, i);
            pstm.setString(2, "test" + i);
            pstm.executeUpdate();
        }
        pstm.close();
        long endTime = System.currentTimeMillis();
        System.out.println("update方法耗時: " + (endTime - startTime));
    }


    public static void addBatch() throws SQLException
    {
        long startTime = System.currentTimeMillis();
        String sql = "insert into `user` (`id`,`name` ) values (?, ?);";
        pstm = connection.prepareStatement(sql);
        for (int i = 1; i <= 10000; i++)
        {
            pstm.setInt(1, i);
            pstm.setString(2, "test" + i);
            pstm.addBatch();
        }
        pstm.executeBatch();
        pstm.close();
        long endTime = System.currentTimeMillis();
        System.out.println("addBatch方法耗時: " + (endTime - startTime));
    }

    /**
     * 刪除user表中的所有數據
     */
    public static void deleteAll() throws SQLException
    {
        String sql = "delete from `user` where `id` > 0;";
        pstm = connection.prepareStatement(sql);
        pstm.executeUpdate();
        pstm.close();
    }
}

代碼執行結果： 

update方法耗時: 19291
addBatch方法耗時: 18971

可以看到區別不大。原因是：

MySQL JDBC驅動在默認情況下會無視executeBatch()語句，把我們期望批量執行的一組sql語句拆散，一條一條地發給MySQL數據庫，批量插入實際上是單條插入，直接造成較低的性能。只有把rewriteBatchedStatements參數置為true, 驅動才會幫你批量執行SQL

修改URL為:

private static final String URL = "jdbc:mysql://localhost:3306/test?characterEncoding=utf-8&rewriteBatchedStatements=true";

然后執行會出現如下結果:

update方法耗時: 20696
Exception in thread "main" java.sql.BatchUpdateException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(2, 'test2'),(3, 'test3'),(4, 'test4'),(5, 'test5'),(6, 'test6'),(7, 'test7'),(8' at line 1

查看錯誤信息提示SQL語法有問題，但是沒有加rewriteBatchedStatements=true參數時代碼可以正常運行。

debug調試源碼發現加了rewriteBatchedStatements=true會執行 this.executeBatchedInserts(batchTimeout);

然后判斷Mysql版本是否高於4.1.0並且數據大於3條執行 this.executePreparedBatchAsMultiStatement(batchTimeout)；

否則執行 this.executeBatchSerially(batchTimeout);

接下來查看executeBatchedInserts方法:

protected int[] executeBatchedInserts(int batchTimeout) throws SQLException {
        String valuesClause = this.getValuesClause();
        Connection locallyScopedConn = this.connection;
        if (valuesClause == null) {
            return this.executeBatchSerially(batchTimeout);
        } else {
            int numBatchedArgs = this.batchedArgs.size();
            if (this.retrieveGeneratedKeys) {
                this.batchedGeneratedKeys = new ArrayList(numBatchedArgs);
            }

            int numValuesPerBatch = this.computeBatchSize(numBatchedArgs);
            if (numBatchedArgs < numValuesPerBatch) {
                numValuesPerBatch = numBatchedArgs;
            }

            java.sql.PreparedStatement batchedStatement = null;
            int batchedParamIndex = 1;
            int updateCountRunningTotal = 0;
            int numberToExecuteAsMultiValue = false;
            int batchCounter = 0;
            CancelTask timeoutTask = null;
            SQLException sqlEx = null;
            int[] updateCounts = new int[numBatchedArgs];

            for(int i = 0; i < this.batchedArgs.size(); ++i) {
                updateCounts[i] = 1;
            }

            Object ex;
            try {
                try {
                    batchedStatement = this.prepareBatchedInsertSQL((ConnectionImpl)locallyScopedConn, numValuesPerBatch);
                    if (this.connection.getEnableQueryTimeouts() && batchTimeout != 0 && this.connection.versionMeetsMinimum(5, 0, 0)) {
                        timeoutTask = new CancelTask(this, (StatementImpl)batchedStatement);
                        ConnectionImpl.getCancelTimer().schedule(timeoutTask, (long)batchTimeout);
                    }

                    int numberToExecuteAsMultiValue;
                    if (numBatchedArgs < numValuesPerBatch) {
                        numberToExecuteAsMultiValue = numBatchedArgs;
                    } else {
                        numberToExecuteAsMultiValue = numBatchedArgs / numValuesPerBatch;
                    }

                    ex = numberToExecuteAsMultiValue * numValuesPerBatch;

                    for(int i = 0; i < ex; ++i) {
                        if (i != 0 && i % numValuesPerBatch == 0) {
                            try {
                                updateCountRunningTotal += batchedStatement.executeUpdate();
                            } catch (SQLException var41) {
                                sqlEx = this.handleExceptionForBatch(batchCounter - 1, numValuesPerBatch, updateCounts, var41);
                            }

                            this.getBatchedGeneratedKeys(batchedStatement);
                            batchedStatement.clearParameters();
                            batchedParamIndex = 1;
                        }

                        batchedParamIndex = this.setOneBatchedParameterSet(batchedStatement, batchedParamIndex, this.batchedArgs.get(batchCounter++));
                    }

                    try {
                        updateCountRunningTotal += batchedStatement.executeUpdate();
                    } catch (SQLException var40) {
                        sqlEx = this.handleExceptionForBatch(batchCounter - 1, numValuesPerBatch, updateCounts, var40);
                    }

                    this.getBatchedGeneratedKeys(batchedStatement);
                    numValuesPerBatch = numBatchedArgs - batchCounter;
                } finally {
                    if (batchedStatement != null) {
                        batchedStatement.close();
                    }

                }

                try {
                    if (numValuesPerBatch > 0) {
                        batchedStatement = this.prepareBatchedInsertSQL((ConnectionImpl)locallyScopedConn, numValuesPerBatch);
                        if (timeoutTask != null) {
                            timeoutTask.toCancel = (StatementImpl)batchedStatement;
                        }

                        batchedParamIndex = 1;

                        while(true) {
                            if (batchCounter >= numBatchedArgs) {
                                try {
                                    int var10000 = updateCountRunningTotal + batchedStatement.executeUpdate();
                                } catch (SQLException var39) {
                                    ex = var39;
                                    sqlEx = this.handleExceptionForBatch(batchCounter - 1, numValuesPerBatch, updateCounts, var39);
                                }

                                this.getBatchedGeneratedKeys(batchedStatement);
                                break;
                            }

                            batchedParamIndex = this.setOneBatchedParameterSet(batchedStatement, batchedParamIndex, this.batchedArgs.get(batchCounter++));
                        }
                    }

                    if (sqlEx != null) {
                        throw new BatchUpdateException(sqlEx.getMessage(), sqlEx.getSQLState(), sqlEx.getErrorCode(), updateCounts);
                    }

                    ex = updateCounts;
                } finally {
                    if (batchedStatement != null) {
                        batchedStatement.close();
                    }

                }
            } finally {
                if (timeoutTask != null) {
                    timeoutTask.cancel();
                }

                this.resetCancelledState();
            }

            return (int[])ex;
        }
    }

這句代碼生成了一個sql語句模板：

batchedStatement = this.prepareBatchedInsertSQL((ConnectionImpl)locallyScopedConn, numValuesPerBatch);

insert into `user` (`id`,`name` ) values (** NOT SPECIFIED **, ** NOT SPECIFIED **);,(** NOT SPECIFIED **, ** NOT SPECIFIED **)...

然后這段代碼是給模板逐個賦值：

 batchedParamIndex = this.setOneBatchedParameterSet(batchedStatement, batchedParamIndex, this.batchedArgs.get(batchCounter++));

 現在問題已經找到了，在生成模板的時候多了個分號。是前面寫的sql語句的，所以將SQL語句最后的分號去掉然后執行就沒問題了。

 分號去掉后的運行結果：

update方法耗時: 20934
addBatch方法耗時: 106

文章到此結束，如有錯誤敬請指正。