批量插入之SqlBulkCopy
作者：NewcatsHuang
時間：2021-12-25
完整代碼：Github傳送門
一.目錄
二.方法介紹
1.for循環插入
對集合數據進行遍歷，每次只插入集合的一條數據，對應的SQL語句為：
insert into UserInfo(Id,Name) values (@Id,@Name);
2.拼接sql
也需要for循環遍歷，只是一條語句能插入多個數據，對應的SQL語句為：
insert into UserInfo(Id,Name) values (@Id1,@Name1),(@Id2,@Name2)...
3.SqlBulkCopy批量插入
利用各個數據庫的特性，直接從文件復制到表
三.SqlBulkCopy介紹
1.SqlServer數據庫
-
使用 Microsoft.Data.SqlClient.SqlBulkCopy 類封裝的方法
-
BULK INSERT Northwind.dbo.[OrderDetails] FROM 'f:\mydata\data.tbl' WITH (FORMATFILE='f:\mydata\data.fmt');
Microsoft SQL Server 包含一個名為 bcp 的受歡迎的命令行實用工具，以便將較大文件快速大容量復制到 SQL Server 數據庫的表或視圖中 。 SqlBulkCopy 類允許你編寫可提供類似功能的托管代碼解決方案。 還可通過其他方法將數據加載到 SQL Server 表（例如 INSERT 語句），但 SqlBulkCopy 可提供顯著的性能優勢。SqlBulkCopy 類可用於只將數據寫入 SQLServer 表。但是， 數據源不限於 SQL Server；可以使用任何數據源，只要數據可以加載到 DataTable 實例或使用IDataReader 實例讀取即可。
使用 SqlBulkCopy 類，你可以執行以下操作：
單次大容量復制操作
多次大容量復制操作
事務中的大容量復制操作
以上內容來自微軟官方文檔： Bulk Copy Operations in SQL Server
2.MySql數據庫
-
使用 MySqlConnector.MySqlBulkCopy 類封裝的方法
-
LOAD DATA LOCAL INFILE '/tmp/test.txt' INTO TABLE test;
LOAD DATA語句以非常高的速度將文本文件中的行讀取到表中。可以從服務器主機或客戶端主機讀取文件，具體取決於是否給出了修飾符。還會影響數據解釋和錯誤處理。
以上內容來自MySql官網：LOAD DATA Statement
3.PostgreSql數據庫
- COPY public.userinfo (id,name,createtime) FROM STDIN (FORMAT BINARY);
COPY moves data between PostgreSQL tables and standard file-system files. COPY TO copies the contents of a table to a file, while COPY FROM copies data from a file to a table (appending the data to whatever is in the table already). COPY TO can also copy the results of a SELECT query.
If a column list is specified, COPY TO copies only the data in the specified columns to the file. For COPY FROM, each field in the file is inserted, in order, into the specified column. Table columns not specified in the COPY FROM column list will receive their default values.
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server. When PROGRAM is specified, the server executes the given command and reads from the standard output of the program, or writes to the standard input of the program. The command must be specified from the viewpoint of the server, and be executable by the PostgreSQL user. When STDIN or STDOUT is specified, data is transmitted via the connection between the client and the server.
Each backend running COPY will report its progress in the pg_stat_progress_copy view. See Section 28.4.6 for details.
COPY在 PostgreSQL表和標准文件系統文件之間移動數據。 將表的內容復制到文件，同時將數據從文件復制到表（將數據追加到表中已有的任何內容）。 還可以復制查詢的結果。COPY TOCOPY FROMCOPY TOSELECT
如果指定了列列表，則 僅將指定列中的數據復制到文件中。對於 ，文件中的每個字段將按順序插入到指定的列中。列列表中未指定的表列將收到其默認值。COPY TOCOPY FROMCOPY FROM
COPY與文件名指示PostgreSQL服務器直接讀取或寫入文件。該文件必須可由PostgreSQL用戶（服務器運行時的用戶 ID）訪問，並且必須從服務器的角度指定名稱。當指定時，服務器執行給定的命令並從程序的標准輸出中讀取，或寫入程序的標准輸入。該命令必須從服務器的角度指定，並且可由PostgreSQL用戶執行。指定 或 時，數據通過客戶端和服務器之間的連接傳輸。PROGRAMSTDINSTDOUT
每個正在運行的后端都將在視圖中報告其進度。有關詳細信息，請參見第 28.4.6 節。COPYpg_stat_progress_copy
以上內容來自PostgreSql官網：sql copy
四.性能測試
1.環境
| 數據庫 | 版本 | OS | CPU | RAM | 說明 |
|---|---|---|---|---|---|
| SqlServer | v2017 | Win10 21H2 | i7-9700k | 64GB | VMWare宿主機 |
| MySql | v8.0.27 | Ubuntu Server 21.10 | i7-9700k 2core | 4GB | VMWare虛擬機1 |
| PostgreSql | v13.5 | Ubuntu Server 21.10 | i7-9700k 2core | 4GB | VMWare虛擬機2 |
2.代碼
2.1 for循環-dapper執行foreach循環插入數據
SqlServer版本
/// <summary>
/// dapper執行foreach循環插入數據
/// </summary>
internal long InsertForEach(List<NewcatsUserInfoTest> list)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
using (SqlConnection conn = new SqlConnection(ConnectionString))
{
foreach (NewcatsUserInfoTest test in list)
{
if (conn.State == ConnectionState.Closed)
conn.Open();
result += conn.Execute(SqlText, test, commandType: System.Data.CommandType.Text);
}
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.2 for循環-ADO.NET的foreach循環插入數據
MySql版本
/// <summary>
/// ADO.NET的foreach循環插入數據
/// </summary>
internal long InsertForEachNative(List<NewcatsUserInfoTest> list)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
using (MySqlConnection conn = new MySqlConnection(ConnectionString))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
foreach (NewcatsUserInfoTest item in list)
{
using (MySqlCommand cmd = new MySqlCommand())
{
string sqlText = $"INSERT INTO {TableName} (Id,Name,CreateTime) VALUES (@Id{result},@Name{result},@CreateTime{result})";
cmd.Connection = conn;
cmd.CommandText = sqlText;
cmd.Parameters.Add(new MySqlParameter("@Id" + result.ToString(), item.Id));
cmd.Parameters.Add(new MySqlParameter("@Name" + result.ToString(), item.Name));
cmd.Parameters.Add(new MySqlParameter("@CreateTime" + result.ToString(), item.CreateTime));
result += cmd.ExecuteNonQuery();
}
}
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.3 for循環-dapper直接傳list參數
PostgreSql版本
/// <summary>
/// dapper直接傳list參數
/// </summary>
internal long InsertBulk(List<NewcatsUserInfoTest> list)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
using (NpgsqlConnection conn = new NpgsqlConnection(ConnectionString))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
result = conn.Execute(SqlText, list, commandType: System.Data.CommandType.Text);
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.4 for循環-dapper拼接sql語句
SqlServer版本
/// <summary>
/// dapper拼接sql語句
/// </summary>
internal long InsertAppend(List<NewcatsUserInfoTest> list)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
const int perCount = 500;
int times = Convert.ToInt32(Math.Ceiling(list.Count * 1.0 / perCount));
for (int i = 0; i < times; i++)
{
using (SqlConnection conn = new SqlConnection(ConnectionString))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
StringBuilder sb = new StringBuilder($"INSERT INTO {TableName} (Id,Name,CreateTime) VALUES");
var perList = list.Skip(i * perCount).Take(perCount);
int index = 0;
DynamicParameters dp = new DynamicParameters();
foreach (NewcatsUserInfoTest test in perList)
{
sb.Append($"(@Id{index},@Name{index},@CreateTime{index}),");
dp.Add($"@Id{index}", test.Id);
dp.Add($"@Name{index}", test.Name);
dp.Add($"@CreateTime{index}", test.CreateTime);
index++;
}
result += conn.Execute(sb.ToString().TrimEnd(','), dp, commandType: System.Data.CommandType.Text);
}
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.5 for循環-SqlBulkCopy插入-FromList
MySql版本
/// <summary>
/// SqlBulkCopy插入-FromList
/// </summary>
internal long SqlBulkCopyFromList(List<NewcatsUserInfoTest> list)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
using (MySqlConnection conn = new MySqlConnection(ConnectionString))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
MySqlBulkCopy copy = new MySqlBulkCopy(conn);
copy.DestinationTableName = TableName;
var r = copy.WriteToServer(list.ToDataTable());
result = r.RowsInserted;
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.6 for循環-SqlBulkCopy插入-FromDataTable
PostgreSql版本
/// <summary>
/// SqlBulkCopy插入-FromDataTable
/// </summary>
internal long SqlBulkCopyFromDataTable(DataTable dt)
{
Stopwatch sw = new Stopwatch();
sw.Start();
int result = 0;
using (NpgsqlConnection conn = new NpgsqlConnection(ConnectionString))
{
if (conn.State == ConnectionState.Closed)
conn.Open();
using (NpgSqlBulkCopy copy = new NpgSqlBulkCopy(conn, TableName))
{
copy.WriteToServer(dt);
}
}
sw.Stop();
return sw.ElapsedMilliseconds;
}
2.7 Benchmark測試(SqlServer示例)
/// <summary>
/// SqlBulkCopy測試
/// </summary>
public class BulkCopyContext
{
const int totalCount = 5000;
#region SqlServer
[Benchmark]
public void SqlServer_InsertForEach()
{
List<NewcatsUserInfoTest> list = new List<NewcatsUserInfoTest>();
for (int i = 0; i < totalCount; i++)
{
NewcatsUserInfoTest u = new NewcatsUserInfoTest()
{
Id = IdHelper.Create(),
Name = EncryptHelper.GetRandomString(Random.Shared.Next(20)),
CreateTime = DateTime.Now
};
list.Add(u);
}
SqlServerTest test = new SqlServerTest();
test.Init();
test.InsertForEach(list);
}
[Benchmark]
public void SqlServer_InsertAppend()
{
List<NewcatsUserInfoTest> list = new List<NewcatsUserInfoTest>();
for (int i = 0; i < totalCount; i++)
{
NewcatsUserInfoTest u = new NewcatsUserInfoTest()
{
Id = IdHelper.Create(),
Name = EncryptHelper.GetRandomString(Random.Shared.Next(20)),
CreateTime = DateTime.Now
};
list.Add(u);
}
SqlServerTest test = new SqlServerTest();
test.Init();
test.InsertAppend(list);
}
[Benchmark]
public void SqlServer_SqlBulkCopy_FromList()
{
List<NewcatsUserInfoTest> list = new List<NewcatsUserInfoTest>();
for (int i = 0; i < totalCount; i++)
{
NewcatsUserInfoTest u = new NewcatsUserInfoTest()
{
Id = IdHelper.Create(),
Name = EncryptHelper.GetRandomString(Random.Shared.Next(20)),
CreateTime = DateTime.Now
};
list.Add(u);
}
SqlServerTest test = new SqlServerTest();
test.Init();
test.SqlBulkCopyFromList(list);
}
[Benchmark]
public void SqlServer_SqlBulkCopy_FromDataTable()
{
DataTable dt = new DataTable("NewcatsUserInfoTest");
dt.Columns.Add("Id", typeof(long));
dt.Columns.Add("Name", typeof(string));
dt.Columns.Add("CreateTime", typeof(DateTime));
for (int i = 0; i < totalCount; i++)
{
var id = IdHelper.Create();
var name = EncryptHelper.GetRandomString(Random.Shared.Next(20));
var now = DateTime.Now;
dt.Rows.Add(id, name, now);
}
SqlServerTest test = new SqlServerTest();
test.Init();
test.SqlBulkCopyFromDataTable(dt);
}
#endregion
}
3.for循環測試結果
| Database/Method | Counts | InsertForEach(ms) | InsertForEachNative(ms) | InsertBulk(ms) | InsertAppend(ms) | SqlBulkCopyFromList(ms) | SqlBulkCopyFromDataTable(ms) |
|---|---|---|---|---|---|---|---|
| SqlServer | 1 | 35 | 1 | 6 | 27 | 2 | |
| MySql | 1 | 26 | 1 | 1 | 1 | 19 | 2 |
| PostgreSql | 1 | 10 | 2 | 3 | 42 | 1 | |
| SqlServer | 10 | 42 | 5 | 1 | 6 | 25 | 1 |
| MySql | 10 | 35 | 6 | 6 | 1 | 20 | 2 |
| PostgreSql | 10 | 14 | 6 | 3 | 3 | 42 | 1 |
| SqlServer | 100 | 55 | 48 | 13 | 13 | 27 | 1 |
| MySql | 100 | 88 | 54 | 53 | 3 | 20 | 1 |
| PostgreSql | 100 | 37 | 52 | 68 | 4 | 43 | 2 |
| SqlServer | 1000 | 185 | 475 | 122 | 256 | 29 | 3 |
| MySql | 1000 | 630 | 582 | 560 | 19 | 25 | 7 |
| PostgreSql | 1000 | 549 | 474 | 521 | 14 | 45 | 4 |
| SqlServer(1w) | 10000 | 1330 | 4506 | 1216 | 2041 | 50 | 16 |
| MySql | 10000 | 6033 | 6815 | 6461 | 113 | 75 | 45 |
| PostgreSql | 10000 | 5310 | 5254 | 5131 | 80 | 72 | 26 |
| SqlServer(10w) | 100000 | 12500 | 43793 | 12065 | 19543 | 264 | 119 |
| MySql | 100000 | 61492 | 60210 | 61099 | 1201 | 534 | 394 |
| PostgreSql | 100000 | 53870 | 53407 | 55238 | 749 | 345 | 208 |
| SqlServer(100w) | 1000000 | 128015 | 452346 | 122330 | 194935 | 2253 | 1123 |
| MySql | 1000000 | 4766 | 3873 | ||||
| PostgreSql | 1000000 | 2657 | 1351 | ||||
| SqlServer(1000w) | 10000000 | 25721 | 15312 | ||||
| MySql | 10000000 | 115881 | 49502 | 31687 | |||
| PostgreSql | 10000000 | 71680 | 25139 | 13041 |
展開查看詳細的for循環測試結果
---
SqlServer測試結果如下：
集合大小:1
1.InsertForEach方法耗時:35ms
2.InsertForEachNative方法耗時:1ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:6ms
5.SqlBulkCopyFromList方法耗時:27ms
6.SqlBulkCopyFromDataTable方法耗時:2ms
MySql測試結果如下：
集合大小:1
1.InsertForEach方法耗時:26ms
2.InsertForEachNative方法耗時:1ms
3.InsertBulk方法耗時:1ms
4.InsertAppend方法耗時:1ms
5.SqlBulkCopyFromList方法耗時:19ms
6.SqlBulkCopyFromDataTable方法耗時:2ms
PostgreSql測試結果如下：
集合大小:1
1.InsertForEach方法耗時:10ms
2.InsertForEachNative方法耗時:2ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:3ms
5.SqlBulkCopyFromList方法耗時:42ms
6.SqlBulkCopyFromDataTable方法耗時:1ms
---
SqlServer測試結果如下：
集合大小:10
1.InsertForEach方法耗時:42ms
2.InsertForEachNative方法耗時:5ms
3.InsertBulk方法耗時:1ms
4.InsertAppend方法耗時:6ms
5.SqlBulkCopyFromList方法耗時:25ms
6.SqlBulkCopyFromDataTable方法耗時:1ms
MySql測試結果如下：
集合大小:10
1.InsertForEach方法耗時:35ms
2.InsertForEachNative方法耗時:6ms
3.InsertBulk方法耗時:6ms
4.InsertAppend方法耗時:1ms
5.SqlBulkCopyFromList方法耗時:20ms
6.SqlBulkCopyFromDataTable方法耗時:2ms
PostgreSql測試結果如下：
集合大小:10
1.InsertForEach方法耗時:14ms
2.InsertForEachNative方法耗時:6ms
3.InsertBulk方法耗時:3ms
4.InsertAppend方法耗時:3ms
5.SqlBulkCopyFromList方法耗時:42ms
6.SqlBulkCopyFromDataTable方法耗時:1ms
---
SqlServer測試結果如下：
集合大小:100
1.InsertForEach方法耗時:55ms
2.InsertForEachNative方法耗時:48ms
3.InsertBulk方法耗時:13ms
4.InsertAppend方法耗時:13ms
5.SqlBulkCopyFromList方法耗時:27ms
6.SqlBulkCopyFromDataTable方法耗時:1ms
MySql測試結果如下：
集合大小:100
1.InsertForEach方法耗時:88ms
2.InsertForEachNative方法耗時:54ms
3.InsertBulk方法耗時:53ms
4.InsertAppend方法耗時:3ms
5.SqlBulkCopyFromList方法耗時:20ms
6.SqlBulkCopyFromDataTable方法耗時:1ms
PostgreSql測試結果如下：
集合大小:100
1.InsertForEach方法耗時:37ms
2.InsertForEachNative方法耗時:52ms
3.InsertBulk方法耗時:68ms
4.InsertAppend方法耗時:4ms
5.SqlBulkCopyFromList方法耗時:43ms
6.SqlBulkCopyFromDataTable方法耗時:2ms
---
SqlServer測試結果如下：
集合大小:1000
1.InsertForEach方法耗時:185ms
2.InsertForEachNative方法耗時:475ms
3.InsertBulk方法耗時:122ms
4.InsertAppend方法耗時:256ms
5.SqlBulkCopyFromList方法耗時:29ms
6.SqlBulkCopyFromDataTable方法耗時:3ms
MySql測試結果如下：
集合大小:1000
1.InsertForEach方法耗時:630ms
2.InsertForEachNative方法耗時:582ms
3.InsertBulk方法耗時:560ms
4.InsertAppend方法耗時:19ms
5.SqlBulkCopyFromList方法耗時:25ms
6.SqlBulkCopyFromDataTable方法耗時:7ms
PostgreSql測試結果如下：
集合大小:1000
1.InsertForEach方法耗時:549ms
2.InsertForEachNative方法耗時:474ms
3.InsertBulk方法耗時:521ms
4.InsertAppend方法耗時:14ms
5.SqlBulkCopyFromList方法耗時:45ms
6.SqlBulkCopyFromDataTable方法耗時:4ms
---
SqlServer測試結果如下：
集合大小:10000
1.InsertForEach方法耗時:1330ms
2.InsertForEachNative方法耗時:4506ms
3.InsertBulk方法耗時:1216ms
4.InsertAppend方法耗時:2041ms
5.SqlBulkCopyFromList方法耗時:50ms
6.SqlBulkCopyFromDataTable方法耗時:16ms
MySql測試結果如下：
集合大小:10000
1.InsertForEach方法耗時:6033ms
2.InsertForEachNative方法耗時:6815ms
3.InsertBulk方法耗時:6461ms
4.InsertAppend方法耗時:113ms
5.SqlBulkCopyFromList方法耗時:75ms
6.SqlBulkCopyFromDataTable方法耗時:45ms
PostgreSql測試結果如下：
集合大小:10000
1.InsertForEach方法耗時:5310ms
2.InsertForEachNative方法耗時:5254ms
3.InsertBulk方法耗時:5131ms
4.InsertAppend方法耗時:80ms
5.SqlBulkCopyFromList方法耗時:72ms
6.SqlBulkCopyFromDataTable方法耗時:26ms
---
SqlServer測試結果如下：
集合大小:100000
1.InsertForEach方法耗時:12500ms
2.InsertForEachNative方法耗時:43793ms
3.InsertBulk方法耗時:12065ms
4.InsertAppend方法耗時:19543ms
5.SqlBulkCopyFromList方法耗時:264ms
6.SqlBulkCopyFromDataTable方法耗時:119ms
MySql測試結果如下：
集合大小:100000
1.InsertForEach方法耗時:61492ms
2.InsertForEachNative方法耗時:60210ms
3.InsertBulk方法耗時:61099ms
4.InsertAppend方法耗時:1201ms
5.SqlBulkCopyFromList方法耗時:534ms
6.SqlBulkCopyFromDataTable方法耗時:394ms
PostgreSql測試結果如下：
集合大小:100000
1.InsertForEach方法耗時:53870ms
2.InsertForEachNative方法耗時:53407ms
3.InsertBulk方法耗時:55238ms
4.InsertAppend方法耗時:749ms
5.SqlBulkCopyFromList方法耗時:345ms
6.SqlBulkCopyFromDataTable方法耗時:208ms
---
SqlServer測試結果如下：
集合大小:1000000
1.InsertForEach方法耗時:128015ms
2.InsertForEachNative方法耗時:452346ms
3.InsertBulk方法耗時:122330ms
4.InsertAppend方法耗時:194935ms
5.SqlBulkCopyFromList方法耗時:2253ms
6.SqlBulkCopyFromDataTable方法耗時:1123ms
MySql測試結果如下：
集合大小:1000000
1.InsertForEach方法耗時:0ms
2.InsertForEachNative方法耗時:0ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:11982ms
5.SqlBulkCopyFromList方法耗時:4766ms
6.SqlBulkCopyFromDataTable方法耗時:3873ms
PostgreSql測試結果如下：
集合大小:1000000
1.InsertForEach方法耗時:0ms
2.InsertForEachNative方法耗時:0ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:7101ms
5.SqlBulkCopyFromList方法耗時:2657ms
6.SqlBulkCopyFromDataTable方法耗時:1351ms
---
SqlServer測試結果如下：
集合大小:10000000
1.InsertForEach方法耗時:0ms
2.InsertForEachNative方法耗時:0ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:0ms
5.SqlBulkCopyFromList方法耗時:25721ms
6.SqlBulkCopyFromDataTable方法耗時:15312ms
MySql測試結果如下：
集合大小:10000000
1.InsertForEach方法耗時:0ms
2.InsertForEachNative方法耗時:0ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:115881ms
5.SqlBulkCopyFromList方法耗時:49502ms
6.SqlBulkCopyFromDataTable方法耗時:31687ms
PostgreSql測試結果如下：
集合大小:10000000
1.InsertForEach方法耗時:0ms
2.InsertForEachNative方法耗時:0ms
3.InsertBulk方法耗時:0ms
4.InsertAppend方法耗時:71680ms
5.SqlBulkCopyFromList方法耗時:25139ms
6.SqlBulkCopyFromDataTable方法耗時:13041ms
---
4.Benchmark測試結果
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19044.1415 (21H2)
Intel Core i7-9700K CPU 3.60GHz (Coffee Lake), 1 CPU, 8 logical and 8 physical cores
.NET SDK=6.0.101
[Host] : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT
DefaultJob : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT
totalCount=1
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 1.451 ms | 0.0046 ms | 0.0043 ms |
| SqlServer_InsertAppend | 1.450 ms | 0.0074 ms | 0.0066 ms |
| SqlServer_SqlBulkCopy_FromList | 2.060 ms | 0.0322 ms | 0.0285 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 2.054 ms | 0.0407 ms | 0.0381 ms |
| MySql_InsertForEach | 7.352 ms | 0.1453 ms | 0.3033 ms |
| MySql_InsertAppend | 7.553 ms | 0.1501 ms | 0.2338 ms |
| MySql_SqlBulkCopy_FromList | 7.875 ms | 0.1539 ms | 0.2735 ms |
| MySql_SqlBulkCopy_FromDataTable | 7.969 ms | 0.1549 ms | 0.1722 ms |
| PostgreSql_InsertForEach | 2.091 ms | 0.0416 ms | 0.1082 ms |
| PostgreSql_InsertAppend | 2.095 ms | 0.0416 ms | 0.1005 ms |
| PostgreSql_SqlBulkCopy_FromList | 4.061 ms | 0.0804 ms | 0.1449 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 4.097 ms | 0.0808 ms | 0.1436 ms |
totalCount=10
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 2.476 ms | 0.0091 ms | 0.0085 ms |
| SqlServer_InsertAppend | 1.776 ms | 0.0129 ms | 0.0121 ms |
| SqlServer_SqlBulkCopy_FromList | 2.079 ms | 0.0175 ms | 0.0146 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 2.081 ms | 0.0264 ms | 0.0220 ms |
| MySql_InsertForEach | 13.359 ms | 0.2609 ms | 0.4212 ms |
| MySql_InsertAppend | 7.787 ms | 0.1556 ms | 0.3350 ms |
| MySql_SqlBulkCopy_FromList | 8.336 ms | 0.1659 ms | 0.3157 ms |
| MySql_SqlBulkCopy_FromDataTable | 8.437 ms | 0.1682 ms | 0.2946 ms |
| PostgreSql_InsertForEach | 7.144 ms | 0.1386 ms | 0.1898 ms |
| PostgreSql_InsertAppend | 2.115 ms | 0.0418 ms | 0.0775 ms |
| PostgreSql_SqlBulkCopy_FromList | 4.020 ms | 0.0799 ms | 0.1684 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 4.082 ms | 0.0772 ms | 0.1373 ms |
totalCount=100
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 13.920 ms | 0.1864 ms | 0.1652 ms |
| SqlServer_InsertAppend | 8.570 ms | 0.0443 ms | 0.0393 ms |
| SqlServer_SqlBulkCopy_FromList | 2.432 ms | 0.0477 ms | 0.0567 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 2.339 ms | 0.0361 ms | 0.0338 ms |
| MySql_InsertForEach | 63.129 ms | 1.2535 ms | 2.6983 ms |
| MySql_InsertAppend | 8.332 ms | 0.1603 ms | 0.1646 ms |
| MySql_SqlBulkCopy_FromList | 8.549 ms | 0.1457 ms | 0.2589 ms |
| MySql_SqlBulkCopy_FromDataTable | 8.567 ms | 0.1697 ms | 0.2434 ms |
| PostgreSql_InsertForEach | 55.740 ms | 1.1029 ms | 2.3503 ms |
| PostgreSql_InsertAppend | 2.853 ms | 0.0550 ms | 0.0752 ms |
| PostgreSql_SqlBulkCopy_FromList | 4.393 ms | 0.0870 ms | 0.1591 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 4.373 ms | 0.0872 ms | 0.1528 ms |
totalCount=1000
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 127.432 ms | 0.6471 ms | 0.5737 ms |
| SqlServer_InsertAppend | 230.073 ms | 0.4374 ms | 0.3877 ms |
| SqlServer_SqlBulkCopy_FromList | 5.425 ms | 0.0194 ms | 0.0162 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 5.283 ms | 0.0263 ms | 0.0233 ms |
| MySql_InsertForEach | 606.386 ms | 11.0222 ms | 16.8320 ms |
| MySql_InsertAppend | 20.716 ms | 0.3511 ms | 0.3285 ms |
| MySql_SqlBulkCopy_FromList | 15.641 ms | 0.3126 ms | 0.4065 ms |
| MySql_SqlBulkCopy_FromDataTable | 15.161 ms | 0.2908 ms | 0.3461 ms |
| PostgreSql_InsertForEach | 540.577 ms | 10.7744 ms | 23.4226 ms |
| PostgreSql_InsertAppend | 8.047 ms | 0.1173 ms | 0.1097 ms |
| PostgreSql_SqlBulkCopy_FromList | 7.259 ms | 0.1192 ms | 0.1057 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 7.232 ms | 0.1189 ms | 0.0993 ms |
totalCount=10000
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 1,253.01 ms | 2.507 ms | 2.345 ms |
| SqlServer_InsertAppend | 1,965.34 ms | 2.126 ms | 1.884 ms |
| SqlServer_SqlBulkCopy_FromList | 30.06 ms | 0.228 ms | 0.202 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 27.53 ms | 0.332 ms | 0.310 ms |
| MySql_InsertForEach | 5,967.03 ms | 116.450 ms | 114.369 ms |
| MySql_InsertAppend | 149.48 ms | 2.252 ms | 1.758 ms |
| MySql_SqlBulkCopy_FromList | 84.15 ms | 1.662 ms | 3.358 ms |
| MySql_SqlBulkCopy_FromDataTable | 82.42 ms | 1.648 ms | 3.476 ms |
| PostgreSql_InsertForEach | 5,357.35 ms | 106.063 ms | 155.466 ms |
| PostgreSql_InsertAppend | 72.69 ms | 0.354 ms | 0.296 ms |
| PostgreSql_SqlBulkCopy_FromList | 31.36 ms | 0.623 ms | 0.718 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 31.37 ms | 0.619 ms | 0.737 ms |
totalCount=100000
| Method | Mean | Error | StdDev |
|---|---|---|---|
| SqlServer_InsertForEach | 12,633.6 ms | 216.57 ms | 202.58 ms |
| SqlServer_InsertAppend | 19,484.5 ms | 35.34 ms | 29.51 ms |
| SqlServer_SqlBulkCopy_FromList | 263.1 ms | 5.13 ms | 6.67 ms |
| SqlServer_SqlBulkCopy_FromDataTable | 239.3 ms | 4.53 ms | 5.56 ms |
| MySql_InsertForEach | 59,331.8 ms | 1,135.02 ms | 1,261.57 ms |
| MySql_InsertAppend | 1,227.6 ms | 23.60 ms | 25.25 ms |
| MySql_SqlBulkCopy_FromList | 540.4 ms | 10.71 ms | 21.64 ms |
| MySql_SqlBulkCopy_FromDataTable | 515.3 ms | 10.16 ms | 20.28 ms |
| PostgreSql_InsertForEach | 53,797.8 ms | 740.99 ms | 693.12 ms |
| PostgreSql_InsertAppend | 722.3 ms | 8.35 ms | 7.40 ms |
| PostgreSql_SqlBulkCopy_FromList | 310.4 ms | 3.55 ms | 3.14 ms |
| PostgreSql_SqlBulkCopy_FromDataTable | 280.6 ms | 5.55 ms | 9.57 ms |
5.結論
- 注：10W條數據的典型場景
| 項目 | SqlServer | MySql | PostgreSql |
|---|---|---|---|
| for循環 | for循環 | for循環 | for循環 |
| 最快 | 119ms(0.12s) | 394ms(0.39s) | 208ms(0.2s) |
| 最慢 | 43793ms(43.79s) | 61492ms(61.49s) | 55238ms(55.23s) |
| 差距 | 368倍 | 156倍 | 265倍 |
| - | - | - | - |
| Benchmark | Benchmark | Benchmark | Benchmark |
| 最快 | 239ms(0.24s) | 515ms(0.51s) | 280ms(0.28s) |
| 最慢 | 19484ms(19.48s) | 59331ms(59.33s) | 53797ms(53.79s) |
| 差距 | 81倍 | 115倍 | 192倍 |
- SqlBulkCopy方法能顯著提高批量插入性能
- SqlServer的insert () values(),(),()....語句似乎沒有優化
- SqlServer的整體表現比MySql和PostgreSql好一些(都是空數據庫,表結構簡單,應該還沒有達到硬件限制)
- MySql和PostgreSql的insert () values(),(),()....語句性能不錯,尤其是PostgreSql
- PostgreSql各項指標均優於MySql
五.注意事項
- 構建的DataTable要跟數據庫表完全一致，包含自增列，排除NotMapped
- 構建DataColumn時列名要跟表一致，類型要傳實際類型，不能不傳或者傳object
- MySql連接字符串需要加上AllowLoadLocalInfile=true且服務端設置local_infile=1(建議修改全局配置文件)
- PostgreSql的copy指令對表名大小寫有特殊要求，建議建表和實體特性都使用小寫
- PostgreSql的NpgsqlBinaryImporter.Write(,)類型要求和數據庫一致，需要使用枚舉NpgsqlDbType
作者：NewcatsHuang
時間：2021-12-25
完整代碼：https://github.com/newcatshuang/Newcats.Infrastructure/tree/master/tests/SqlBulkCopyTest
轉載請注明出處，謝謝O(∩_∩)O