[已解決]C#批量高效率導入大數據到數據庫[百萬級以上]


將幾百萬條數據導入到數據庫中,怎么樣高效率的導入?
下面我就介紹一個高效率的方法:
1、將數據庫文件(DB.csv)導入到DataTable中:

/// <summary>
        /// 將CSV文件的數據讀取到DataTable中
        /// </summary>
        /// <param name="fileName">CSV文件路徑</param>
        /// <returns>返回讀取了CSV數據的DataTable</returns>
        public static DataTable OpenCSV(string filePath)
        {
            Encoding encoding = Encoding.GetEncoding("utf-8"); //Encoding.ASCII;//
            DataTable dt = new DataTable();
            FileStream fs = new FileStream(filePath, System.IO.FileMode.Open, System.IO.FileAccess.Read);

            //StreamReader sr = new StreamReader(fs, Encoding.UTF8);
            StreamReader sr = new StreamReader(fs, encoding);
            //string fileContent = sr.ReadToEnd();
            //encoding = sr.CurrentEncoding;
            //記錄每次讀取的一行記錄
            string strLine = "";
            //記錄每行記錄中的各字段內容
            string[] aryLine = null;
            string[] tableHead = null;
            //標示列數
            int columnCount = 0;
            //標示是否是讀取的第一行
            bool IsFirst = true;
            //逐行讀取CSV中的數據
            while ((strLine = sr.ReadLine()) != null)
            {
                //strLine = Common.ConvertStringUTF8(strLine, encoding);
                //strLine = Common.ConvertStringUTF8(strLine);

                if (IsFirst == true)
                {
                    tableHead = strLine.Split(',');
                    IsFirst = false;
                    columnCount = tableHead.Length;
                    //創建列
                    for (int i = 0; i < columnCount; i++)
                    {
                        DataColumn dc = new DataColumn(tableHead[i]);
                        dt.Columns.Add(dc);
                    }
                }
                else
                {
                    if (!String.IsNullOrEmpty(strLine))
                    {
                        aryLine = strLine.Split(',');
                        DataRow dr = dt.NewRow();
                        for (int j = 0; j < columnCount; j++)
                        {
                            dr[j] = aryLine[j];
                        }
                        dt.Rows.Add(dr);
                    }
                }
            }
            if (aryLine != null && aryLine.Length > 0)
            {
                dt.DefaultView.Sort = tableHead[0] + " " + "asc";
            }

            sr.Close();
            fs.Close();
            return dt;
        }
    }

2、將數據庫保存到數據庫:

public static void TableValuedToDB(DataTable dt)
        {
            SqlConnection sqlConn = new SqlConnection(
              ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString);
            const string TSqlStatement =
             "insert into table (col1,col2)" +
             " SELECT nc.col1,nc.col2" +
             " FROM @NewBulkTestTvp AS nc";
            SqlCommand cmd = new SqlCommand(TSqlStatement, sqlConn);
            SqlParameter catParam = cmd.Parameters.AddWithValue("@NewBulkTestTvp", dt);
            catParam.SqlDbType = SqlDbType.Structured;
            //表值參數的名字叫BulkUdt,在上面的建立測試環境的SQL中有。
            catParam.TypeName = "dbo.BulkUdt";
            try
            {
                sqlConn.Open();
                if (dt != null && dt.Rows.Count != 0)
                {
                    cmd.ExecuteNonQuery();
                }
            }
            catch (Exception ex)
            {
                throw ex;
            }
            finally
            {
                sqlConn.Close();
            }
        }

3、在數據庫創建表值參數類型:

create type dbo.BulkUdt(col1 bigint,col2 nvarchar(10));

4、開始導入數據:

            Stopwatch sw = new Stopwatch();
            string filePath = @"C:\DB.csv";
            DataTable dt = CSVFileHelper.OpenCSV(filePath);
            sw.Start();
            TableValuedToDB(dt);
            sw.Stop();
            Trace.WriteLine(string.Format("Elapsed Time is {0} Milliseconds", sw.ElapsedMilliseconds));

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM