數據篩選和API優化

篩選數據

需求：如果數據庫中存在OrderNum相同，且IsDefault不同的記錄，那么IsDefault值為0的記錄將替換值為1的記錄（IsDefault值為1的記錄不展示）。

由於查出來的數據不多，100條以內，所以我是直接全部查詢到List內存中，然后在內存中進行數據過濾的操作，思來想去都覺得我如下的實現方式很low，但是我一時又沒想到好的辦法，不知道大家有沒有好的辦法？

            var newList = list.ToList();
            //篩選出哪些排序號有重復
            var orderNumList = newList.GroupBy(g => g.OrderNum).Select(g => new { orderNum = g.Key, count = g.Count() }).Where(g => g.count > 1).Select(s => s.orderNum).ToList();
            var cfList = newList.Where(w => orderNumList.Contains(w.OrderNum)); //獲取有重復排序號的記錄

            var cfDefaultList = cfList.Where(w => w.IsDefault);  //默認模塊記錄
            var cfNoDefaultList = cfList.Where(w => w.IsDefault == false); //非默認模塊記錄

            var intersectedList = from d in cfDefaultList join f in cfNoDefaultList on d.OrderNum equals f.OrderNum where d.IsDefault!= f.IsDefault select d;

            var newIntersectedList = intersectedList.Distinct().ToArray(); //排序號相同，既存在默認記錄也存在非默認記錄的數據 

                if (newIntersectedList != null && newIntersectedList.Length > 0)
                {
                    for (int i = 0; i < newIntersectedList.Length; i++)
                    {
                        if (newList.Contains(newIntersectedList[i]))
                        {
                            newList.Remove(newIntersectedList[i]);
                        }
                    }
                }
           newList = newList.OrderBy(x => x.OrderNum).ToList();

 以上的newList就代碼截圖中的數據。

優化API接口

有一個API接口經常卡頓，而且很不穩定，快的時候2~3秒，慢的時候10秒去了。

接口需求：根據社區ID獲取優惠券記錄。

分析：

• 負責給API接口提供數據的系統中，缺少許多索引，存在許多慢查詢視圖。
• 原來的LINQ實現方式是在內存中分頁，響應速度太慢。
• 並發請求的情況下，資源占用。

優化思路：

1、使用緩存

同一個社區的人在同一時間所看到的優惠券記錄應該是一樣的，而且我們應該允許臟讀，我們在12306上面買火車票的時候，經常也會看到顯示有票，但是下單又沒有了，可能是使用了緩存，那么我們這里其實同樣的可以采用緩存來緩解並發問題。

在WebAPI上面加緩存，那么又分為客戶端緩存和服務器緩存。而我們知道，在ASP.NET WebForm和ASP.NET MVC中都是有頁面輸出緩存的，而在WebAPI中默認沒有，從NuGet上面下載WebApi.OutputCache.V2，然后再API接口上添加 

[CacheOutput(ClientTimeSpan = 5)]//, ServerTimeSpan = 5

我這里沒法直接使用服務器輸出緩存，那是因為無法捕獲緩存變量參數。因為我們API接口的請求參數是string appParam，字符串類型的，它是一個json對象進過base64位編碼，然后再進過url編碼生成的字符串。

我們只能解析后，獲取社區ID，然后根據社區ID來設置緩存，把社區ID+ pageIndex就作為緩存的Key，考慮到需要緩存的數據量很小，這里我直接使用.NET自帶的緩存，引入命名空間：System.Web.Caching;

      private static System.Web.Caching.Cache ObjCache = HttpRuntime.Cache;
        /// <summary>
        /// 設置當前指定Key的Cache值,並限定過期時間 
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="TimeOuts">超時時間（秒）</param>
        public static void SetCacheSeconds(string Key, object Obj, double TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddSeconds(TimeOuts), TimeSpan.Zero);
        }
        /// <summary>
        /// 獲取當前指定Key的Cache值
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <returns>緩存的值</returns>
        public static object GetCache(string Key)
        {
            return ObjCache[Key];
        }

緩存操作類Cache完整代碼如下：

/*==================================
 * Author:
 * CreateTime：2014-7-15 17:26:29
 * Description：Cache操作類
 ===================================*/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Caching;
using System.Runtime.CompilerServices;
using System.Web;
using System.Security.Policy;

namespace SSY.Util
{
    /// <summary>
    /// 緩存處理相關類
    /// </summary>
    public class Cache
    {
        private static System.Web.Caching.Cache ObjCache = HttpRuntime.Cache;
        private static short TimeOut = 720;

        #region 清除指定鍵值的緩存
        /// <summary>
        /// 清除指定鍵值的緩存
        /// </summary>
        /// <param name="Key">要清除的緩存的key值</param>
        public static void Clear(string Key)
        {
            if (ObjCache[Key] != null)
            {
                ObjCache.Remove(Key);
            }
        }
        #endregion

        #region 返回系統中緩存的個數
        /// <summary>
        /// 返回系統中緩存的個數
        /// </summary>
        /// <returns>緩存個數</returns>
        public static int Count()
        {
            return ObjCache.Count;
        }
        #endregion

        #region 獲取當前指定Key的Cache值
        /// <summary>
        /// 獲取當前指定Key的Cache值
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <returns>緩存的值</returns>
        public static object GetCache(string Key)
        {
            return ObjCache[Key];
        }
        #endregion

        #region 設置當前指定Key的Cache值
        /// <summary>
        /// 設置當前指定Key的Cache值 
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        public static void SetCache(string Key, object Obj)
        {
            ObjCache.Insert(Key, Obj);
        }
        #endregion

        #region 設置當前指定Key的Cache值,並限定過期時間
        /// <summary>
        /// 設置當前指定Key的Cache值,並限定過期時間 
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="TimeOuts">超時時間（分鍾）</param>
        public static void SetCache(string Key, object Obj, int TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddMinutes((double)TimeOuts), TimeSpan.Zero);
        }
        /// <summary>
        /// 設置當前指定Key的Cache值,並限定過期時間 
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="TimeOuts">超時時間（秒）</param>
        public static void SetCacheSeconds(string Key, object Obj, double TimeOuts)
        {
            ObjCache.Insert(Key, Obj, null, System.DateTime.Now.AddSeconds(TimeOuts), TimeSpan.Zero);
        }
        #endregion

        #region 設置當前指定Key的Cache值,依賴文件過期
        /// <summary>
        /// 設置當前指定Key的Cache值,依賴文件過期 
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="Files">相對地址，例如"~/files.xml"</param>
        public static void SetCache(string Key, object Obj, string Files)
        {
            CacheDependency cacheDep = new CacheDependency(System.Web.HttpContext.Current.Server.MapPath(Files),System.DateTime.Now);
            SetCache(Key, Obj, TimeOut, cacheDep, CacheItemPriority.High);
        }
        #endregion

        #region 設置當前指定Key的Cache值
        /// <summary>
        /// 設置當前指定Key的Cache值
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="Priority">撤銷緩存的優先值，此參數的值取自枚舉變量“CacheItemPriority”，優先級低的數據項將先被刪除。此參數主要用在緩存退出對象時.</param>
        public static void SetCache(string Key, object Obj, CacheItemPriority Priority)
        {
            SetCache(Key, Obj, TimeOut, null, Priority);
        }
        #endregion

        #region 設置當前指定Key的Cache值
        /// <summary>
        /// 設置當前指定Key的Cache值
        /// </summary>
        /// <param name="Key">緩存Key</param>
        /// <param name="Obj">緩存的值</param>
        /// <param name="TimeOuts">一個TimeSpan，表示緩存參數將在多長時間以后被刪除</param>
        /// <param name="CacheDep">緩存的依賴項,需要一個CacheDependency，可初始化一個</param>
        /// <param name="Priority">撤銷緩存的優先值，此參數的值取自枚舉變量“CacheItemPriority”，優先級低的數據項將先被刪除。此參數主要用在緩存退出對象時</param>
        public static void SetCache(string Key, object Obj, int TimeOuts, CacheDependency CacheDep, CacheItemPriority Priority)
        {
            ObjCache.Insert(Key, Obj, CacheDep, System.DateTime.MaxValue, TimeSpan.FromHours((double)TimeOuts), Priority, null);
        }
        #endregion
    }
}

修改API接口代碼：

           #region added by zouqj 2017-3-7
           var result = Util.Cache.GetCache(CommunityID+ pageIndex) as Result<List<GetAvailableCouponsModel>>;
           if (result==null) //不存在則寫入緩存
           {
               //組裝參數
               Dictionary<string, string> inParams = new Dictionary<string, string>();
               inParams.Add("UserId", userId);
               inParams.Add("PageIndex", pageIndex);
               inParams.Add("PageSize", pageSize);
               inParams.Add("CommunityID", CommunityID+ pageIndex);
               ...

               RequestParam RequestParam = GetRequestParam(methodName, inParams, AuthenticationId);
               var jsonContent = JsonConvert.SerializeObject(RequestParam);
               result = DoPost<List<GetAvailableCouponsModel>>(jsonContent, PostUrl);

               Util.Cache.SetCacheSeconds(CommunityID, result, 10); //寫入緩存
           }
           return result;

2、改為存儲過程實現

因為這個接口的業務邏輯比較復雜，之前的Linq代碼寫了好長一大串，獲取的記錄數很多，而且還是在內存中進行分頁實現，所以我將原來的LINQ實現代碼修改為分頁存儲過程實現。

存儲過程代碼如下：

------------------------------------------創建領券中心存儲過程 created by zouqj-2017-3-1-----------------------------------
IF EXISTS(Select Name From Sysobjects Where Name='usp_GetAvailableCoupons' And Xtype='P')
DROP PROCEDURE usp_GetAvailableCoupons
GO
CREATE PROC usp_GetAvailableCouponsl
(
@PageIndex int, --頁碼
@PageSize int, --每頁容納的記錄數
@Sort NVARCHAR(50), --排序字段及規則,不用加order by
@hostName nvarchar(100),--服務器地址
@CommunityID UNIQUEIDENTIFIER, --社區
@IsGetCount BIT --是否得到記錄總數，1為得到記錄總數，0為不得到記錄總數，返回記錄集
)
AS
-------------------------------定義變量-----------------------
declare @strSql NVARCHAR(max);
DECLARE  @dt datetime2(7) --查詢時間
SET @dt=GETDATE();
set nocount on;
----------------------------------------SQL開始--------------------------------------------無分類
IF @IsGetCount=1
BEGIN

SET @strSql=N'SELECT COUNT(*) FROM (SELECT DISTINCT t.CampaignID from
(SELECT 
c.ID AS CampaignID
 FROM MK_Campaign c WITH ( NOLOCK ) INNER JOIN MK_CouponConfig f WITH ( NOLOCK ) ON c.ID=f.CampaignID
           INNER JOIN MK_Coupon p WITH ( NOLOCK ) ON f.CouponID=p.ID
           WHERE f.CouponGetType=2 AND f.ReceiveStartTime <=@dt AND f.ReceiveEndTime>= @dt
           AND c.State=4 --執行中
           AND p.WholeNetwork=1 --全網優惠
UNION ALL
SELECT
    a.[ID] AS CampaignID
    FROM  [dbo].[MK_Campaign] AS a WITH ( NOLOCK )
        INNER JOIN [dbo].[MK_CouponConfig] AS b  WITH ( NOLOCK ) ON a.[ID] = b.[CampaignID]
        AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) AND (2 = b.[CouponGetType]) AND (4 = a.[State]) 
        INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID]
        INNER JOIN [dbo].[MK_CouponRestriction] AS d  WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]
        LEFT OUTER JOIN [dbo].[MK_CouponRestrictCategory] AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID]
        LEFT OUTER JOIN [dbo].[MK_CouponRestrictionOrg] AS f WITH ( NOLOCK ) ON d.[ID] = f.[CouponRestrictionID]
        INNER JOIN [dbo].[ViewOrganizationCommunityForInterface] AS v ON f.[OrgID] = v.[ID]    
    where v.CommunityID=@CommunityID  and 1 = d.[Type]
UNION ALL
SELECT 
   a.ID AS CampaignID
   FROM MK_Campaign a  WITH ( NOLOCK ) 
             INNER JOIN MK_CouponConfig b WITH ( NOLOCK ) ON a.ID=b.CampaignID AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >=@dt) 
             INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID] and (2 = b.[CouponGetType]) AND (4 = a.[State]) 
             INNER JOIN [dbo].[MK_CouponRestriction] AS d WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]  AND (2 = d.[Type]) 
             LEFT OUTER JOIN MK_CouponRestrictionProduct AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID] 
             LEFT OUTER JOIN SL_Product AS f WITH ( NOLOCK ) ON (e.ProductID = f.ID)
             INNER JOIN ViewOrganizationCommunityForInterface v ON v.ID=f.PublisherID
   where v.CommunityID=@CommunityID
   ) t
 ) AS tt'
----------------------------------------------------------------------------------
END
ELSE
BEGIN
SET @strSql=N'SELECT DISTINCT t.* from(
SELECT 
          c.ID AS CampaignID,
          c.Name AS CampaignName,
          f.ValidityStartTime AS CampaignStartTime,
          f.ValidityEndTime AS CampaignEndTime,
          p.Name AS CouponsName,
          p.Price AS CouponsAmount,
          (CASE WHEN p.IsLimited =1 THEN 1 ELSE 2 END) AS IsLimited,
          p.FullAmount AS MinAmount,
          f.ValidityEndTime AS CouponsEndTime,
          f.ValidityStartTime AS CouponsStartTime,
          (CASE WHEN f.IsRepeateGet =1 THEN 1 ELSE 2 END) AS IsCanRepeatedReceive,
          f.ReceiveAddress AS ReceiveAddress,
          f.ReceiveEndTime AS ReceiveEndTime,
          f.ReceiveStartTime AS ReceiveStartTime,
          f.ReceiveMode AS ReceiveMethod,
          f.ProvideNum AS ReceiveNum,
          p.CreateTime AS CreateTime,
          p.Price AS Price,
          f.RemainCouponNum AS RemainCouponNum,
          f.ID AS CouponConfigId,
          p.WholeNetwork AS CouponsType,
          (CASE WHEN f.IconUrl IS NULL THEN N'''' WHEN f.IconUrl=N'''' THEN N'''' ELSE @hostName+f.IconUrl END) AS IconUrl 
FROM MK_Campaign c WITH ( NOLOCK ) 
                 INNER JOIN MK_CouponConfig f WITH ( NOLOCK ) ON c.ID=f.CampaignID
                 INNER JOIN MK_Coupon p WITH ( NOLOCK ) ON f.CouponID=p.ID
          WHERE f.CouponGetType=2 AND f.ReceiveStartTime <= @dt AND f.ReceiveEndTime>= @dt AND c.State=4 AND p.WholeNetwork=1 --全網優惠
--
UNION ALL
          SELECT a.[ID] AS CampaignID,
          a.[Name] AS CampaignName, 
          b.[ValidityStartTime] AS CampaignStartTime,
          b.[ValidityEndTime] AS CampaignEndTime, 
          c.[Name] AS CouponsName, 
          c.[Price] AS CouponsAmount, 
          CASE WHEN (c.[IsLimited] = 1) THEN 1 ELSE 2 END AS IsLimited, 
          c.[FullAmount] AS MinAmount, 
          b.[ValidityEndTime] AS CouponsEndTime, 
          b.[ValidityStartTime] AS CouponsStartTime, 
          CASE WHEN (b.[IsRepeateGet] = 1) THEN 1 ELSE 2 END AS IsCanRepeatedReceive, 
          b.[ReceiveAddress] AS [ReceiveAddress], 
          b.[ReceiveEndTime] AS [ReceiveEndTime], 
          b.[ReceiveStartTime] AS [ReceiveStartTime], 
          b.[ReceiveMode] AS ReceiveMethod, 
          b.[ProvideNum] AS ReceiveNum, 
          c.[CreateTime] AS CreateTime, 
          c.[Price] AS Price, 
          b.[RemainCouponNum] AS RemainCouponNum, 
          b.[ID] AS CouponConfigId, 
          e.[Type] AS CouponsType, 
          CASE WHEN (b.[IconUrl] = N'''' OR b.[IconUrl] IS NULL) THEN N'''' ELSE @hostName+b.[IconUrl] END AS IconUrl
                    FROM  [dbo].[MK_Campaign] AS a WITH ( NOLOCK )
                        INNER JOIN [dbo].[MK_CouponConfig] AS b  WITH ( NOLOCK ) ON a.[ID] = b.[CampaignID] AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) AND (2 = b.[CouponGetType]) AND (4 = a.[State]) 
                        INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID]
                        INNER JOIN [dbo].[MK_CouponRestriction] AS d  WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]
                        LEFT OUTER JOIN [dbo].[MK_CouponRestrictCategory] AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID]
                        LEFT OUTER JOIN [dbo].[MK_CouponRestrictionOrg] AS f WITH ( NOLOCK ) ON d.[ID] = f.[CouponRestrictionID]
                        INNER JOIN [dbo].[ViewOrganizationCommunityForInterface] AS v ON f.[OrgID] = v.[ID]    
                    where v.CommunityID=@CommunityID  and 1 = d.[Type]
UNION ALL
SELECT 
       a.ID AS CampaignID,
       a.Name AS CampaignName,
       b.ValidityStartTime AS CampaignStartTime,
       b.ValidityEndTime AS CampaignEndTime,
       c.Name AS CouponsName,
       c.Price AS CouponsAmount,
       (CASE WHEN c.IsLimited=1 then 1 else 2 END) AS IsLimited,
       c.FullAmount AS MinAmount,
       b.ValidityEndTime AS CouponsEndTime,
       b.ValidityStartTime AS CouponsStartTime,
       (case when b.IsRepeateGet=1 then 1 else 2 END) AS IsCanRepeatedReceive,
       b.ReceiveAddress AS ReceiveAddress,
       b.ReceiveEndTime AS ReceiveEndTime,
       b.ReceiveStartTime AS ReceiveStartTime,
       b.ReceiveMode AS ReceiveMethod,
       b.ProvideNum AS ReceiveNum,
       c.CreateTime AS CreateTime,
       c.Price AS Price,
       b.RemainCouponNum AS RemainCouponNum,
       b.ID AS CouponConfigId,
       d.[TYPE] AS CouponsType,
       (CASE WHEN b.IconUrl IS NULL THEN N'''' WHEN b.IconUrl=N'''' THEN N'''' ELSE @hostName+b.IconUrl END) AS IconUrl
   FROM MK_Campaign a  WITH ( NOLOCK ) 
                    INNER JOIN MK_CouponConfig b WITH ( NOLOCK ) ON a.ID=b.CampaignID AND (b.[ReceiveStartTime] <= @dt) AND (b.[ReceiveEndTime] >= @dt) 
                    INNER JOIN [dbo].[MK_Coupon] AS c WITH ( NOLOCK ) ON b.[CouponID] = c.[ID] and (2 = b.[CouponGetType]) AND (4 = a.[State]) 
                    INNER JOIN [dbo].[MK_CouponRestriction] AS d WITH ( NOLOCK ) ON c.[ID] = d.[CompainID]  AND (2 = d.[Type]) 
                    LEFT OUTER JOIN MK_CouponRestrictionProduct AS e WITH ( NOLOCK ) ON d.[ID] = e.[CouponRestrictionID] 
                    LEFT OUTER JOIN SL_Product AS f WITH ( NOLOCK ) ON (e.ProductID = f.ID)
                    INNER JOIN ViewOrganizationCommunityForInterface v ON v.ID=f.PublisherID
                    where v.CommunityID=@CommunityID
   ) t
ORDER BY t.Price--@Sort
offset (@PageIndex-1)*@PageSize ROWS  FETCH NEXT @PageIndex*@PageSize ROWS ONLY'
end
--執行SQL
exec sp_executesql @strSql,N'@PageIndex int,@PageSize int,@Sort nvarchar(50),@hostName nvarchar(100),@CommunityID UNIQUEIDENTIFIER,@IsGetCount bit,@dt datetime2(7)',@PageIndex =@PageIndex,@PageSize =@PageSize,@Sort=@Sort,@hostName=@hostName,@CommunityID=@CommunityID,@IsGetCount=@IsGetCount,@dt=@dt
set nocount off;

這里需要注意的是，存儲執行是，先關閉計數，set nocount on;，然后再打開set nocount off;，這樣可以提升性能。還有就是使用WITH ( NOLOCK )允許臟讀，提升查詢效率。

這里遇到一個很詭異的問題，我使用exec sp_executesql @strSql,N'....'的方式來執行是沒有問題的，而如果我使用拼接sql的方式，會報錯，因為sql字符串被截斷了，只截取到了4000個字符長度，即便我把字符串變量長度設置為nvarchar(max)也沒用。

分頁方式采用Sqlserver2012以上版本才支持的高效方式：offset ... FETCH NEXT ...ROWS ONLY

原來Linq的執行時間測試：

我修改為存儲過程之后的執行時間測試：

性能差距非常明顯。