基於locust全鏈路壓測系統

本文轉載自查看原文 2021-12-23 18:20 867 locust/ python/ 平台/ 性能/ flask/ 全鏈路

2021年中旬就計划着搭建一套壓測系統，大約9月份已經搭建完成，使用至今還是比較穩定了，分享一下搭建思路及過程：

為什么選擇Locust呢，因為Locust可以僅需要執行命令就可以完成壓測任務，並且集群壓測也很簡單，只需壓測機安裝locust並把壓測腳本推送到服務器即可。

Locust QQ群：

畫了一個大致的思路圖：

我們說的全鏈路其實有幾層意思：

1.多接口多場景，而非單接口或單url

2.按照用戶訪問場景及頻率，用戶訪問的路徑是有先后的，訪問的接口頻率也是不一樣的。怎么理解這個呢，很簡單，比如獲取列表的接口（get_list）和獲取內容的接口（get_content），用戶訪問任何頁面有可能都會訪問

get_list,但用戶可能都不會點擊詳情，所以調用get_list的頻率會更多。

怎么真實的獲取到用戶訪問的鏈路場景呢？

1.通過用戶訪問的日志，分析用戶的行為，然后編寫壓測場景用例

2.模擬用戶場景，導出用戶記錄
　　A.瀏覽器直接導出記錄生成.har文件

　　B.app通過抓包工具獲取用戶記錄導出生成.har文件

當然有的人說har文件解析生成接口后，后續壓測能一直有效么，比如token等校驗通不過，解決這個問題很簡單，和研發商量一下，請求參數里加每個值或對特定設備或標識放開就行，后續一路暢通無阻。

壓測腳本來源有了，第二步就是解析har文件，模塊庫里有解析har的，但發現不滿足自己使用，自己寫吧，項目結構僅供參考：

解析Har文件：

 1 # -*- coding = utf-8 -*-
 2 # ------------------------------
 3 # @time: 2021/3/22 14:53
 4 # @Author: drew_gg
 5 # @File: disassemble_har.py
 6 # @Software: cover_app_platform
 7 # ------------------------------
 8 
 9 import json
10 from app.locust.anasiysis_har import judgment_exist as jud
11 from app.locust.anasiysis_har import deal_headers as dh
12 from app.locust.anasiysis_har import deal_request_data as dr
13 from app.config.har_to_api import api_filter as af
14 
15 
16 key_words = af.key_words
17 
18 
19 def disassemble_har(har_file, api_only=0):
20     """
21     提取分解har文件
22     :param har_file: .har文件
23     :param api_only: 1:去重，其他：不去重
24     :return:
25     """
26 
27     req_l = []
28     rdl = []
29     rdl_set = []
30     host = ''
31     count = 1
32     # url過濾非接口請求
33     with open(har_file, "r", encoding='utf-8') as f:
34         f = json.loads(f.read())
35         for i in f['log']['entries']:
36             if jud.judgment_exist(i['request']['url'], key_words) is False:
37                 req_l.append(i)
38     for index, i in enumerate(req_l):
39         rd = {}
40         # 解析host
41         host = i['request']['url'].split('//')[0] + '//' + i['request']['url'].split('//')[1].split('/')[0]
42         # 解析子url
43         # son_url = i['request']['url'].split(host)[1].split('&')[0]
44         son_url = i['request']['url'].split(host)[1]
45         deal_url = son_url.split('?')[0]
46         if deal_url == '/':
47             if len(son_url.split('?'))> 1:
48                 deal_url = son_url.split('?')[1]
49             else:
50                 deal_url = '/'
51         deal_url = deal_url.replace('/', '_').replace('-', '_').replace('.', '_').strip('_').lstrip('_')
52         if api_only == 1:
53             method_name = 'api_' + deal_url.lower()
54         else:
55             method_name = 'api_' + deal_url.lower() + '_' + str(index)
56         # 解析處理header
57         headers = dh.deal_headers(i['request']['headers'])
58         method = i['request']['method']
59         # 解析處理請求參數
60         if method.upper() == "POST":
61             request_data = dr.deal_request_data(method, i['request']['postData'])
62         if method.upper() == "GET":
63             request_data = '\'' + i['request']['url'].split(son_url)[1] + '\''
64         host = '"' + host + '"'
65         son_url = '"' + son_url + '"'
66         rd['host'] = host
67         rd['url'] = son_url
68         rd['headers'] = headers
69         rd['method'] = method
70         rd['method_name'] = method_name
71         rd['request_data'] = request_data
72         if api_only == 1:
73             # 去重並計數判斷
74             if index == 0:
75                 rd['count'] = count
76                 rdl_set.append(rd)
77             else:
78                 for x in rdl_set:
79                     if son_url == x['url']:
80                         x['count'] += 1
81                         count = x['count']
82                 else:
83                     if count == 1:
84                         rd['count'] = count
85                         rdl_set.append(rd)
86                     count = 1
87         else:
88             rd['count'] = count
89         rdl.append(rd)
90     if api_only != 1:
91         rdl_set = rdl
92     return rdl_set, host
93 
94 
95 if __name__ == '__main__':
96     har_path = r'D:\thecover_project\cover_app_platform\app\file_upload\首頁普通\20210803-113719\syptxq.har'
97     disassemble_har(har_path)

解析har文件，處理header、獲取接口必要參數，然后對請求做分析，如果要去重，則統計相同請求的數量，壓測時生成壓測權重，如果不去重，后續生成壓測腳本時則需要對處理方法名稱。

解析好har文件后，需要生成調試腳本和壓測腳本：

我處理方式實直接生成py文件，事先創建好模板，如：

生成調試腳本比較簡單，只需要一個模板就行，生成locust壓測腳本則稍微負責點，我是分拆成多個模板，然后整合到一個模板。

生成的腳本都規范放在目錄里：

生成腳本目錄結構：

生成壓測腳本示例：

 1 # -*- coding = utf-8 -*-
 2 # ------------------------------
 3 # @time: 2021-04-19 13:43:10.380837
 4 # @Author: drew_gg
 5 # @File: liao_bao.py
 6 # @Software: api_locust
 7 # ------------------------------
 8 
 9 
10 from locust import SequentialTaskSet, task, constant, tag, TaskSet
11 from locust.contrib.fasthttp import FastHttpUser
12 
13 
14 class LiaoBao20210419(TaskSet):
15 
16     @task(1)
17     @tag('api_getlist')
18     def api_getlist(self):
19         headers = {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'tenantId': '7'}
20         # 請求參數組裝 ## r_url:固定參數
21         r_url = "/getList?vno=6.4.0"
22         requests_data = {'account': 'E2247B94-51E2-4952-BC06-24752911C060', 'client': 'iOS', 'data': '{"operation_type":0,"news_id":0,xxxxxxxxxxxxxxxxxxx'}
23         # 發起請求
24         with self.client.post(r_url, data=requests_data, catch_response=True, name=r_url) as r:
25             if r.content == b"":
26                 r.failure("No data")
27             if r.status_code != 200:
28                 em = "request error --" + str(r.status_code)
29                 r.failure(em)
30 
31     @task(4)
32     @tag('api_getsysnotice')
33     def api_getsysnotice(self):
34         headers = {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'tenantId': '7'}
35         # 請求參數組裝 ## r_url:固定參數
36         r_url = "/getSysnotice?vno=6.4.0"
37         requests_data = {'account': 'E251179A-6309-4326-9827-73C892131605', 'client': 'iOS', 'data': '{"page_size":15,"page":1}', xxxxxxxxxxxxxxxxxxxxxxxx}
38         # 發起請求
39         with self.client.post(r_url, data=requests_data, catch_response=True, name=r_url) as r:
40             if r.content == b"":
41                 r.failure("No data")
42             if r.status_code != 200:
43                 em = "request error --" + str(r.status_code)
44                 r.failure(em)
45 
46     @task(4)
47     @tag('api_user_preparecancelaccount')
48     def api_user_preparecancelaccount(self):
49         headers = {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'tenantId': '7'}
50         # 請求參數組裝 ## r_url:固定參數
51         r_url = "/user/prepareCancelAccount?vno=6.4.0"
52         requests_data = {'account': '2FF3D47C-995B-4D7E-93CD-58B4F1E94B74', 'client': 'iOS', 'data': '{}', xxxxxxxxxxxxxxxxxxxxxxx}
53         # 發起請求
54         with self.client.post(r_url, data=requests_data, catch_response=True, name=r_url) as r:
55             if r.content == b"":
56                 r.failure("No data")
57             if r.status_code != 200:
58                 em = "request error --" + str(r.status_code)
59                 r.failure(em)
60 
61 
62 class liao_bao_locust(FastHttpUser):
63     host = "https://xxxxxx.xxxxx.com"
64     wait_time = constant(0)
65     tasks = {LiaoBao20210419: 1}

生成好腳本后，需要生成執行命令：

 1 # -*- coding = utf-8 -*-
 2 # ------------------------------
 3 # @time: 2021/3/3 11:08
 4 # @Author: drew_gg
 5 # @File: locust_create_cmd.py
 6 # @Software: cover_app_platform
 7 # ------------------------------
 8 
 9 
10 def create_master_cmd(locust_pra):
11     """
12     生成master命令
13     :param locust_pra:
14     :return:
15     """
16     # locust master 命令樣式：
17     """    
18         locust -f /work/locust/api_locust/locust_view/fm_api/locust_api/locust_fm_640.py  
19         --master  
20         --master-bind-port 9800 
21         --headless 
22         -u 600 
23         -r 200 
24         --expect-worker 16 
25         -t 10m 
26         -s 10  
27         --csv /work/locust/locust_report/fm/locust_get_dynamic.py0223145309 
28         --html /work/locust/api_locust/resource/html/new_html/locust_get_operation_parm.html
29     """
30     run_port = '9800'
31     master_cmd = "locust -f %s  --master  --master-bind-port %s --headless " % (locust_pra['to_file'], run_port)
32     master_pra = "-u %s -r %s --expect-worker %s -t %ss -s 10  --csv %s --html %s > %s" % \
33                  (locust_pra['user'], locust_pra['rate'], locust_pra['thread'], locust_pra['time'], locust_pra['csv'],
34                   locust_pra['html'], locust_pra['master_log'])
35     master_cmd = master_cmd + master_pra
36     return master_cmd
37 
38 
39 def create_slave_cmd(locust_pra):
40     """
41     生成slave命令
42     :return:
43     """
44     run_port = '9800'
45     if len(locust_pra['api']) == 1 and locust_pra['api'][0] == '':
46         slave_cmd = "locust -f %s --master-host  %s --master-port %s --headless --worker > %s" % \
47                     (locust_pra['to_file'], locust_pra['master'].split('-')[0], run_port, locust_pra['slave_log'])
48     else:
49         tags = ''
50         for i in locust_pra['api']:
51             tags += i.split(".py")[0] + ' '
52         slave_cmd = "locust -f %s --master-host  %s --master-port %s --headless --worker -T %s > %s" % \
53                     (locust_pra['to_file'], locust_pra['master'].split('-')[0], run_port, tags, locust_pra['slave_log'])
54     return slave_cmd