1.思路
1）獲取全量代碼覆蓋率報告；
2）指定兩個版本對比，得到增量代碼；
3）通過增量代碼獲取到增量包名、類、方法、新增行數組成的字典；
4）通過全量覆蓋率文件獲取到文件增量代碼行、增量代碼行數、覆蓋行、覆蓋行數；
5）循環讀取，更改各個目錄下的index.html和類名.html文件；顯示新增行數、覆蓋行數和覆蓋率；
6）循環讀取，更改類名.java.html文件，在新增行數前加上藍色的鑽石標記；
2.實現
diff_processor.py 執行增量篩選操作
class DiffProcessor():
def __init__(self, diffpath,code_path,coco_path):
'''
:param diffpath:gitdiff代碼文件路徑
:param code_path: 源碼路徑
:param coco_path: jacoco全量覆蓋率報告路徑
'''
self.diffpath = diffpath
self.code_path = code_path
self.coco_path = coco_path
2.1 get_diff方法：
1）循環讀取gitdiff文件，如果當前行是以"diff --git"開頭的，獲取完整文件名，如：src/main/webapp/WEB-INF/default/jsp/tlfund/management/fund_transfer_record_list.jsp;
2）如果當前行中包含"@@XXX @@"字樣的，獲取classname；
3）如果當前行是以“-”開頭的，跳過；
4）過濾方法名：如果當前行不是以"+//"、“//”開頭的，包含"private"或"public"和"("，且"function"、“=”、“if(”、"if ("、"for "、“for(”、“catch”、"logger."、“.”不在行內，且不是以";"結尾的，獲取當前行並過濾出方法名；如果方法名不是"if"、"for"且"{"、"}"、"."、"+"、"@"不在行內的；將方法名添加到字典中，如果方法名已存在，將當前行添加到方法名的列表中；
5）如果當前行是以"+"開頭的，添加到對應的方法名列表中；
6）最終返回{"文件名":{"diff_voids":{{方法名:[新增行]},'diff_lines':[所有新增行]}}的字典；
def get_diff(self):
"""獲取diff詳情"""
with open(self.diffpath, 'r+') as e:
data = e.read()
diff = data.split("\n")
ret = {}
void_dics = {}
diff_lines = []
file_name = ""
line_void = ""
current_line = 0
for line in diff:
if line.startswith('diff --git'):
# 進入新的block
if file_name != "":
ret[file_name] = {}
ret[file_name]["diff_lines"] = diff_lines
ret[file_name]['diff_voids'] = void_dics
file_name = re.findall('b/(\S+)$', line)[0]
diff_lines = []
void_dics = {}
current_line = 0
classname = ""
elif re.match('@@ -\d+,\d+ \+(\d+),\d+ @@', line):
match = re.match('@@ -\d+,\d+ \+(\d+),\d+ @@', line)
current_line = int(match.group(1)) - 1
if "class" in line:
classname = line.split("class")[-1].split(" ")[1]
if classname and classname not in void_dics.keys():
void_dics[classname]=[]
line_void = ""
elif classname in void_dics:
line_void = ""
elif line.startswith("-"):
continue
elif not line.startswith("+//") and not line.startswith("//") and ("public" in line or "private" in line) and "(" in line and "function" not in line and "=" not in line \
and "if(" not in line and "if (" not in line and "for " not in line and "for(" not in line and "catch" not in line and "logger." not in line and not line.strip("\t").endswith(";") \
and "." not in line:
current_line += 1
line_void = line.split("(")[0].split(" ")[-1].strip("//").strip("\t")
if line_void and line_void != "if" and line_void != "for" and "{" not in line_void and "}" not in line_void and "." not in line_void and "+" not in line_void and "@" not in line_void:
if line_void in void_dics.keys() and isinstance(void_dics[line_void], list):
void_dics[line_void].append(current_line)
else:
void_dics[line_void] = []
elif line.startswith("+") and not line.startswith('+++'):
current_line += 1
diff_lines.append(current_line)
if line_void in void_dics.keys() and isinstance(void_dics[line_void], list):
void_dics[line_void].append(current_line)
elif classname:
void_dics[classname].append(current_line)
else:
current_line += 1
ret[file_name] = {}
ret[file_name]["diff_lines"] = diff_lines
ret[file_name]['diff_voids'] = void_dics
return ret
2.2 is_interface方法：
根據傳遞過來的文件名，判斷是否為接口；返回判斷狀態
def is_interface(self, file_name):
"""判斷某個文件是否是接口"""
ret = False
name = re.search('(\w+)\.java$', file_name).group(1)
reg_interface = re.compile('public\s+interface\s+{}'.format(name))
with open(file_name) as fp:
for line in fp:
line = line.strip()
match = re.match(reg_interface, line)
if match:
ret = True
break
return ret
2.3 modify_html方法
根據傳遞過來的文件路徑和新增總行列表，對新增行通過class新增藍色鑽石標志，並統計新增行、覆蓋行、新增行數和覆蓋行數后返回；
def modify_html(self, html_file_name, diff_lines):
new_line_count = 0
cover_line_count = 0
content = []
new_lines = []
cover_lines = []
with open(html_file_name, 'r') as fp:
content = fp.readlines()
for i in range(1, len(content)):
if i + 1 in diff_lines:
match = re.search('class="([^"]+)"', content[i])
if match:
content[i] = re.sub('class="([^"]+)"', lambda m: 'class="{}-diff"'.format(m.group(1)) if "diff" not in m.group(1) else "class={}".format(m.group(1)), content[i])
css_class = match.group(1)
new_line_count += 1
new_lines.append(i+1)
if css_class.startswith("fc") or css_class.startswith("pc"):
cover_line_count += 1
cover_lines.append(i+1)
with open(html_file_name, 'w') as fp:
fp.write("".join(content))
return new_line_count,cover_line_count,new_lines,cover_lines
2.4 get_package方法
通過傳遞過來的完整文件名，獲取包名
def get_package(self, file_name):
"""獲取package名"""
ret = ''
with open(file_name) as fp:
for line in fp:
line = line.strip()
match = re.match('package\s+(\S+);', line)
if match:
ret = match.group(1)
break
return ret
2.5 resolve_file_info方法
根據傳遞過來的完整文件名，組成完整路徑，調用get_package方法獲取包名，通過正則獲取類名，調用is_interface方法判斷是否為接口方法
def resolve_file_info(self,file_name):
module_name = file_name.split('/')[0]
full_path = os.path.join(self.code_path,file_name)
package = self.get_package(full_path)
class_ = re.search('(\w+)\.java$',file_name).group(1)
is_interface = self.is_interface(full_path)
return (module_name,package, class_, is_interface)
2.6 process_diff方法
調用get_diff方法獲取返回的字典；返回兩個字典：ret是{"new":新增行數,"cover":覆蓋行數,"new_lines":新增的行,"cover_lines":覆蓋行}，diff_result是get_diff方法返回的字典；
def process_diff(self):
'''
過濾出新增包名、類和新增代碼行、已覆蓋行數並返回
'''
ret = {}
diff_result = self.get_diff()
for file_name in diff_result:
# 過濾掉只有刪除，沒有新增的代碼
if diff_result[file_name]['diff_lines'] == []:
continue
# 過濾掉非 java 文件和測試代碼
if file_name.endswith("$.java") or not file_name.endswith(".java") or 'src/test/java/' in file_name:
continue
module_name, package, class_, is_interface = self.resolve_file_info(file_name)
# 過濾掉接口和非指定的module
if is_interface:
continue
html_file_name = os.path.join(self.coco_path,package,"{}.java.html".format(class_))
new_line_count,cover_line_count,new_lines,cover_lines= self.modify_html(html_file_name, diff_result[file_name]['diff_lines'])
# 信息存進返回值
if package not in ret:
ret[package] = {}
ret[package][class_] = {"new": new_line_count,"cover": cover_line_count,"new_lines":new_lines,"cover_lines":cover_lines}
return ret,diff_result
2.7 Del_Dr方法
該方法需要傳入的參數為：要修改的html文件的路徑、html頁面要保留的類名列表或文件列表、process_diff方法返回的ret字典、要修改的類型(root/package/file)；通過傳入參數針對多余的文件進行刪除操作並修改html頁面元素展示增量報告；
def Del_Dr(self, htmlpath, dirlist, ret, filetype, *kwargs):
'''
該函數的目的是為了修改html頁面的內容
:param htmlpath:要修改的html文件的路徑
:param dirlist:html頁面要保留的類名列表或文件
:param ret:ret
:param diff_results:diff文件過濾的字典
:param filetype:需要修改的類型(root指文件根目錄，package指包目錄下，file指文件類型)
'''
with open(htmlpath, 'r') as e:
html_doc = "".join(e.readlines())
soup = BeautifulSoup(html_doc, 'lxml')
a_list = soup.select("a") # 獲取html頁面所有的a標簽
for a_s in a_list:
a_s_text = a_s.text.strip("\n").strip(" ").strip("\n") # 循環獲取a標簽的text屬性並過濾掉\n和空格
if filetype == "file":
a_s_text = a_s_text.split("(")[0]
if str(a_s_text) not in dirlist and a_s.parent.parent.name == "tr": # 如果text不等於要保留的類名，則直接刪除該節點所屬的tr標簽
a_s.parent.parent.extract()
del_td = soup.find_all("tr")[0].find_all("td")[1:]
for td in del_td:
td.extract()
# 新增td行Add lines
new_tr = soup.new_tag("td")
new_tr.string = "Add lines"
soup.thead.tr.append(new_tr)
new_tr.attrs = {'class': 'sortable'}
# 新增td行Overlay lines
overlay_tr = soup.new_tag("td")
overlay_tr.string = "Overlay lines"
soup.thead.tr.append(overlay_tr)
overlay_tr.attrs = {'class': 'sortable'}
# 新增td行Coverage
coverage_tr = soup.new_tag("td")
coverage_tr.string = "Coverage"
soup.thead.tr.append(coverage_tr)
coverage_tr.attrs = {'class': 'sortable'}
pack_tr_list = soup.find_all("tbody")[0].find_all("tr") # 獲取tbody中tr組成的列表
for tpack in pack_tr_list: # 刪除tbody中tr中除類名或文件名的其他列
for pa_td in tpack.find_all("td")[1:]:
pa_td.extract()
tfoot_list = soup.find_all("tfoot")[0].find_all("td")[1:] # 刪除tfoot中除Total外的其他列
for tfoot in tfoot_list:
tfoot.extract()
for npack in pack_tr_list:
pack_name = npack.find_all("a")[0].string.strip("\n").strip(" ").strip("\n")
addlines = 0
covlines = 0
if filetype == "package": # 如果是包名下的index.html文件做如下處理
addlines = ret[pack_name]['new']
covlines = ret[pack_name]['cover']
elif filetype == "root":
for k, v in enumerate(ret[pack_name]):
addlines += ret[pack_name][v]['new']
covlines += ret[pack_name][v]['cover']
elif filetype == "file":
pack_void_name = pack_name.split("(")[0]
filename, diff_dict = kwargs
filename_new_list = filename.split("src/main/java/")[-1].split("/")
filename_new = ".".join(filename_new_list[:-1])
class_name = filename_new_list[-1].split(".")[0]
if filename in diff_dict.keys() and class_name in ret[filename_new].keys():
void_lines_list = diff_dict[filename]['diff_voids'][pack_void_name]
new_line_list = list(
set(ret[filename_new][class_name]['new_lines']).intersection(set(void_lines_list)))
cover_line_list = list(
set(ret[filename_new][class_name]['cover_lines']).intersection(set(void_lines_list)))
addlines = len(new_line_list)
covlines = len(cover_line_list)
if addlines == 0:
coverage = '{:.2%}'.format(0)
else:
coverage = '{:.2%}'.format(covlines / addlines) # 覆蓋率
addlines_tr = soup.new_tag("td")
if addlines:
addlines_tr.string = "%s" % addlines
npack.append(addlines_tr)
covlines_tr = soup.new_tag("td")
covlines_tr.string = "%s" % covlines
npack.append(covlines_tr)
coverage_tr = soup.new_tag("td")
coverage_tr.string = "%s" % coverage
npack.append(coverage_tr)
else:
npack.extract()
# 重新生成index.html頁面
html_path_new = htmlpath + "_bat"
with open(html_path_new, 'w+') as f:
f.write(HTMLParser.HTMLParser().unescape(soup.prettify()))
os.remove(htmlpath)
os.rename(html_path_new, htmlpath)
2.8 diff_file
循環調用Del_Dr方法執行增量代碼覆蓋率報告篩選操作；
def diff_file(self):
'''刪除多余文件夾並修改index.html文件'''
ret,diff_results = self.process_diff()
diffpaths_list = diff_results.keys()
package_list = ret.keys()
dirs_name = os.listdir(self.coco_path)
del_dirs = list(set(dirs_name).difference(set(package_list)))
for i in del_dirs:
if os.path.isdir(os.path.join(self.coco_path, i)): # 若差集列表中的文件是文件夾則循環刪除
re_file = os.path.join(self.coco_path, i)
shutil.rmtree(re_file)
htmlpath = os.path.join(self.coco_path,'index.html')
new_package_list=[]
for pack in package_list:
java_path = os.path.join(self.coco_path,pack)
java_list = []
fina_java_list = []
for java_name in ret[pack].keys():
java_file_name = "src/main/java/" + pack.replace(".","/") + "/" + java_name + ".java"
if java_file_name in diffpaths_list:
javafile_path = os.path.join(java_path,java_name+ '.html')
diff_void_list = diff_results[java_file_name]['diff_voids'].keys()
if len(diff_void_list):
self.Del_Dr(javafile_path,diff_void_list,ret,'file',java_file_name,diff_results)
java_list.append(java_name + '.html')
java_list.append(java_name + '.java.html')
fina_java_list.append(java_name)
new_package_list.append(pack)
fina_java_list = list(set(fina_java_list))
self.Del_Dr(os.path.join(java_path, 'index.html'),fina_java_list, ret[pack],'package')
if len(java_list):
java_list.append("index.html")
java_list.append("index.source.html")
java_dirs = os.listdir(java_path)
del_javas = list(set(java_dirs).difference(set(java_list)))
for d_java in del_javas:
os.remove(os.path.join(java_path,d_java))
new_package_list = list(set(new_package_list))
self.Del_Dr(htmlpath,new_package_list, ret, 'root')
main.py 接收傳遞過來的參數，調用diff_processor中的類執行篩選操作
import argparse,sys,shutil,os
from diff_processor import DiffProcessor
def main(argv):
#解析命令行參數
parser = argparse.ArgumentParser(description=u"計算增量覆蓋率")
parser.add_argument("-dir",type=str,help=u"工程根目錄")
parser.add_argument("-giffdir",type=str,help=u"gitdiff文件路徑")
parser.add_argument("-jareport",type=str,help=u"jacoco_report路徑")
opts = parser.parse_args(argv[1:])
if opts.dir is None or opts.giffdir is None or opts.jareport is None:
parser.print_help()
sys.exit()
#生成增量報告
processor = DiffProcessor(opts.giffdir,opts.dir,os.path.join(opts.jareport,'Check_Order_related'))
processor.diff_file()
#拷貝css和圖片資源
shutil.copy('diff.gif', os.path.join(opts.jareport, "jacoco-resources"))
shutil.copy('report.css', os.path.join(opts.jareport, "jacoco-resources"))
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
使用方法：
執行命令：
python main.py -d 源碼路徑 -g diff文件路徑 -j jacoco_report路徑 如： python main.py -d /root/.jenkins/workspace/Autotest -g /opt/girdiff_1.txt -j /opt/update_report
3.持續集成
3.1 思路
1）jenkins job構建時支持可選擇git分支，執行構建，生產gitdiff文件和項目部署；
2）執行手工測試；
3）生成全量覆蓋率報告並執行篩選操作；
4）將篩選后的增量覆蓋率文件部署到apache下，瀏覽器訪問查看增量報告；
3.2 步驟
1）服務器部署apache並將端口修改為8033；
2）新建部署job1，選擇參數化構建git parameter，如下圖所示；
3）git源碼管理中將分支更改為${new_barnch}；
4）構建執行shell中新增以下代碼，其他參照獲取之前的文章，部署環境：
echo "開始git diff"
git diff ${old_branch} ${new_branch} >/opt/girdiff_1.txt
echo "git diff已完成，生成增量代碼文件，請查看"
5）選擇build with parameters，選擇old_branch和new_branch，點擊執行構建；
6）部署成功后，執行手工測試；
7）復制python_diff文件夾到/opt目錄下；（文件夾中包含__init__.py、diff.gif、diff_processor.py、main.py和report.css文件）
8）新建job2，構建選擇執行shell，shell內容如下：
cd /opt ant dump ant report rm -rf /opt/update_report #刪除增量報告 mkdir /opt/update_report #新建增量文件路徑 cp -r /opt/jacoco_report/* /opt/update_report #復制全量報告到update_report目錄下 cd python_diff python main.py -d /root/.jenkins/workspace/Autotest -g /opt/girdiff_1.txt -j /opt/update_report #執行增量過濾操作 cd /var/www/html/ rm -rf ./* #刪除/var/www/html/目錄下的所有文件 cp -r /opt/update_report/* /var/www/html/ chmod -R 755 ./* service httpd restart #重啟apache echo "過濾完成，請訪問http://ip:8033/index.html查看報告！"
9）訪問http://ip:8033/index.html查看報告；