iOS自動化探索（四）自動化測試框架pytest - 安裝和使用

自動化測試框架 - pytest

pytest是Python最流行的單元測試框架之一, 幫助更便捷的編寫測試腳本， 並支持多種功能復雜的測試場景， 能用來做app測試也能用作函數測試

官方文檔: https://docs.pytest.org/en/latest/

 

pytest具有以下優點:

• 允許使用assert進行斷言 
• 自動識別測試腳本、類、函數
• 可用於管理小型或者參數類型的測試數據或資源
• 兼容unittest和nose測試框架
• 支持Python2.7/Python3.4+
• 豐富的插件支持，超過315個插件支持

 

pytest安裝

pip install -U pytest

如果提示下面的錯誤，說明是pip的版本太老了， 要更新下：

  Could not find a version that satisfies the requirement pytest (from versions: )
No matching distribution found for pytest

更新方式:

easy_install --upgrade pip

 

官方示例

准備一個test_sample.py, 內容如下：

def inc(x):
    return x + 1

def test_answer():
    assert inc(3) == 5

在文件所在目錄執行:

pytest

這里我們做下說明：

pytest腳本都以test_xxx.py為文件名；

inc方法是我們定義的一個自增函數，該函數將傳遞進來的參數加1后返回；

test_answer是我們編寫的一個測試函數，其中我們使用基本的斷言語句assert來對結果進行驗證，測試函數以test_xxx作為命名

 

執行結果如下：

============================================================ test session starts ============================================================
platform darwin -- Python 2.7.15, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:
collected 1 item                                                                                                                            

test_sample.py F                                                                                                                      [100%]

================================================================= FAILURES ==================================================================
________________________________________________________________ test_answer ________________________________________________________________

    def test_answer():
>       assert inc(3) == 5
E       assert 4 == 5
E        +  where 4 = inc(3)

test_sample.py:5: AssertionError
========================================================= 1 failed in 0.05 seconds ==========================================================
(wda_python) bash-3.2$ 

當執行到assert inc(3) == 5時，報錯

 

執行pytest會在當前目錄和子目錄中尋找test_xx.py的測試文件，並進入到測試文件中尋找test_xx開頭的測試函數開始執行

執行pytest -q  test_xxx.py是執行執行的腳本

 

在看一個例子,測試指定錯誤： (Assert that a certain exception is raised)

import pytest

def f():
    raise SystemExit(1)

def test_mytest():
    with pytest.raises(SystemExit):
        f()

執行指令：

pytest -q test_sysexit.py

輸出:

(wda_python) bash-3.2$ pytest -q test_sysexit.py 
.                                                                                                                                      [100%]
1 passed in 0.04 seconds
(wda_python) bash-3.2$ 

 

如果要開發多個測試方法，可以把方法寫進一個class中

class TestClass(object):
    def test_one(self):
        x = 'this'
        assert 'h' in x

    def test_two(self):
        x = 'hello'
        assert hasattr(x, 'check')

pytest能夠自動識別類中的測試方法， 也不用我們去創建子類或者實實例, 運行結果如下：

(wda_python) bash-3.2$ pytest -q test_sample.py 
.F                                                                                                                                     [100%]
================================================================== FAILURES ==================================================================
_____________________________________________________________ TestClass.test_two _____________________________________________________________

self = <test_sample.TestClass object at 0x102e151d0>

    def test_two(self):
        x = 'hello'
>       assert hasattr(x, 'check')
E       AssertionError: assert False
E        +  where False = hasattr('hello', 'check')

test_sample.py:8: AssertionError
1 failed, 1 passed in 0.08 seconds
(wda_python) bash-3.2$ 

 

除了直接在腳本路徑執行pytest外， 還可以用以下方式

python -m pytest xxx.py

 

出現第一個（或第N個）錯誤時停止

pytest -x            # stop after first failure
pytest --maxfail=2    # stop after two failures

 

運行執行測試腳本

pytest test_mod.py

 

運行指定目錄下的所有腳本

pytest testing/

 

運行包含指定關鍵字的測試方法, 可以是文件名、類名、測試函數名

pytest -k "MyClass and not method"

 

執行node id運行測試腳本，每一個被收集的測試方法都會分配一個指定的id， 我們可以用一下方式運行執行的測試方法:

# To run a specific test within a module
pytest test_mod.py::test_func     

# To run a test within a class
pytest test_mod.py::TestClass::test_method

 

日志打印的不同方式

pytest --showlocals # show local variables in tracebacks
pytest -l           # show local variables (shortcut)

pytest --tb=auto    # (default) 'long' tracebacks for the first and last
                     # entry, but 'short' style for the other entries
pytest --tb=long    # exhaustive, informative traceback formatting
pytest --tb=short   # shorter traceback format
pytest --tb=line    # only one line per failure
pytest --tb=native  # Python standard library formatting
pytest --tb=no      # no traceback at all

 

測試報告

pytest默認是完整的測試報告， 我們可以加上-r標簽顯示簡短測試報告，可再搭配一下參數

Here is the full list of available characters that can be used:

f - failed
E - error
s - skipped
x - xfailed
X - xpassed
p - passed
P - passed with output
a - all except pP

可以多個參數一起使用

 

Debug模式

pytest --pdb

示例:

(wda_python) bash-3.2$ pytest --pdb
========================================================== test session starts ===========================================================
platform darwin -- Python 2.7.15, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:
collected 3 items                                                                                                                        

test_sample.py .F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <test_sample.TestClass object at 0x10e928610>

    def test_two(self):
        x = 'hello'
>       assert hasattr(x, 'check')
E       AssertionError: assert False
E        +  where False = hasattr('hello', 'check')

test_sample.py:8: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test/test_sample.py(8)test_two()
-> assert hasattr(x, 'check')
(Pdb) print x
hello
(Pdb) print hasattr(x,'check')
False
(Pdb) 

 

還可以指定第幾次失敗開始進入debug:

pytest -x --pdb   # drop to PDB on first failure, then end test session
pytest --pdb --maxfail=3  # drop to PDB for first three failures

任何失敗的異常信息都會存儲在sys.last_value，sys.last_type 以及 sys_last_traceback

在debug中可以通過以下方式獲取最后報錯的內容

(Pdb) import sys
(Pdb) sys.last_traceback.tb_lineno
1357
(Pdb) sys.last_value
AssertionError(u"assert False\n +  where False = hasattr('hello', 'check')",)
(Pdb) 

 

在執行一開始就進入到debug模式

pytest --trace

輸入next執行下一步, exit退出

 

腳本中設置斷點

import pdb

pdb.set_trace()

例如：

import pdb

class TestClass(object):
    def test_one(self):
        x = 'this' pdb.set_trace()
        assert 'h' in x

    def test_two(self):
        x = 'hello'
        assert hasattr(x, 'check')

 

獲取執行最慢的n個測試步驟

pytest --durations=10

======================================================= slowest 10 test durations ========================================================

(0.00 durations hidden.  Use -vv to show these durations.)

但如果所有腳本的運行時間都小於0.01s, 就不顯示了， 除非帶上-vv參數

pytest --durations=10 -vv

輸出結果:

======================================================= slowest 10 test durations ========================================================
0.00s call     test_sample.py::TestClass::test_two
0.00s setup    test_sysexit.py::test_mytest
0.00s setup    test_sample.py::TestClass::test_two
0.00s setup    test_sample.py::TestClass::test_one
0.00s teardown test_sample.py::TestClass::test_two
0.00s teardown test_sample.py::TestClass::test_one
0.00s call     test_sysexit.py::test_mytest
0.00s teardown test_sysexit.py::test_mytest
0.00s call     test_sample.py::TestClass::test_one
=================================================== 1 failed, 2 passed in 0.06 seconds ===================================================
(wda_python) bash-3.2$ 

 

將日志保存到指定文件

pytest --resultlog=path

 

Disabling plugins

To disable loading specific plugins at invocation time, use the -p option together with the prefix no:.

Example: to disable loading the plugin doctest, which is responsible for executing doctest tests from text files, invoke pytest like this:

pytest -p no:doctest

 

我們也可以在pytestdemo腳本中去啟動pytest:

import pytest

pytest.main()

執行python pytestdemo.py就可以執行pytest

main()不會拋出SystemExit的異常, 但會返回exitcode, 一共有6種exitcode

Exit code 0:    All tests were collected and passed successfully
Exit code 1:    Tests were collected and run but some of the tests failed
Exit code 2:    Test execution was interrupted by the user
Exit code 3:    Internal error happened while executing tests
Exit code 4:    pytest command line usage error
Exit code 5:    No tests were collected

我們試着加上打印

import pytest

print pytest.main()

輸出：

(wda_python) bash-3.2$ python pytestDemo.py 
========================================================== test session starts ===========================================================
platform darwin -- Python 2.7.15, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/jackey/Documents/iOS/code/iOS-Auto/Agent_Test, inifile:
collected 3 items                                                                                                                        

test_sample.py .F                                                                                                                  [ 66%]
test_sysexit.py .                                                                                                                  [100%]

================================================================ FAILURES ================================================================
___________________________________________________________ TestClass.test_two ___________________________________________________________

self = <test_sample.TestClass object at 0x1038ba650>

    def test_two(self):
        x = 'hello'
>       assert hasattr(x, 'check')
E       AssertionError: assert False
E        +  where False = hasattr('hello', 'check')

test_sample.py:11: AssertionError
=================================================== 1 failed, 2 passed in 0.05 seconds ===================================================
1
(wda_python) bash-3.2$ 

 

我們還可以在main中傳遞參數:

pytest.main(['-q','test_sample.py'])

 

給pytest.main添加plugin, 如下示例在執行的開頭和結尾， 添加打印信息

import pytest

class MyPlugin(object):
    def pytest_sessionfinish(self):
        print '*** Test run reporting finishing'

    def pytest_sessionstart(self):
        print '*** Test run report beginning'

pytest.main(['-q','test_sample.py'], plugins=[MyPlugin()])

輸出：

(wda_python) bash-3.2$ python pytestDemo.py
*** Test run report beginning
.F [100%]*** Test run reporting finishing

================================================================ FAILURES ================================================================
___________________________________________________________ TestClass.test_two ___________________________________________________________

self = <test_sample.TestClass object at 0x1090843d0>

def test_two(self):
x = 'hello'
> assert hasattr(x, 'check')
E AssertionError: assert False
E + where False = hasattr('hello', 'check')

test_sample.py:11: AssertionError
1 failed, 1 passed in 0.05 seconds