python asyncio 異步 I/O - 實現並發http請求(asyncio + aiohttp)

前言

如果需要並發 http 請求怎么辦呢？requests庫是同步阻塞的，必須等到結果才會發第二個請求，這里需使用http請求異步庫 aiohttp。

環境准備

aiohttp 用於 asyncio 和 Python 的異步 HTTP 客戶端/服務器。
使用pip安裝對應的包。當前使用版本v3.8.1

pip install aiohttp 

並發http請求

如果使用requests 庫，發10個請求訪問我的博客，那么這10個請求是串行的。

import requests
import time

url = "https://www.cnblogs.com/yoyoketang/"

start_time = time.time()
for i in range(10):
    r = requests.get(url)
    print(r)
print('總耗時：', time.time()-start_time)

運行結果,總共耗時1秒左右

<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
總耗時： 1.0870192050933838

使用asyncio + aiohttp 並發請求

import asyncio
from aiohttp import ClientSession
import time


async def bai_du(url):
    print(f'啟動時間: {time.time()}')
    async with ClientSession() as session:
        async with session.get(url) as response:
            res = await response.text()
            return res


async def main():
    url = "https://www.cnblogs.com/yoyoketang/"
    task_list = []
    for i in range(10):
        task = asyncio.create_task(bai_du(url))
        task_list.append(task)
    done, pending = await asyncio.wait(task_list, timeout=None)
    # 得到執行結果
    for done_task in done:
        print(f"{time.time()} 得到執行結果 {done_task.result()}")

# asyncio.run(main())
start_time = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print("總耗時: ", time.time()-start_time)

運行結果

啟動時間: 1646028822.3952098
啟動時間: 1646028822.4161537
啟動時間: 1646028822.4171515
啟動時間: 1646028822.4171515
啟動時間: 1646028822.4171515
啟動時間: 1646028822.4171515
啟動時間: 1646028822.4181483
啟動時間: 1646028822.4181483
啟動時間: 1646028822.4181483
啟動時間: 1646028822.4181483
....
總耗時:  0.17400002479553223

從運行結果可以看到，啟動時間非常接近，也就是並發的請求，總過耗時 0.17 秒。

asyncio.run

需注意的是這里使用 asyncio.run(main()) 會報錯RuntimeError: Event loop is closed

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000238675A8F70>
Traceback (most recent call last):
  File "D:\python3.8\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\python3.8\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\python3.8\lib\asyncio\base_events.py", line 719, in call_soon
    self._check_closed()
  File "D:\python3.8\lib\asyncio\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

解決辦法，把執行方式 asyncio.run(main())改成

# asyncio.run(main())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

注意原因是asyncio.run()會自動關閉循環,並且調用_ProactorBasePipeTransport.__del__報錯, 而asyncio.run_until_complete()不會.
詳情參考https://zhuanlan.zhihu.com/p/365815189