提交任務的兩種方式:
同步調用:提交完一個任務之后,就在原地等待,等待任務完完整整地運行完畢拿到結果后,再執行下一行代碼,會導致任務是串行執行
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor import time,random,os def task(name,n): print('%s%s is running' %(name,os.getpid())) time.sleep(random.randint(1,3)) return n**2 if __name__ == '__main__': # print(os.cpu_count()) #查看cpu的個數 p=ProcessPoolExecutor(4) for i in range(10): # 同步提交 res=p.submit(task,'進程pid: ',i).result() print(res) print("主") 結果： 進程pid: 10720 is running 0 進程pid: 10724 is running 1 進程pid: 5948 is running 4 進程pid: 2068 is running 9 進程pid: 10720 is running 16 進程pid: 10724 is running 25 進程pid: 5948 is running 36 進程pid: 2068 is running 49 進程pid: 10720 is running 64 進程pid: 10724 is running 81 主
異步調用:提交完一個任務之后,不在原地等待,結果直接執行下一行代碼,會導致任務是並發執行的
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor import time,random,os def task(name,n): print('%s%s is running' %(name,os.getpid())) time.sleep(random.randint(1,3)) return n**2 if __name__ == '__main__': p=ProcessPoolExecutor(4) l = [] for i in range(10): # 異步提交 future = p.submit(task, '進程pid: ', i) l.append(future) p.shutdown(wait=True) #shutdown關閉進程池入口（不能將任務放入進程池）並且在原地等待進程池內所有任務運行完畢 for future in l: print(future.result()) print("主") 結果： 進程pid: 10956 is running 進程pid: 11040 is running 進程pid: 10552 is running 進程pid: 11332 is running 進程pid: 10552 is running 進程pid: 11040 is running 進程pid: 10552 is running 進程pid: 10956 is running 進程pid: 11332 is running 進程pid: 10956 is running 0 1 4 9 16 25 36 49 64 81 主
案例：
#進程池並發爬取網站 from concurrent.futures import ProcessPoolExecutor import time,os import requests def get(url): print('%s GET %s' %(os.getpid(),url)) time.sleep(3) response=requests.get(url) if response.status_code == 200: res=response.text else: res='下載失敗' parse(res) def parse(res): time.sleep(1) print('%s 解析結果為%s' %(os.getpid(),len(res))) if __name__ == '__main__': urls=[ 'https://www.baidu.com', 'https://www.sina.com.cn', 'https://www.tmall.com', 'https://www.jd.com', 'https://www.python.org', 'https://www.openstack.org', 'https://www.baidu.com', 'https://www.baidu.com', 'https://www.baidu.com', ] p=ProcessPoolExecutor(9) l=[] start=time.time() for url in urls: future=p.submit(get,url) l.append(future) p.shutdown(wait=True) print('主',time.time()-start) 結果： 11952 GET https://www.baidu.com 11992 GET https://www.sina.com.cn 7136 GET https://www.tmall.com 11984 GET https://www.jd.com 11948 GET https://www.python.org 5952 GET https://www.openstack.org 12056 GET https://www.baidu.com 11128 GET https://www.baidu.com 11728 GET https://www.baidu.com 11952 解析結果為2443 11992 解析結果為578360 7136 解析結果為217570 11984 解析結果為90905 12056 解析結果為2443 11128 解析結果為2443 11728 解析結果為2443 11948 解析結果為48413 5952 解析結果為66284 主 10.874621868133545
from concurrent.futures import ProcessPoolExecutor import time,os import requests #並發下載，異步提交 def get(url): print('%s GET %s' %(os.getpid(),url)) time.sleep(3) response=requests.get(url) if response.status_code == 200: res=response.text else: res='下載失敗' return res def parse(future): time.sleep(1) res=future.result() print('%s 解析結果為%s' %(os.getpid(),len(res))) if __name__ == '__main__': urls=[ 'https://www.baidu.com', 'https://www.sina.com.cn', 'https://www.tmall.com', 'https://www.jd.com', 'https://www.python.org', 'https://www.openstack.org', 'https://www.baidu.com', 'https://www.baidu.com', 'https://www.baidu.com', ] p=ProcessPoolExecutor(9) start=time.time() for url in urls: future=p.submit(get,url) # 異步調用:提交完一個任務之后,不在原地等待,而是直接執行下一行代碼,會導致任務是並發執行的,,結果futrue對象會在任務運行完畢后自動傳給回調函數 future.add_done_callback(parse) #parse會在任務運行完畢后自動觸發,然后接收一個參數future對象 #add_done_callback 添加一個回調函數 p.shutdown(wait=True) print('主',time.time()-start) print('主',os.getpid()) 結果： 10056 GET https://www.baidu.com 11584 GET https://www.sina.com.cn 3624 GET https://www.tmall.com 12116 GET https://www.jd.com 10072 GET https://www.python.org 9784 GET https://www.openstack.org 11924 GET https://www.baidu.com 10272 GET https://www.baidu.com 3084 GET https://www.baidu.com 11744 解析結果為2443 11744 解析結果為217570 11744 解析結果為90981 11744 解析結果為2443 11744 解析結果為2443 11744 解析結果為2443 11744 解析結果為66304 11744 解析結果為578616 11744 解析結果為48181 主 24.2570583820343 主 11744
from concurrent.futures import ThreadPoolExecutor from threading import current_thread import time,requests def get(url): print('%s GET %s' %(current_thread().name,url)) time.sleep(3) response=requests.get(url) if response.status_code == 200: res=response.text else: res='下載失敗' return res def parse(future): time.sleep(1) res=future.result() print('%s 解析結果為%s' %(current_thread().name,len(res))) if __name__ == '__main__': urls=[ 'https://www.baidu.com', 'https://www.sina.com.cn', 'https://www.tmall.com', 'https://www.jd.com', 'https://www.python.org', 'https://www.openstack.org', 'https://www.baidu.com', 'https://www.baidu.com', 'https://www.baidu.com', ] p=ThreadPoolExecutor(4) for url in urls: future=p.submit(get,url) future.add_done_callback(parse) p.shutdown(wait=True) print('主',current_thread().name) #阻塞：遇到io行為,進入阻塞狀態，等待一會。 結果： ThreadPoolExecutor-0_0 GET https://www.baidu.com ThreadPoolExecutor-0_1 GET https://www.sina.com.cn ThreadPoolExecutor-0_2 GET https://www.tmall.com ThreadPoolExecutor-0_3 GET https://www.jd.com ThreadPoolExecutor-0_3 解析結果為90936 ThreadPoolExecutor-0_3 GET https://www.python.org ThreadPoolExecutor-0_0 解析結果為2443 ThreadPoolExecutor-0_0 GET https://www.openstack.org ThreadPoolExecutor-0_2 解析結果為217570 ThreadPoolExecutor-0_2 GET https://www.baidu.com ThreadPoolExecutor-0_1 解析結果為578616 ThreadPoolExecutor-0_1 GET https://www.baidu.com ThreadPoolExecutor-0_2 解析結果為2443 ThreadPoolExecutor-0_2 GET https://www.baidu.com ThreadPoolExecutor-0_1 解析結果為2443 ThreadPoolExecutor-0_0 解析結果為66304 ThreadPoolExecutor-0_3 解析結果為48181 ThreadPoolExecutor-0_2 解析結果為2443 主 MainThread