python並行多個線程和進程

python並行多個線程和進程

工作站配置了20核CPU，平時運行程序只讓一個CPU進行運轉，浪費了很多時間。下面介紹同時啟動多個CPU運行一個程序的方法：

一個進程（CPU）包含多個線程，線程並行的python庫為threading，進程並行的庫為multiprocessing。

父進程（主函數）運行結束后，如果子進程（子函數）還沒有運行結束，需要使用join方法讓父進程等待全部結束后再結束。

下面程序中，父進程每隔30s檢查子進程的存留情況，等全部子進程運行結束后通過join結束整個程序。

並行多個線程的例子

from threading import Thread #並行線程
from threading import active_count
from time import time, sleep

def process(times,data):
    print("now is",times,"th process")
    sleep(15)
    f   = open(times,"w")
    f.write(data)
    f.close()
    print("OK!",times,"is finished")

if __name__ == '__main__':
    tstart    = time()
    threads   = []
    for i in range(5): #一個進程里並行5個線程
        name='testdata{:02d}.txt'.format(i)
        na  ='{:6.4f}'.format(i)
        print(name)
        t = Thread(target=process,args=(name,na))
        threads.append(t)
        t.setDaemon(True)
        t.start()
        sleep(2)
    th_no = active_count()
    while (th_no>1): #主線程監視間隔30s
        sleep(30)
        th_no = active_count()
        trun = time()
        print("now there are",th_no,"threads, time=",trun-tstart,"sec")
    for t in threads:
        t.join()
    tend      = time()
    print("sum up cost",tend-tstart,"sec")

 

並行多個進程的例子

CPU占用較低的小程序，並行多線程效率更高，CPU占用100%的大程序，並行多個進程效率更高，並行多個線程反而會使效率降低。

from multiprocessing import Process #不同點1
from multiprocessing import active_children #不同點2
from time import time, sleep

def process(times,data):
    print("now is",times,"th process")
    sleep(15)
    f   = open(times,"w")
    f.write(data)
    f.close()
    print("OK!",times,"is finished")

if __name__ == '__main__':
    tstart    = time()
    threads   = []
    for i in range(10): #數量要小於CPU核數
        name='testdata{:02d}.txt'.format(i)
        na  ='{:6.4f}'.format(i)
        print(name)
        t = Process(target=process,args=(name,na)) #與threading.Thread使用方法相似
        threads.append(t)
        t.start()
        sleep(1.0)
    pro_no = len(active_children()) #每隔30s父進程監視子進程留存數量
    while (pro_no>1):
        sleep(30)
        pro_no = len(active_children())
        trun     = time()
        print("Running CPU NO=",pro_no," Running time=",trun-tstart,"sec")
    for t in threads: #等待子進程全部結束開始join
        t.join()
    tend      = time()
    print("sum up cost",tend-tstart,"sec")

參考官方文檔：

並行多個進程 https://docs.python.org/zh-cn/3/library/multiprocessing.html

並行多個線程 https://docs.python.org/zh-cn/3/library/threading.html