python multiprocessing.Pool - PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed

示例1：

import multiprocessing

def run(msg):
    print '%s, it works!' % msg

def start_process():
    print 'Starting',multiprocessing.current_process().name

if __name__=='__main__':
    pool = multiprocessing.Pool(processes=2, initializer=start_process,)
    pool.apply_async(run('congbo'))
    pool.close()
    pool.join()

運行結果：

Starting PoolWorker-1
Starting PoolWorker-2
congbo, it works!

pool.apply_async(run('congbo')) 這種使用方式也行？見示例2。

 

示例2：

import multiprocessing

def run(msg):
    print '%s, it works!' % msg
    while True: pass

def start_process():
    print 'Starting',multiprocessing.current_process().name

if __name__=='__main__':
    pool = multiprocessing.Pool(processes=2, initializer=start_process,)
    pool.apply_async(run('congbo'))

　  print "It's my turn!"

    pool.close()
    pool.join()

運行結果：

Starting PoolWorker-1
Starting PoolWorker-2
congbo, it works!
...
...

沒有打印出“It's my turn!”，程序沒有切換到主進程接着執行，子進程死循環。

說明這種使用方式是不對的，只能像 apply_async 原型那樣使用。見示例3。

 

示例3：

import multiprocessing

def run(msg):
    print '%s, it works!' % msg

def start_process():
    print 'Starting',multiprocessing.current_process().name

if __name__=='__main__':
    pool = multiprocessing.Pool(processes=2, initializer=start_process,)
    pool.apply_async(run, args=('congbo',))
    pool.close()
    pool.join()

運行結果：

Starting PoolWorker-1
Starting PoolWorker-2
congbo, it works!

tips：apply_async 中 args 應為元組，故單個參數后面需加逗號。如果參數錯誤，不成功執行，不報異常！

 

示例4：

import multiprocessing

class Worker:
    def run(self, msg):
        print '%s, it works!' % msg

def start_process():
    print 'Starting',multiprocessing.current_process().name

if __name__=='__main__':
    pool = multiprocessing.Pool(processes=2, initializer=start_process,)
    pool.apply_async(Worker().run, args=('congbo',))
    pool.close()
    pool.join()

運行結果：

Starting PoolWorker-1
Starting PoolWorker-2
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

 

示例5：

import multiprocessing

class Worker:
    def run(self, msg):
        print '%s, it works!' % msg

def wrap(worker, msg): worker.run(msg) def start_process():
    print 'Starting',multiprocessing.current_process().name

if __name__=='__main__':
    pool = multiprocessing.Pool(processes=2, initializer=start_process,)
 pool.apply_async(wrap, args=(Worker(), 'congbo'))
    pool.close()
    pool.join()

不直接傳入類實例的方法，用另一個函數包裝一下，將類的實例作為參數傳入即可。

為什么要這樣？有沒有更好的辦法？ 

 

參考：http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma

 

原文：http://www.cnblogs.com/congbo/archive/2012/08/23/2652322.html