Python 多进程踩过的一些坑


spawn fork 和 forkserver

fork 最快但是fork处理文件的时候有一些问题,如果需要利用mp.Manager().Queue()交换数据的情况,可能会发生死锁,死锁机制未找到To Do.
利用 spawn 和 mp.Queue() 结合的时候,mp.Queue()有一点bug,比如最后一个数据读不出来

Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe. This has some consequences which are a little surprising, but should not cause any practical difficulties – if they really bother you then you can instead use a queue created with a manager. After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False and get_nowait() can return without raising Queue.Empty. If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order. However, objects enqueued by the same process will always be in the expected order with respect to each other.

解决方案

在我的实践中 使用 spawn 和 mp.Manager().Queue()结合会是一个比较好的解决方案,但是运行速度较慢


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM