- 微信
- 微博
  
  分享文章到微博
- 复制链接
  
  复制链接到剪贴板

多线程 VS 多进程（三）

ruochen 发表于 2021/03/29 00:50:44 2021/03/29

【摘要】线程替代方案 subprocess 完全跳过线程，使用进程是派生进程的主要替代方案python2.4后引入 multiprocessiong 使用threading接口派生，使用子进程允许为多核或者多cpu派生进程，接口跟threading非常相似python2.6后引入 concurrent.futures 新的异步执行模块任务级别的操作python3.2后引入 ...

线程替代方案

subprocess
- 完全跳过线程，使用进程
- 是派生进程的主要替代方案
- python2.4后引入
multiprocessiong
- 使用threading接口派生，使用子进程
- 允许为多核或者多cpu派生进程，接口跟threading非常相似
- python2.6后引入
concurrent.futures
- 新的异步执行模块
- 任务级别的操作
- python3.2后引入

多进程

进程间通讯(InterprocessCommunication, IPC)
进程之间无任何共享状态

进程的创建

直接生成Process实例对象，案例19

import multiprocessing
from time import sleep, ctime

def clock(interval): while True: print("The time is %s" % ctime()) sleep(interval)

if __name__ == '__main__': p = multiprocessing.Process(target=clock, args=(5, )) p.start() while True: print("sleeping......") sleep(1)

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15

  sleeping......
  The time is Tue Aug 13 19:47:41 2019
  sleeping......
  sleeping......
  sleeping......
  sleeping......
  sleeping......
  The time is Tue Aug 13 19:47:46 2019
  sleeping......
  sleeping......
  sleeping......
  sleeping......
  ...

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13

派生子类，案例20

import multiprocessing
from time import sleep, ctime

class ClockProcess(multiprocessing.Process): ''' 两个函数比较重要 1. init构造函数 2. run ''' def __init__(self, interval): super(ClockProcess, self).__init__() self.interval = interval def run(self): while True: print("The time is %s" % ctime()) sleep(self.interval)

if __name__ == '__main__': p = ClockProcess(3) p.start()

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20

  The time is Tue Aug 13 19:48:49 2019
  The time is Tue Aug 13 19:48:52 2019
  The time is Tue Aug 13 19:48:55 2019
  The time is Tue Aug 13 19:48:58 2019
  The time is Tue Aug 13 19:49:01 2019
  The time is Tue Aug 13 19:49:04 2019
  ...

      
     
      1
      2
      3
      4
      5
      6
      7

在os中查看pid，ppid以及他们的关系

案例21

from multiprocessing import Process
import os

def info(title): print(title) print("module name:", __name__) # 得到父亲进程的id print("parent process:", os.getppid()) # 得到本身进程的id print("process id:", os.getpid())

def f(name): info('function f') print('hello', name)

if __name__ == '__main__': info('main line') p = Process(target=f, args=('bob', )) p.start() p.join()

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20

  main line
  module name: __main__
  parent process: 11900
  process id: 18832
  function f
  module name: __mp_main__
  parent process: 18832
  process id: 20868
  hello bob

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9

生产者消费者模型

JoinableQueue

案例22

import multiprocessing
from time import ctime

def consumer(input_q): print("Into consumer:", ctime) while True: # 处理项 item = input_q.get() print('pull', item, 'out of q') input_q.task_done()  # 发出信号通知任务完成 print("Out of consumer:", ctime()) ## 此句未执行，因为q.join()收集到四个task_done()信号后，主进程启动，未等到print此句完成，程序就结束了

def producer(sequence, output_q): print("Into procuder:", ctime()) for item in sequence: output_q.put(item) print('put', item, 'into q') print('Out of procuder', ctime())

# 建立进程
if __name__ == '__main__': q = multiprocessing.JoinableQueue() # 进行消费者进程 cons_p = multiprocessing.Process(target=consumer, args=(q, )) cons_p.daemon = True cons_p.start() # 生产多个项，sequence代表要发送给消费者的项序列 # 在实践中，这可能是生成器的输出或通过一些其他地方生产出来 sequence = [1, 2, 3, 4] producer(sequence, q) # 等待所有项被处理 q.join()

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33

  Into procuder: Tue Aug 13 19:50:38 2019
  put 1 into q
  put 2 into q
  put 3 into q
  put 4 into q
  Out of procuder Tue Aug 13 19:50:38 2019
  Into consumer: <built-in function ctime>
  pull 1 out of q
  pull 2 out of q
  pull 3 out of q
  pull 4 out of q

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

队列中哨兵的使用，案例23

import multiprocessing
from time import ctime

# 设置哨兵问题
def consumer(input_q): print("Into consumer:", ctime()) while True: item = input_q.get() if item is None: break print("pull", item, "out of q") print ("Out of consumer:", ctime()) ## 此句执行完成，再转入主进程


def producer(sequence, output_q): print ("Into procuder:", ctime()) for item in sequence: output_q.put(item) print ("put", item, "into q") print ("Out of procuder:", ctime())

if __name__ == '__main__': q = multiprocessing.Queue() cons_p = multiprocessing.Process(target = consumer, args = (q,)) cons_p.start() sequence = [1,2,3,4] producer(sequence, q) q.put(None) cons_p.join()

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31

  Into procuder: Tue Aug 13 19:51:23 2019
  put 1 into q
  put 2 into q
  put 3 into q
  put 4 into q
  Out of procuder: Tue Aug 13 19:51:23 2019
  Into consumer: Tue Aug 13 19:51:24 2019
  pull 1 out of q
  pull 2 out of q
  pull 3 out of q
  pull 4 out of q
  Out of consumer: Tue Aug 13 19:51:24 2019

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

哨兵的改进，案例24

import multiprocessing
from time import ctime

def consumer(input_q): print ("Into consumer:", ctime()) while True: item = input_q.get() if item is None: break print("pull", item, "out of q") print ("Out of consumer:", ctime())

def producer(sequence, output_q): for item in sequence: print ("Into procuder:", ctime()) output_q.put(item) print ("Out of procuder:", ctime())

if __name__ == '__main__': q = multiprocessing.Queue() cons_p1 = multiprocessing.Process (target = consumer, args = (q,)) cons_p1.start() cons_p2 = multiprocessing.Process (target = consumer, args = (q,)) cons_p2.start() sequence = [1,2,3,4] producer(sequence, q) q.put(None) q.put(None) cons_p1.join() cons_p2.join()

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34

  Into procuder: Tue Aug 13 19:52:08 2019
  Out of procuder: Tue Aug 13 19:52:08 2019
  Into procuder: Tue Aug 13 19:52:08 2019
  Out of procuder: Tue Aug 13 19:52:08 2019
  Into procuder: Tue Aug 13 19:52:08 2019
  Out of procuder: Tue Aug 13 19:52:08 2019
  Into procuder: Tue Aug 13 19:52:08 2019
  Out of procuder: Tue Aug 13 19:52:08 2019
  Into consumer: Tue Aug 13 19:52:08 2019
  pull 1 out of q
  pull 2 out of q
  pull 3 out of q
  pull 4 out of q
  Out of consumer: Tue Aug 13 19:52:08 2019
  Into consumer: Tue Aug 13 19:52:08 2019
  Out of consumer: Tue Aug 13 19:52:08 2019

      
     
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16

文章来源: ruochen.blog.csdn.net，作者：若尘，版权归原作者所有，如需转载，请联系作者。

原文链接：ruochen.blog.csdn.net/article/details/99467129

点赞
收藏
关注作者

0/1000

抱歉，系统识别当前为高风险访问，暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称，即可参与社区互动！

*长度不超过10个汉字或20个英文字符，设置后3个月内不可修改。

确认取消

加入云驻计划，成为创作者

华为云周边好礼
免费体验产品
特殊身份标识
线下官方门票
内部专家零距离
与10000+优质创作者共同成长

立即加入

多线程 VS 多进程（三）

线程替代方案

多进程

全部回复

设置昵称

关于作者

目录

加入云驻计划，成为创作者

多线程 VS 多进程（三）

线程替代方案

多进程

全部回复

设置昵称

关于作者

目录

加入云驻计划，成为创作者

推荐阅读

相关产品