线程替代方案

• subprocess
  • 完全跳过线程，使用进程
  • 是派生进程的主要替代方案
  • python2.4后引入
• multiprocessiong
  • 使用threading接口派生，使用子进程
  • 允许为多核或者多cpu派生进程，接口跟threading非常相似
  • python2.6后引入
• concurrent.futures
  • 新的异步执行模块
  • 任务级别的操作
  • python3.2后引入

多进程

• 进程间通讯(InterprocessCommunication, IPC)

• 进程之间无任何共享状态

• 进程的创建

  • 直接生成Process实例对象，案例19

    import multiprocessing
    from time import sleep, ctime
    
    def clock(interval):
        while True:
            print("The time is %s" % ctime())
            sleep(interval)
    
    if __name__ == '__main__':
        p = multiprocessing.Process(target=clock, args=(5, ))
        p.start()
    
        while True:
            print("sleeping......")
            sleep(1)
    

      sleeping......
      The time is Tue Aug 13 19:47:41 2019
      sleeping......
      sleeping......
      sleeping......
      sleeping......
      sleeping......
      The time is Tue Aug 13 19:47:46 2019
      sleeping......
      sleeping......
      sleeping......
      sleeping......
      ...
    

  • 派生子类，案例20

    import multiprocessing
    from time import sleep, ctime
    
    class ClockProcess(multiprocessing.Process):
        '''
        两个函数比较重要
        1. init构造函数
        2. run
        '''
        def __init__(self, interval):
            super(ClockProcess, self).__init__()
            self.interval = interval
        def run(self):
            while True:
                print("The time is %s" % ctime())
                sleep(self.interval)
    
    if __name__ == '__main__':
        p = ClockProcess(3)
        p.start()
    

      The time is Tue Aug 13 19:48:49 2019
      The time is Tue Aug 13 19:48:52 2019
      The time is Tue Aug 13 19:48:55 2019
      The time is Tue Aug 13 19:48:58 2019
      The time is Tue Aug 13 19:49:01 2019
      The time is Tue Aug 13 19:49:04 2019
      ...
    

• 在os中查看pid，ppid以及他们的关系

  • 案例21

    from multiprocessing import Process
    import os
    
    def info(title):
        print(title)
        print("module name:", __name__)
        # 得到父亲进程的id
        print("parent process:", os.getppid())
        # 得到本身进程的id
        print("process id:", os.getpid())
    
    def f(name):
        info('function f')
        print('hello', name)
    
    if __name__ == '__main__':
        info('main line')
        p = Process(target=f, args=('bob', ))
        p.start()
        p.join()
    

      main line
      module name: __main__
      parent process: 11900
      process id: 18832
      function f
      module name: __mp_main__
      parent process: 18832
      process id: 20868
      hello bob
    

• 生产者消费者模型

  • JoinableQueue
  • 案例22

    import multiprocessing
    from time import ctime
    
    def consumer(input_q):
        print("Into consumer:", ctime)
        while True:
            # 处理项
            item = input_q.get()
            print('pull', item, 'out of q')
            input_q.task_done()  # 发出信号通知任务完成
        print("Out of consumer:", ctime()) ## 此句未执行，因为q.join()收集到四个task_done()信号后，主进程启动，未等到print此句完成，程序就结束了
    
    def producer(sequence, output_q):
        print("Into procuder:", ctime())
        for item in sequence:
            output_q.put(item)
            print('put', item, 'into q')
        print('Out of procuder', ctime())
    
    # 建立进程
    if __name__ == '__main__':
        q = multiprocessing.JoinableQueue()
        # 进行消费者进程
        cons_p = multiprocessing.Process(target=consumer, args=(q, ))
        cons_p.daemon = True
        cons_p.start()
    
        # 生产多个项，sequence代表要发送给消费者的项序列
        # 在实践中，这可能是生成器的输出或通过一些其他地方生产出来
        sequence = [1, 2, 3, 4]
        producer(sequence, q)
        # 等待所有项被处理
        q.join()
    

      Into procuder: Tue Aug 13 19:50:38 2019
      put 1 into q
      put 2 into q
      put 3 into q
      put 4 into q
      Out of procuder Tue Aug 13 19:50:38 2019
      Into consumer: <built-in function ctime>
      pull 1 out of q
      pull 2 out of q
      pull 3 out of q
      pull 4 out of q
    

  • 队列中哨兵的使用，案例23

    import multiprocessing
    from time import ctime
    
    # 设置哨兵问题
    def consumer(input_q):
        print("Into consumer:", ctime())
        while True:
            item = input_q.get()
            if item is None:
                break
            print("pull", item, "out of q")
        print ("Out of consumer:", ctime()) ## 此句执行完成，再转入主进程
    
    
    def producer(sequence, output_q):
        print ("Into procuder:", ctime())
        for item in sequence:
            output_q.put(item)
            print ("put", item, "into q")
        print ("Out of procuder:", ctime())
    
    if __name__ == '__main__':
        q = multiprocessing.Queue()
        cons_p = multiprocessing.Process(target = consumer, args = (q,))
        cons_p.start()
    
        sequence = [1,2,3,4]
        producer(sequence, q)
    
        q.put(None)
        cons_p.join()
    

      Into procuder: Tue Aug 13 19:51:23 2019
      put 1 into q
      put 2 into q
      put 3 into q
      put 4 into q
      Out of procuder: Tue Aug 13 19:51:23 2019
      Into consumer: Tue Aug 13 19:51:24 2019
      pull 1 out of q
      pull 2 out of q
      pull 3 out of q
      pull 4 out of q
      Out of consumer: Tue Aug 13 19:51:24 2019
    

  • 哨兵的改进，案例24

    import multiprocessing
    from time import ctime
    
    def consumer(input_q):
        print ("Into consumer:", ctime())
        while True:
            item = input_q.get()
            if item is None:
                break
            print("pull", item, "out of q")
        print ("Out of consumer:", ctime())
    
    def producer(sequence, output_q):
        for item in sequence:
            print ("Into procuder:", ctime())
            output_q.put(item)
            print ("Out of procuder:", ctime())
    
    if __name__ == '__main__':
        q = multiprocessing.Queue()
        cons_p1 = multiprocessing.Process (target = consumer, args = (q,))
        cons_p1.start()
    
        cons_p2 = multiprocessing.Process (target = consumer, args = (q,))
        cons_p2.start()
    
        sequence = [1,2,3,4]
        producer(sequence, q)
    
        q.put(None)
        q.put(None)
    
        cons_p1.join()
        cons_p2.join()
    

      Into procuder: Tue Aug 13 19:52:08 2019
      Out of procuder: Tue Aug 13 19:52:08 2019
      Into procuder: Tue Aug 13 19:52:08 2019
      Out of procuder: Tue Aug 13 19:52:08 2019
      Into procuder: Tue Aug 13 19:52:08 2019
      Out of procuder: Tue Aug 13 19:52:08 2019
      Into procuder: Tue Aug 13 19:52:08 2019
      Out of procuder: Tue Aug 13 19:52:08 2019
      Into consumer: Tue Aug 13 19:52:08 2019
      pull 1 out of q
      pull 2 out of q
      pull 3 out of q
      pull 4 out of q
      Out of consumer: Tue Aug 13 19:52:08 2019
      Into consumer: Tue Aug 13 19:52:08 2019
      Out of consumer: Tue Aug 13 19:52:08 2019