原理：

1个进程 -> 多个子进程 -> scrapy进程
  

 

  
1
 

代码示例

将以下代码文件放入scrapy项目中任意位置即可

# -*- coding: utf-8 -*-

# @File : run_spider.py
# @Date : 2018-08-06
# @Author  : Peng Shiyu

from multiprocessing import Process
from scrapy import cmdline
import time
import logging

# 配置参数即可, 爬虫名称，运行频率
confs = [ { "spider_name": "hexun_pdf", "frequency": 2, },
]


def start_spider(spider_name, frequency): args = ["scrapy", "crawl", spider_name] while True: start = time.time() p = Process(target=cmdline.execute, args=(args,)) p.start() p.join() logging.debug("### use time: %s" % (time.time() - start)) time.sleep(frequency)


if __name__ == '__main__': for conf in confs: process = Process(target=start_spider, args=(conf["spider_name"], conf["frequency"])) process.start() time.sleep(10)

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
  
8
  
9
  
10
  
11
  
12
  
13
  
14
  
15
  
16
  
17
  
18
  
19
  
20
  
21
  
22
  
23
  
24
  
25
  
26
  
27
  
28
  
29
  
30
  
31
  
32
  
33
  
34
  
35
  
36
  
37
  
38
 

文章来源: pengshiyu.blog.csdn.net，作者：彭世瑜，版权归原作者所有，如需转载，请联系作者。

原文链接：pengshiyu.blog.csdn.net/article/details/81514569