Scrapy添加代理验证

middlewares.py

from w3lib.http import basic_auth_header

class CustomProxyMiddleware(object): def process_request(self, request, spider): request.meta['proxy'] = "https://<PROXY_IP_OR_URL>:<PROXY_PORT>" request.headers['Proxy-Authorization'] = basic_auth_header( '<PROXY_USERNAME>', '<PROXY_PASSWORD>')

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
 

settings.py

DOWNLOADER_MIDDLEWARES = { '<PROJECT_NAME>.middlewares.CustomProxyMiddleware': 350, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

  

 

  
1
  
2
  
3
  
4
 

问题

1、如果代理验证设置不对，状态码会返回407
407 Proxy Authentication Required

刚开始采用以下格式配置，发现部分请求可以发送，不过会有一个重试，部分请求直接报错

request.meta['proxy'] = "https://<PROXY_USERNAME>:<PROXY_PASSWORD>@<PROXY_IP_OR_URL>:<PROXY_PORT>"

  

 

  
1
 

正确的设置是在请求头中设置 Proxy-Authorization

参考

1. Using a custom proxy in a Scrapy spider
2. Proxy-Authorization

文章来源: pengshiyu.blog.csdn.net，作者：彭世瑜，版权归原作者所有，如需转载，请联系作者。

原文链接：pengshiyu.blog.csdn.net/article/details/106438932