Python爬蟲：scrapy爬蟲設置隨機訪問時間間隔

本文鏈接： https://blog.csdn.net/mouday/article/details/81512748

scrapy中有一個參數：DOWNLOAD_DELAY 或者 download_delay 可以設置下載延時，不過Spider類被初始化的時候就固定了，爬蟲運行過程中沒發改變。

隨機延時，可以降低被封ip的風險

代碼示例

random_delay_middleware.py

# -*- coding:utf-8 -*-

import logging
import random
import time


class RandomDelayMiddleware(object):
    def __init__(self, delay):
        self.delay = delay

    @classmethod
    def from_crawler(cls, crawler):
        delay = crawler.spider.settings.get("RANDOM_DELAY", 10)
        if not isinstance(delay, int):
            raise ValueError("RANDOM_DELAY need a int")
        return cls(delay)

    def process_request(self, request, spider):
        delay = random.randint(0, self.delay)
        logging.debug("### random delay: %s s ###" % delay)
        time.sleep(delay)

 

使用方式：

custom_settings = {
        "RANDOM_DELAY": 3,
        "DOWNLOADER_MIDDLEWARES": {
            "middlewares.random_delay_middleware.RandomDelayMiddleware": 999,
        }
    }

 

說明：
RANDOM_DELAY: 下載隨機延時范圍，[0, RANDOM_DELAY]
比如上面我設置了3秒，那么隨機延時范圍將是[0, 3]
如果設置了DOWNLOAD_DELAY 那么，總的延時應該是兩者之和：

total_delay = DOWNLOAD_DELAY + RANDOM_DELAY

 

更精確的說，應該是：

DOWNLOAD_DELAY + 0 < total_delay < DOWNLOAD_DELAY + RANDOM_DELAY