Scrapy中間件user-agent和ip代理使用

一、定義實現隨機User-Agent的下載中間件

1.在middlewares.py中完善代碼

import random
from Tencent.settings import USER_AGENTS_LIST # 注意導入路徑,請忽視pycharm的錯誤提示

class UserAgentMiddleware(object):
    def process_request(self, request, spider):
        user_agent = random.choice(USER_AGENTS_LIST)
        request.headers['User-Agent'] = user_agent
        # 不寫return

class CheckUA:
    def process_response(self,request,response,spider):
        print(request.headers['User-Agent'])
        return response # 不能少！

2.在settings中設置開啟自定義的下載中間件，設置方法同管道

DOWNLOADER_MIDDLEWARES = {
   'Tencent.middlewares.UserAgentMiddleware': 543, # 543是權重值
   'Tencent.middlewares.CheckUA': 600, # 先執行543權重的中間件，再執行600的中間件
}

 

3.在settings中添加UA的列表

USER_AGENTS_LIST = [
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
    "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
    "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5"
]

 

二、代理ip的使用

1.在middlewares.py中完善代碼

class RandomProxy(object):

    def process_request(self, request, spider):

        proxy = random.choice(PROXY_LIST)
        print(proxy)

        if 'user_passwd' in proxy:
            # 對賬號密碼進行編碼，基礎認證,python3中需要是bytes類型的數據才能編碼
            b64_up = base64.b64encode(proxy['user_passwd'].encode())

            # 進行代理認證
            request.headers['Proxy-Authorization'] = 'Basic ' + b64_up.decode()

            # 設置代理
            request.meta['proxy'] = proxy['ip_port']
        else:
            #設置代理
            request.meta['proxy'] = proxy['ip_port']

2.檢測代理ip是否可用

在使用了代理ip的情況下可以在下載中間件的process_response()方法中處理代理ip的使用情況，如果該代理ip不能使用可以替換其他代理ip

class ProxyMiddleware(object):
    ......
    def process_response(self, request, response, spider):
        if response.status != '200':
            request.dont_filter = True # 重新發送的請求對象能夠再次進入隊列
            return requst

3.在settings中添加代理ip的列表

PROXY_LIST = [
    {"ip_port": "139.199.121.163:16818", "user_passwd": "user:password"},#收費代理
    # {"ip_port": "114.234.81.72:9000"} # 免費代理
]

 

三. 在中間件中使用selenium

以github登陸為例

1. 完成爬蟲代碼

import scrapy

class Login4Spider(scrapy.Spider):
    name = 'login4'
    allowed_domains = ['github.com']
    start_urls = ['https://github.com/returnes'] # 直接對驗證的url發送請求

    def parse(self, response):
        with open('check.html', 'w') as f:
            f.write(response.body.decode())

2.在middlewares.py中使用selenium獲取cookie信息

import time
from selenium import webdriver


def getCookies():
    # 使用selenium模擬登陸，獲取並返回cookie
    username = input('輸入github賬號:')
    password = input('輸入github密碼:')
    options = webdriver.ChromeOptions()
    options.add_argument('--headless')
    options.add_argument('--disable-gpu')
    driver = webdriver.Chrome('/home/worker/Desktop/driver/chromedriver',
                              chrome_options=options)
    driver.get('https://github.com/login')
    time.sleep(1)
    driver.find_element_by_xpath('//*[@id="login_field"]').send_keys(username)
    time.sleep(1)
    driver.find_element_by_xpath('//*[@id="password"]').send_keys(password)
    time.sleep(1)
    driver.find_element_by_xpath('//*[@id="login"]/form/div[3]/input[3]').click()
    time.sleep(2)
    cookies_dict = {cookie['name']: cookie['value'] for cookie in driver.get_cookies()}
    driver.quit()
    return cookies_dict

class LoginDownloaderMiddleware(object):

    def process_request(self, request, spider):
        cookies_dict = getCookies()
        print(cookies_dict)
        request.cookies = cookies_dict # 對請求對象的cookies屬性進行替換

 

3.在middlewares.py中使用selenium獲取指定頁面渲染后的html源碼

class SelMiddleWare(object):

    def process_request(self, request, spider):

        url = request.url
        # 過濾需要渲染的請求對象
        if 'daydata' in url:

            driver = webdriver.Chrome()

            driver.get(url)
            time.sleep(3)

            data = driver.page_source
            driver.close()

            res = HtmlResponse(
                url=url,
                body=data,
                encoding='utf-8',
                request=request
            )

            return res