爬蟲實戰篇---使用Scrapy框架進行模擬登錄(包括借助阿里雲服務自動識別驗證碼)

（1）、前言

原理分析：我們編寫代碼模擬向網站發出登錄請求，也就是提交包含登錄信息的表單（用戶名、密碼等）。

實現方式：當我們想在請求數據時發送post請求，這時候需要借助Request的子類FormRequest來實現，如果想進一步在爬蟲一開始時就發送post請求，那么我們需要重寫start_request（）方法，舍棄原先的start_url()(采用get請求)

（2）、模擬登錄人人網（例子1）

1、創建項目

scrapy startproject renren---cd renren--創建爬蟲scrapy genspider spider renren.com

2、改寫settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for renren project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'renren'

SPIDER_MODULES = ['renren.spiders']
NEWSPIDER_MODULE = 'renren.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'renren (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 1
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36',


}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'renren.middlewares.RenrenSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'renren.middlewares.RenrenDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'renren.pipelines.RenrenPipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

3、改寫spider.py

# -*- coding: utf-8 -*-
import scrapy


class SpiderSpider(scrapy.Spider):
    name = 'spider'
    allowed_domains = ['renren.com']
    start_urls = ['http://renren.com/']


    def start_requests(self):
        url = 'http://www.renren.com/PLogin.do'
        data = {
            'email':'[e-mail]',
            'password':'56571218lu',
        } #構造表單數據
        request = scrapy.FormRequest(url ,formdata=data, callback=self.parse_page)
        yield request

    def parse_page(self,response):
        url2 = 'http://www.renren.com/880792860/profile'
        request = scrapy.Request(url2 ,callback=self.parse_profile)
        yield request

    def parse_profile(self,response):
        with open('baobeier.html','w',encoding='utf-8') as f: 寫入文件
            f.write(response.text)
            f.close()

4、運行爬蟲

#author: "xian"
#date: 2018/6/13
from scrapy import cmdline
cmdline.execute('scrapy crawl spider'.split())

5、效果展示（我們成功登錄並爬取了包貝兒的人人主頁）

（3）、使用阿里雲驗證碼服務自動識別驗證碼（服務地址：https://market.aliyun.com/products/57126001/cmapi014396.html#sku=yuncode=839600006）

測試服務：我們同樣使用豆瓣登錄頁面的驗證碼進行測試：

#author: "xian"
#date: 2018/6/13
from urllib import request
from base64 import b64encode
import requests

captcha_url = 'https://www.douban.com/misc/captcha?id=oL8chJoRiCTIikzwtEECZNGH:en&size=s'

request.urlretrieve(captcha_url ,'captcha.png')

recognize_url = 'http://jisuyzmsb.market.alicloudapi.com/captcha/recognize?type=e'

formdata = {}
with open('captcha.png','rb') as f:
    data = f.read()
    pic = b64encode(data)
    formdata['pic'] = pic

appcode = '614a1376aa4340b7a159d551d4eb0179'
headers = {
    'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
    'Authorization':'APPCODE ' + appcode,
}

response = requests.post(recognize_url,data = formdata,headers =headers)
print(response.json()) #返回json格式

運行效果展示：（我們借助阿里雲平台成功進行了驗證碼的自動識別）

（4）、使用阿里雲服務進行驗證碼驗證並模擬登錄豆瓣網

1、創建項目scrapy startproject douban---cd douban---創建爬蟲scrapy genspider spider doubao.com（樹形目錄如下：）

改寫settings.py

不遵循robots協議

設置請求頭

設置爬取時間間隔

改寫spider.py

# -*- coding: utf-8 -*-
import scrapy
from urllib import request
from PIL import Image #導入識別圖形的庫
from base64 import b64encode #導入b64編碼庫
import requests


class SpiderSpider(scrapy.Spider):
    name = 'spider'
    allowed_domains = ['douban.com']
    start_urls = ['https://accounts.douban.com/login'] #起始url
    login_url = 'https://accounts.douban.com/login' #登錄界面url
    profile_url = 'https://www.douban.com/people/179834288/' #個人主要url
    editsignature_url = 'https://www.douban.com/j/people/179834288/edit_signature' #編輯簽名的接口url

    def parse(self, response):
        formdata = {
            'source': 'None',
            'redir':'https://www.douban.com',
            'form_email': '[e-mail]',
            'form_password': '56571218lu',
            'remember': 'on',
            'login': '登錄',

        } #傳入部分表單數據
        captcha_url = response.css('img#captcha_image::attr(src)').get() #獲取驗證碼
        if captcha_url: #判斷是否存在驗證碼
            captcha = self.regonize_captcha(captcha_url) #識別驗證碼
            formdata['captcha-solution'] = captcha #獲取captcha-solution表單字段
            captcha_id = response.xpath('//input[@name = "captcha-id"]/@value').get() #獲取captcha_id表單字段
            formdata['captcha-id'] = captcha_id 
        yield scrapy.FormRequest(url = self.login_url,formdata=formdata,callback=self.parse_after_login) #提交表單數據


    def parse_after_login(self,response): #解析登錄頁面函數
        if response.url == 'https://www.douban.com': #判斷是否登錄成功
            yield scrapy.Request(self.profile_url,callback=self.parse_profile) #如果登錄成功向個人主頁發送請求並回調解析函數
            print('登錄成功!')
        else:
            print('登錄失敗！')

    def parse_profile(self,response): #解析個人主頁函數
        print(response.url)
        if response.url == self.profile_url: #判斷是否成功到達個人主頁
            ck = response.xpath('//input[@name = "ck"]/@value').get() #獲取ck value的值
            formdata = {
                'ck': ck,
                'signature':'積土成山，風雨興焉！',
            } #構造表單數據
            yield scrapy.FormRequest(self.editsignature_url,formdata=formdata,callback=self.parse_None) 提交表單數據，最后callback 指定回調函數，這里如果不指定回調函數默認回調parse，最后會出現登錄失敗信息
        else:
            print('進入個人中心失敗了！')

    def parse_None(self,response):
        pass


#部分為人工驗證碼識別登錄方式
    # def regonize_captcha(self,image_url):
    #     request.urlretrieve(image_url,'captcha.png')
    #     image = Image.open('captcha.png')
    #     image.show()
    #     captcha = input('請您輸入驗證碼：')
    #     return captcha

    def regonize_captcha(self, image_url): #這里使用上面的阿里雲服務識別驗證碼，參考阿里雲上提供的使用手冊即可
        captcha_url = image_url

        request.urlretrieve(captcha_url, 'captcha.png')

        recognize_url = 'http://jisuyzmsb.market.alicloudapi.com/captcha/recognize?type=e'

        formdata = {}
        with open('captcha.png', 'rb') as f:
            data = f.read()
            pic = b64encode(data)
            formdata['pic'] = pic

        appcode = '614a1376aa4340b7a159d551d4eb0179'
        headers = {
            'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
            'Authorization': 'APPCODE ' + appcode,
        }

        response = requests.post(recognize_url, data=formdata, headers=headers)
        result = response.json()
        code = result['result']['code']
        return code

最后運行爬蟲項目：

新建一個main.py方便調試

#author: "xian"
#date: 2018/6/13
from scrapy import cmdline
cmdline.execute('scrapy crawl spider'.split())

運行結果：（部分）

我們可以看到我的主頁的個性簽名已經改變為我們設置的了！（程序成功運行了！）

（5）、總結

1、在scrapy中想要發送Post請求，推薦使用scrapy.FormRequest方法，並指定表單數據

2、在爬蟲開始時發送Post請求，請重寫start_requests（）方法