scrapy 爬取鏈家網站房價爬蟲 多頁爬取

直接上代碼，順便在這里記錄，時間2190906.

剛開始爬貝殼網的，發現有反爬蟲，我也不會繞，換了鏈家網，原來中文也可以做變量。

spider.py

# -*- coding: utf-8 -*-
import scrapy

from beike.items import BeikeItem

class BeikewSpider(scrapy.Spider):
    name = 'beikew'
    allowed_domains = ['lianjia.com']
    start_urls = ['https://su.lianjia.com/ershoufang/']
    page = 1


    def parse(self, response):
        li_list = response.xpath('//*[@id="content"]/div[1]/ul/li')
        for li in li_list:
            item = BeikeItem()
            name = li.xpath('./div[1]/div[1]/a/text()').extract_first()
            單價 = li.xpath('./div[1]/div[6]/div[2]/span/text()').extract_first()
            totalprice = li.xpath('./div[1]/div[6]/div[1]/span/text()').extract_first()
            xiaoqu = li.xpath('./div[1]/div[2]/div/a/text()').extract_first()
            local = li.xpath('./div[1]/div[3]/div/a/text()').extract_first()
            item['name'] = name
            item['單價'] = 單價 #在這里試試中文的，才知道原來中文也可以做變量
            item['totalprice'] = totalprice
            item['xiaoqu'] = xiaoqu
            item['local'] = local
            yield item

        if self.page <= 50:#這里爬取了50頁數據，可以隨意更改
            self.page += 1
            url_new = str(self.page)
            new_page_url = 'https://su.lianjia.com/ershoufang/pg' + url_new
            yield scrapy.Request(url = new_page_url, callback = (self.parse))

item.py

import scrapy

class BeikeItem(scrapy.Item):
    xiaoqu = scrapy.Field()
    name = scrapy.Field()
    單價 = scrapy.Field()
    totalprice = scrapy.Field()
    local = scrapy.Field()

settings.py

BOT_NAME = 'beike' #這些代碼在settings里啟用或者添加的。
SPIDER_MODULES = ['beike.spiders']
NEWSPIDER_MODULE = 'beike.spiders'
FEED_EXPORT_ENCODING ='utf-8'
FEED_EXPORT_ENCODING = 'gb18030'
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 1

只用到了3個y文件，其他的都是命令生成的，保持默認。

執行結果：