用scrapy爬取京東商城的商品信息

軟件環境：

gevent (1.2.2)
greenlet (0.4.12)
lxml (4.1.1)
pymongo (3.6.0)
pyOpenSSL (17.5.0)
requests (2.18.4)
Scrapy (1.5.0)
SQLAlchemy (1.2.0)
Twisted (17.9.0)
wheel (0.30.0)

 

1.創建爬蟲項目

2創建京東網站爬蟲. 進入爬蟲項目目錄，執行命令：

scrapy genspider jd www.jd.com

會在spiders目錄下會創建和你起的名字一樣的py文件：jd.py，這個文件就是用來寫你爬蟲的請求和響應邏輯的

3. jd.py文件配置

分析的amazon網站的url規則：

https://search.jd.com/Search?

以防關鍵字是中文，所以要做urlencode

        1.首先寫一個start_request函數，用來發送第一次請求，並把請求結果發給回調函數parse_index，同時把reponse返回值傳遞給回調函數,response類型<class                 'scrapy.http.response.html.HtmlResponse'>

    def start_requests(self):
        # https://www.amazon.cn/s/ref=nb_sb_ss_i_1_6?field-keywords=macbook+pro
        # 拼接處符合條件的URL地址
        # 並通過scrapy.Requst封裝請求，並調用回調函數parse_index處理,同時會把response傳遞給回調函數
        url = 'https://search.jd.com/Search?'
        # 拼接的時候field-keywords后面是不加等號的
        url += urlencode({"keyword": self.keyword, "enc": "utf-8"})
        yield scrapy.Request(url,
                             callback=self.parse_index,
                             )

        2.parse_index從reponse中獲取所有的產品詳情頁url地址，並遍歷所有的url地址發送request請求，同時調用回調函數parse_detail去處理結果

def parse_detail(self, response):
    """
    接收parse_index的回調，並接收response返回值，並解析response
    :param response:
    :return:
    """
    jd_url = response.url
    sku = jd_url.split('/')[-1].strip(".html")
    # price信息是通過jsonp獲取，可以通過開發者工具中的script找到它的請求地址
    price_url = "https://p.3.cn/prices/mgets?skuIds=J_" + sku
    response_price = requests.get(price_url)
    # extraParam={"originid":"1"}  skuIds=J_3726834
    # 這里是物流信息的請求地址，也是通過jsonp發送的，但目前沒有找到它的參數怎么獲取的，這個是一個固定的參數，如果有哪位大佬知道，好望指教
    express_url = "https://c0.3.cn/stock?skuId=3726834&area=1_72_4137_0&cat=9987,653,655&extraParam={%22originid%22:%221%22}"
    response_express = requests.get(express_url)
    response_express = json.loads(response_express.text)['stock']['serviceInfo'].split('>')[1].split('<')[0]
    title = response.xpath('//*[@class="sku-name"]/text()').extract_first().strip()
    price = json.loads(response_price.text)[0]['p']
    delivery_method = response_express
    # # 把需要的數據保存到Item中，用來會后續儲存做准備
    item = AmazonItem()
    item['title'] = title
    item['price'] = price
    item['delivery_method'] = delivery_method

    # 最后返回item，如果返回的數據類型是item，engine會檢測到並把返回值發給pipelines處理
    return item

4. item.py配置

import scrapy


class JdItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # amazome Item
    title = scrapy.Field()
    price = scrapy.Field()
    delivery_method = scrapy.Field()

5. pipelines.py配置

from pymongo import MongoClient


class MongoPipeline(object):
    """
    用來保存數據到MongoDB的pipeline
    """

    def __init__(self, db, collection, host, port, user, pwd):
        """
        連接數據庫
        :param db: databaes name
        :param collection: table name
        :param host: the ip for server
        :param port: thr port for server
        :param user: the username for login
        :param pwd: the password for login
        """
        self.db = db
        self.collection = collection
        self.host = host
        self.port = port
        self.user = user
        self.pwd = pwd

    @classmethod
    def from_crawler(cls, crawler):
        """
        this classmethod is used for to get the configuration from settings
        :param crwaler:
        :return:
        """
        db = crawler.settings.get('DB')
        collection = crawler.settings.get('COLLECTION')
        host = crawler.settings.get('HOST')
        port = crawler.settings.get('PORT')
        user = crawler.settings.get('USER')
        pwd = crawler.settings.get('PWD')

        return cls(db, collection, host, port, user, pwd)

    def open_spider(self, spider):
        """
        run once time when the spider is starting
        :param spider:
        :return:
        """
        # 連接數據庫
        self.client = MongoClient("mongodb://%s:%s@%s:%s" % (
            self.user,
            self.pwd,
            self.host,
            self.port
        ))

    def process_item(self, item, spider):
        """
        storage the data into database
        :param item:
        :param spider:
        :return:
        """
　　　　　　# 獲取item數據，並轉換成字典格式

        d = dict(item)
　　　　　　 # 有空值得不保存
        if all(d.values()):
　　　　　　　　　　# 保存到mongodb中
            self.client[self.db][self.collection].save(d)
        return item

        # 表示將item丟棄，不會被后續pipeline處理
        # raise DropItem()

 

6. 配置文件

# database server
DB = "jd"
COLLECTION = "goods"
HOST = "127.0.0.1"
PORT = 27017
USER = "root"
PWD = "123"
ITEM_PIPELINES = {
   'MyScrapy.pipelines.MongoPipeline': 300,
}