scrapy爬蟲中如何實現翻頁請求

通過scrapy.Request實現翻頁請求：

 scrapy.Request(url, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None,
                         encoding='utf-8', priority=0, dont_filter=False, errback=None, flags=None)

這里一爬取騰訊招聘網站的崗位信息為例制作一個爬蟲進行翻頁請求的實現

# -*- coding: utf-8 -*-
import scrapy


class HrSpider(scrapy.Spider):
    name = 'Hr'
    allowed_domains = ['tencent.com']
    start_urls = ['https://hr.tencent.com/position.php']

    def parse(self, response):
        tr_list=response.xpath("//table[@class='tablelist']/tr")[1:-1]
        for tr in tr_list:
            item={}
            item["title"]=tr.xpath("./td[1]/a/text()").extract_first()
            item["postion"]=tr.xpath("./td[2]/text()").extract_first()
            item["publish_date"]=tr.xpath("./td[5]/text()").extract_first()
            yield item
        #找到下一頁的URL地址,實現翻頁請求
        next_url=response.xpath("//a[@id='next']/@href").extract_first()
        if next_url !=" javascript:;":
            next_url="https://hr.tencent.com/"+next_url
            yield scrapy.Request(
                next_url,
                callback=self.parse
            )