python scrapy 报错 raise ValueError(f'Missing scheme in request url:


class ImgSpider(scrapy.Spider):
    name = 'img'
    # allowed_domains = ['https://sc.chinaz.com/tupian/']
    start_urls = ['https://sc.chinaz.com/tupian/']

    def parse(self, response):
        res_list = response.xpath('//div[@id="container"]/div')
        for i in res_list:
            res_item = i.xpath('./div/a/img/@src2').extract_first()
            item = ImgsproItem()
            item['src'] = res_item
            yield item

错误提示如下

 raise ValueError(f'Missing scheme in request url: {self._url}')

原因分析:

  res_item得到的地址缺少域名,是不完整的url,为确保请求地址正确,可以使用urljoin()。

参见 https://stackoverflow.com/questions/42026244/scrapy-valueerrormissing-scheme-in-request-url-s-self-url

代码修改如下后可正确执行

    def parse(self, response):
        res_list = response.xpath('//div[@id="container"]/div')
        for i in res_list:
            res_item = i.xpath('./div/a/img/@src2').extract_first()
            item = ImgsproItem()
            # item['src'] = res_item
            item['src'] = response.urljoin(res_item)
            yield item

 


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM