scrapy 爬取圖片 最基本操作

使用scrapy里自帶的Image功能下載,下面貼代碼，解釋在代碼的注釋里。

 

items.py

import scrapy

class ImageItem(scrapy.Item):
    #注意這里的item是ImageItem

    image_urls = scrapy.Field()
    images = scrapy.Field()

#image_urls和images是固定的,不能改名字

 

settings.py

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
#上面只是個訪問header，加個降低被拒絕的保險

ITEM_PIPELINES = {
    'scrapy.pipelines.images.ImagesPipeline': 1
}
#打開Images的通道

IMAGES_STORE = 'F:\\pics'
#一定要設置這個存儲地址，要是真實的硬盤，可以不創建pics文件夾，會自己生成，還生成默認的full文件夾

 

 spider.py(這里是carhome） 

import scrapy

from car.items import ImageItem

class CarhomeSpider(scrapy.Spider):
    name = 'carhome'
    allowed_domains = ['sohu.com']
    start_urls = ['http://www.sohu.com/a/337634404_100032610']
    download_delay = 1

    def parse(self, response):
        item = ImageItem()
        srcs = response.css('.article img::attr(src)').extract() #用css方法找到的所有圖片地址
        item['image_urls'] = srcs
        yield item

 

pipelines.py 

from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
from scrapy.http import Request
#這里的兩個函數get_media_requests和item_completed都是scrapy的內置函數，想重命名的就這這里操作
#可以直接復制這里的代碼就可以用了
class MyImagesPipeline(ImagesPipeline) :
    def get_media_requests(self, item, info) :
        for image_url in item['image_urls'] :
            yield Request(image_url)

    def item_completed(self, results, item, info) :
        image_path = [x['path'] for ok, x in results if ok]
        if not image_path :
            raise DropItem('Item contains no images')
        item['image_paths'] = image_path
        return item

 代碼就這么多，下面執行一下：

 

 文件夾結果：

 

 還不會的快去試試吧！