爬取淘寶“手機信息”

打開淘寶，我們搜索手機，返回以下界面

 

 接下來我們來爬取這些數據

 

 第一步我們先獲取網頁html

html = requests.get(url,headers=headers)
    print(html.text)

將結果打印后發現所需要的信息在網頁代碼中沒有，所以它應該是動態加載的，這里再network中一個一個找，找到它返回的json文件，就是這樣的。

 

 

最后可以發現內容都在一個json文件中，到這里接下來的就很簡單了，我們只需要解析json找我我們需要的數據就可以了，將這一部分封裝起來。

def get_data(url):
    html = requests.get(url,headers=headers,)
    html_text = html.text
    starts = html_text.find('{"pageName":"mainsrp"')
    end = html_text.find('"shopcardOff":true}}')+len('"shopcardOff":true}}')
    json_data = json.loads(html_text[starts:end])
    get_json_data = json_data['mods']['itemlist']['data']['auctions']
    for data in get_json_data:
        title = data['title']
        item_loc = data['item_loc']
        view_sales = data['view_sales']
        nick = data['nick']
        view_price = data['view_price']
        pic_url = data['pic_url']
        pic_url = parse.urljoin('http:',pic_url)
        print(title,'\n',item_loc,'\n',view_sales,'店鋪:',nick,"價格：",view_price)
        download(pic_url)
        print('-'*80)

結果如下

 

 

 

 這里我們就將一頁的爬完了，這里我們嘗試將所有的都爬下來，這里我們點擊第二頁

https://s.taobao.com/search?ie=utf8&initiative_id=staobaoz_20200402&stats_click=search_radio_all%3A1&js=1&imgfile=&q=%E6%89%8B%E6%9C%BA&suggest=history_2&_input_charset=utf-8&wq=&suggest_query=&source=suggest&bcoffset=3&ntoffset=3&p4ppushleft=1%2C48&s=44

https://s.taobao.com/search?ie=utf8&initiative_id=staobaoz_20200402&stats_click=search_radio_all%3A1&js=1&imgfile=&q=%E6%89%8B%E6%9C%BA&suggest=history_2&_input_charset=utf-8&wq=&suggest_query=&source=suggest&bcoffset=6&ntoffset=6&p4ppushleft=1%2C48&s=0

可以看到下一頁的url為上一頁中s+44這樣我們就可以爬下一頁了

    for each in range(0,1000,44):
        url = 'https://s.taobao.com/search?q=%E6%89%8B%E6%9C%BA&imgfile=&commend=all&ssid=s5-e&search_type=item&sourceId=tb.index&spm=a21bo.2017.201856-taobao-item.1&ie=utf8&initiative_id=tbindexz_20170306&sort=sale-desc&bcoffset=0&p4ppushleft=%2C44&s={}'.format(each)

OK全部解決

完整代碼如下：

import requests,json,lxml,os
from lxml import etree
from urllib import parse
from uuid import uuid4
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36 Edg/80.0.361.69'
     ,'cookie':自己的cookie
                }
def get_data(url):
    html = requests.get(url,headers=headers,)
    html_text = html.text
    starts = html_text.find('{"pageName":"mainsrp"')
    end = html_text.find('"shopcardOff":true}}')+len('"shopcardOff":true}}')
    json_data = json.loads(html_text[starts:end])
    get_json_data = json_data['mods']['itemlist']['data']['auctions']
    for data in get_json_data:
        title = data['title']
        item_loc = data['item_loc']
        view_sales = data['view_sales']
        nick = data['nick']
        view_price = data['view_price']
        pic_url = data['pic_url']
        pic_url = parse.urljoin('http:',pic_url)
        print(title,'\n',item_loc,'\n',view_sales,'店鋪:',nick,"價格：",view_price)
        download(pic_url)
        print('-'*80)

def download(url):
    response = requests.get(url)
    img = response.content
    with open('文件路徑{}.jpg'.format(uuid4()),'wb') as f:
        f.write(img)


if __name__ == '__main__':
    for each in range(0,1000,44):
        url = 'https://s.taobao.com/search?q=%E6%89%8B%E6%9C%BA&imgfile=&commend=all&ssid=s5-e&search_type=item&sourceId=tb.index&spm=a21bo.2017.201856-taobao-item.1&ie=utf8&initiative_id=tbindexz_20170306&sort=sale-desc&bcoffset=0&p4ppushleft=%2C44&s={}'.format(each)

        get_data(url)