沒想到上面好看的跳舞小姐姐蠻多的，【Python爬蟲】采集微博視頻數據

前言

隨時隨地發現新鮮事！微博帶你欣賞世界上每一個精彩瞬間，了解每一個幕后故事。分享你想表達的，讓全世界都能聽到你的心聲！今天我們通過python去采集微博當中好看的視頻！

沒錯，今天的目標是微博數據采集，爬的是那些好看的小姐姐視頻

對於本篇文章有疑問的同學可以加【資料白嫖、解答交流群：910981974】

知識點

• requests
• pprint

開發環境

• 版 本：python 3.8
  -編輯器：pycharm 2021.2

爬蟲原理

• 作用：批量獲取互聯網數據(文本, 圖片, 音頻, 視頻)
• 本質：一次次的請求與響應
  

 

案例實現

1. 導入所需模塊

import requests
import pprint

 

2. 找到目標網址

打開開發者工具，選中Fetch/XHR，選中數據所在的標簽，找到目標所在url

 https://www.weibo.com/tv/api/component?page=/tv/channel/4379160563414111/editor

 

3. 發送網絡請求

headers = {
    'cookie': '',
    'referer': 'https://weibo.com/tv/channel/4379160563414111/editor',
    'user-agent': '',
}
data = {
    'data': '{"Component_Channel_Editor":{"cid":"4379160563414111","count":9}}'
}
url = 'https://www.weibo.com/tv/api/component?page=/tv/channel/4379160563414111/editor'
json_data = requests.post(url=url, headers=headers, data=data).json()

 

4. 獲取數據

json_data_2 = requests.post(url=url_1, headers=headers, data=data_1).json()

 

5. 篩選數據

dict_urls = json_data_2['data']['Component_Play_Playinfo']['urls']
video_url = "https:" + dict_urls[list(dict_urls.keys())[0]]
print(title + "\t" + video_url)

 

6. 保存數據

video_data = requests.get(video_url).content
with open(f'video\\{title}.mp4', mode='wb') as f:
    f.write(video_data)
print(title, "爬取成功................")

 

完整代碼

import requests
import pprint

headers = {
    'cookie': '添加自己的',
    'referer': 'https://weibo.com/tv/channel/4379160563414111/editor',
    'user-agent': '',
}
data = {
    'data': '{"Component_Channel_Editor":{"cid":"4379160563414111","count":9}}'
}
url = 'https://www.weibo.com/tv/api/component?page=/tv/channel/4379160563414111/editor'
json_data = requests.post(url=url, headers=headers, data=data).json()
print(json_data)

ccs_list = json_data['data']['Component_Channel_Editor']['list']
next_cursor = json_data['data']['Component_Channel_Editor']['next_cursor']
for ccs in ccs_list:
    oid = ccs['oid']
    title = ccs['title']
    data_1 = {
        'data': '{"Component_Play_Playinfo":{"oid":"' + oid + '"}}'
    }
    url_1 = 'https://weibo.com/tv/api/component?page=/tv/show/' + oid
    json_data_2 = requests.post(url=url_1, headers=headers, data=data_1).json()
    dict_urls = json_data_2['data']['Component_Play_Playinfo']['urls']
    video_url = "https:" + dict_urls[list(dict_urls.keys())[0]]
    print(title + "\t" + video_url)

    video_data = requests.get(video_url).content
    with open(f'video\\{title}.mp4', mode='wb') as f:
        f.write(video_data)
    print(title, "爬取成功................")