Python爬蟲post一例

抓取博客園(https://www.cnblogs.com/)分類列表(下圖紅框所示),在瀏覽器直接查看網頁的源碼,是看不到這部分內容的.

抓取方法如下:

使用谷歌瀏覽器,按F12,切換到Network,點擊第一個按鈕開始抓包.

 

按F5重新刷新頁面,左邊會出現所有請求內容.

這里就需要一個一個人工查看,如下圖,找到所需內容.

切換到Headers選項,可以查看到請求的類型:

方式:post

"content-type":"application/json; charset=UTF-8",
"user-agent":"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Mobile Safari/537.36",
"x-requested-with": "XMLHttpRequest",
"accept":"text/plain, */*; q=0.01",
"accept-encoding": "gzip, deflate, br",

最下面的Request Payload就是post的數據.

 

 

from MyRequestPost import MyRequestPost#自己寫的一個公用類
from lxml import etree

headers = {
"content-type":"application/json; charset=UTF-8",
"user-agent":"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Mobile Safari/537.36",
"x-requested-with": "XMLHttpRequest",
"accept":"text/plain, */*; q=0.01",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7",
#"cookie": "xxx"
};
data = '{"cateIds": "108698,2,108701,108703,108704,108705,108709,108712,108724,4"}'

post = MyRequestPost(url="https://www.cnblogs.com/aggsite/SubCategories",headers = headers,data = data.encode('utf-8'))

print(post.htmlCode);

然后輸出結果如下:

這里有個小坑:

提交的數據看起來是json數據,開始我是這么寫的:

data = {"cateIds": "108698,2,108701,108703,108704,108705,108709,108712,108724,4"},看起來名正言順的json呢,但是總是返回500錯誤:

 

后來直接把json字符串加了引號,終於成功.

另外:

"content-type"="application/json; charset=UTF-8"的時候,數據是json數據.

Content-Type=application/x-www-form-urlencoded; charset=UTF-8,數據應該是表單數據.

例如:

data = [

            (‘user’, 'xxxx'),

            ('password', '5xxxx'),]