使用requests模塊簡單獲取數據

一、使用ruquests的基本步驟：

• 指定url
• 發起請求
• 獲取響應對象中的數據
• 持久化存儲

#1
url = 'https://www.sogou.com/'
#2.
response = requests.get(url=url)
#3.
page_text = response.text
#4.
with open('./sogou.html','w',encoding='utf-8') as fp:
    fp.write(page_text)

二、爬取搜狗指定搜索

import requests
url = "'https://www.sogou.com/web"
wd = input("請輸入搜索關鍵字")
param = {
    'query':wd
}

response = requests.get(url=url,params=param).content
filename = wd+'.html'
with open(filename,'w',encoding='utf8') as f1:
        f1.write(response)

三、Ajax請求

通過抓包，獲取請求攜帶的參數，

例如獲取分頁顯示的數據，當點擊下一頁時，發送ajax請求，對此時的url請求可以動，這里我們定義好請求參數param，動態的指定頁碼和每頁顯示的數據，通過ajax請求，返回一組json數據

存儲每頁獲取的數據的id，編輯new_url，獲取詳情的信息

import requests
url = 'http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsList'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0'
}
param = {
   "on":"true",
    "page":1,
    "pageSize":"15",
    "productName":"",
    "conditionType":"1",
    "applyname":"",
    "applysn":"",
}
id_list = []
json_object = requests.post(url=url,headers=headers,params=param).json()
print(json_object['list'])
for i in json_object['list']:
    id_list.append(i['ID'])
    
new_url = 'http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsById'
filename = 'yaojians.text'
f = open(filename,'w',encoding='utf8')
for id in id_list:
    param = {
        'id':id
    }
    content_json = requests.post(url=new_url,params=param,headers=headers).json()
    f.write(str(content_json)+'\n')
    
    
    

四、使用BeautifullSoup爬取數據

bs4解析：
　　pip install bs4
　　pip install lxml

解析原理
　　1、將要進行解析的源碼加載到bs對象

　　2、調用bs對象中相關的方法或屬性進行源碼中的相關標簽的定位

　　3、將定位到的標簽之間存在的文本或屬性值獲取

import requests
from bs4 import BeautifulSoup

url = 'http://www.shicimingju.com/book/sanguoyanyi.html'
headers = {
    "User-Agent":'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'
}
res = requests.get(url=url, headers=headers).text
soup = BeautifulSoup(res, 'lxml')
a_tags_list = soup.select('.book-mulu > ul > li > a')
filename = 'snaguo.text'
fp = open(filename, 'w', encoding='utf-8')
for a_tag in a_list:
    title = a_tag.string
    detail_url = "http://www.shicimingju.com"+a_tag["href"] 
    detail_content = requests.get(url=detail_url, headers=headers).text
    soup = BeautifulSoup(detail_content, "lxml")
    detail_text = soup.find('div', class_="chapter_content").text
    fp.write(title+'\n'+detail_text)
    print(title, '下載完畢')
print('over')
fp.close()
    

 

五、簡單使正則爬取圖片

url = 'https://www.qiushibaike.com/pic/page/%d/?s=5170552'
start_page = int(input("請輸入起始頁:"))
end_page = int(input("請輸入結束頁:"))
headers = {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"
}
for page in range(start_page,end_page+1):
    new_url = format(url%page)
    response = requests.get(url=new_url, headers=headers).text
    # 每一頁的圖片url
    images_url = re.findall('<div class="thumb">.*?<img src="(.*?)" alt=.*?</div>',response,re.S)
os.mkdir('qiutu')
    
    for image_url in images_url:
        detail_url = 'http:'+image_url
        # 獲取到當前圖片的二進制流
        content = requests.get(url=detail_url,headers=headers).content
        # 切割 把圖片路徑最后的字符作為圖片名
        image_name = image_url.split('/')[-1]
        with open('./qiutu/'+image_name,'wb')as f1:
            f1.write(content)
print('over')