使用Python的requests 和BeautifulSoup模塊，Python 2.7.12可在命令行中直接使用pip進行模塊安裝。爬蟲的核心是利用BeautifulSoup的select語句獲取需要的信息。
pip install requests pip install bs4以武漢市2017年5~7月的歷史為例爬取天氣網中武漢市的歷史天氣數據。
7月對應的網址為http://lishi.tianqi.com/wuhan/201707.html
1.requests模塊獲取網頁內容
url='http://lishi.tianqi.com/wuhan/201707.html' response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') 2.利用.select語句找到網頁中天氣數據所在的div
weather_list = soup.select('div[class="tqtongji2"]') 
3.找出日期、最高氣溫、最低氣溫、天氣等數據，用li.string獲取li中的信息。
ul_list = weather.select('ul') for ul in ul_list: li_list= ul.select('li') for li in li_list: li.string.encode('utf-8') #具體的天氣信息
具體代碼實現如下：
#encoding:utf-8 import requests from bs4 import BeautifulSoup urls = ["http://lishi.tianqi.com/wuhan/201707.html", "http://lishi.tianqi.com/wuhan/201706.html", "http://lishi.tianqi.com/wuhan/201705.html"] file = open('wuhan_weather.csv','w') for url in urls: response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') weather_list = soup.select('div[class="tqtongji2"]') for weather in weather_list: weather_date = weather.select('a')[0].string.encode('utf-8') ul_list = weather.select('ul') i=0 for ul in ul_list: li_list= ul.select('li') str="" for li in li_list: str += li.string.encode('utf-8')+',' if i!=0: file.write(str+'\n') i+=1 file.close() 最后的結果： 