爬取豆瓣網影評數據並進行簡單分析與展示

編譯在線環境：

https://www.kesci.com

一.  內容

1、使用Python爬蟲爬取豆瓣網某一部電影的評論信息；

2、從評論信息中統計各級星評的數量占比

from urllib import request
from bs4 import BeautifulSoup
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import requests
import re

"""
生成分頁網址
n:需要生成的頁數
"""
def get_urls(n):
    urls=[]
    for i in range(0,n):
        u='https://movie.douban.com/subject/30166972/comments?start='+str(i*20)+'&limit=20&sort=new_score&status=P'
        urls.append(u)
    return urls
#get_urls(5)

"""
【從分頁采集數據】
u:輸入分頁網址
"""
def get_informations(u):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
    r = requests.get(u, headers=headers)
    soup = BeautifulSoup(r.text, 'lxml')
    ims = soup.find_all('div', class_='comment-item')

    # print(ims)
    data = []

    for i in ims:
        dic = {}
        dic['用戶名'] = i.find('span', class_='comment-info').find('a', class_="").text
        mystar = i.find('span', class_='comment-info').find('span').next_sibling.next_sibling
        mystar = str(mystar)
        t = re.findall(r'\d+', mystar)
        dic['所評星級'] = str(t[0])
        dic['評論'] = i.find('span', class_='short').text
        dic['評論獲贊數'] = i.find('span', class_='votes').text
        dic['評論時間'] = i.find('span', class_='comment-time').text.strip()
        data.append(dic)
    return data
#get_informations('https://movie.douban.com/subject/30166972/comments?start=0&limit=20&sort=new_score&status=P')

"""
【從每一頁采取所有數據】
n:需要采集的頁數
"""
def get_alldata(n):
    alldata = []
    for u in get_urls(n):
        alldata.extend(get_informations(u))
    """
    清洗數據
    轉換數據類型以及星級去除%
    .loc就是索引，逗號前面是行索引，逗號后面是列索引
    """
    df.loc[:, '所評星級'] = df['所評星級'].astype('float')
    df.loc[:, '評論獲贊數'] = df['評論獲贊數'].astype('int')

    return pd.DataFrame(alldata)

"""
內容1、爬取豆瓣網某一部電影的評論信息
"""
#保存數據到《少年的你》影評.csv
df = get_alldata(5)
df.to_csv('《少年的你》影評.csv',index=False,encoding='utf-8')
print(df)

"""pandas繪圖：直方圖，bins是像素"""
df['星級'].hist(bins=20)

二. 所涉及的知識點：

1.python去除空格和換行符的方法

#str.strip()
s1='\nabcdefg\n'
print(s1.strip())

字符竄：DataFrame數據框里邊的name列為字符竄形式

清除字符竄左側是空值：

newname=df['name'].str.lstrip()


刪除右側：
newname=df['name'].str.rstrip()


刪除全部：
newname=df['name'].str.strip()

2.pandas常用函數

https://www.cnblogs.com/rexyan/p/7975707.html

https://www.jianshu.com/p/7332afd70797

3.Pandas 關於統計個數

https://blog.csdn.net/cyx441984694/article/details/85029009?utm_source=distribute.pc_relevant.none-task

4.幾個相同的標簽<>,如果想獲得第二個標簽里面的內容

next_sibling
“”“
<p class='comment-time'>ads<\p>
<p class='comment-time'>bdg<\p>
<p class='comment-time'>fgh<\p >
可以通過next_sibling
”“”
Tsecond =  i.find('p',class_='comment-time').next_sibling.next_sibling

TsecondText =  i.find('p',class_='comment-time').next_sibling.next_sibling.text

5.matplotlib畫圖時的中文設置

 采用matplotlib作圖時默認設置下是無法顯示中文的,加上以下語句：

import matplotlib as mpl
import matplotlib.pyplot as plt

mpl.rcParams['font.sans-serif'] = ['KaiTi']
mpl.rcParams['font.serif'] = ['KaiTi']