Python 英文詞頻統計

1. 詞頻統計預處理
2. 下載一首英文的歌詞或文章
3. 將所有,.？！’:等分隔符全部替換為空格
4. 將所有大寫轉換為小寫
5. 生成單詞列表
6. 生成詞頻統計
7. 排序
8. 排除語法型詞匯，代詞、冠詞、連詞
9. 輸出詞頻最大TOP10

article ='''
Big data analytics and business analytics
by Duan, Lian; Xiong, Ye
Over the past few decades, with the development of automatic identification, data capture and storage technologies, 
people generate data much faster and collect data much bigger than ever before in business, science, engineering, education and other areas. 
Big data has emerged as an important area of study for both practitioners and researchers. 
It has huge impacts on data-related problems. 
In this paper, we identify the key issues related to big data analytics and then investigate its applications specifically related to business problems.
'''

split = article.split()
print(split)

#使用空格替換標點符號
article = article.replace(",","").replace(".","").replace(":","").replace(";","").replace("?","")


#大寫字母轉換成小寫字母
exchange = article.lower();
print(exchange)

#生成單詞列表
list = exchange.split()
print(list)

#生成詞頻統計
dic = {}
for i in list:
    count = list.count(i)
    dic[i] = count
print(dic)

#排除特定單詞
word = {'and','the','with','in','by','its','for','of','an','to'}
for i in word:
    del(dic[i])
print(dic)

#排序
dic1= sorted(dic.items(),key=lambda d:d[1],reverse= True)
print(dic1)

#輸出詞頻最大的前十位單詞
for i in range(10):
    print(dic1[i])