python文本處理 數據挖掘 停用詞檢索

簡單描述程序功能：

1.停用詞為csv文件

2.源文件為txt文件

3.文本處理，將原文件中出現的停用詞去除

代碼實現：

1.文件讀取，分詞，源文件詞頻統計

python 讀取 西班牙語文本編碼： encoding='ISO-8859-1'

#csv 文件讀取，此處編碼為西班牙語
def csvfile():
    file_path = os.path.join(upload_path, "SpanishStopWords.csv")
    with open(file_path,'r',encoding='ISO-8859-1') as f:
        reader = csv.reader(f)
        fieldnames = next(reader)#獲取數據的第一列，作為后續要轉為字典的鍵名 生成器，next方法獲取
        # print(fieldnames)
        data1=[]
        csv_reader = csv.DictReader(f,fieldnames=fieldnames) #self._fieldnames = fieldnames   # list of keys for the dict 以list的形式存放鍵名
        for row in csv_reader:
            dic1={}
            for k,v in row.items():
                dic1[k]=v
            data1.append(dic1)
        return data1
#txt文件讀取
def eachcount():
    file_path = os.path.join(upload_path, "Alamo.txt")
    txt = open(file_path, 'r', encoding='ISO-8859-1').read()
    #分詞
    txt = txt.replace(',', ' ').replace('.', ' ')
    txt = txt.split()
    counts = {}  # 定義一個空字典類型
    print(txt)
    for word in txt:
        counts[word] = counts.get(word, 0) + 1  # 獲取word當前有幾個，如果word不存在則為0
    items = list(counts.items())
    # 對一個列表按照鍵值對的兩個元素的第二個元素進行排序，由大到小的倒排，詞頻排序
    items.sort(key=lambda x: x[1], reverse=False)
    return items

2.顯示在原文件中出現的所有停用詞

#顯示在源文件中出現過的所有停用詞
@application.route('/listsearch/', methods=['GET', 'POST'])
def listsearch():
   file_path = os.path.join(upload_path, "SpanishStopWords.csv")
   txt = open(file_path, 'r', encoding='ISO-8859-1').read()
   txt = txt.split()
   filelist=txt
   # filelist=csvfile()
   filelist2=docu2()
   # wordlist=["my","name","boy","chirs","Dave"]
   result=[]
   result2=[]
   # for j in wordlist:
   #  for i in filelist:
   #     if i[0]== j :
   #        result.append(i)
   for j in filelist:
      for i in filelist2:
         if j== i :
            result2.append(j)
   return render_template('index.html',result2=result2)

前端代碼展現：

<form action="/listsearch" method="get"  enctype="multipart/form-data">
         <button type="submit" value="submit">search</button>

     <p>result</p>
     {% for line2 in result2 %}
             <p>{{ line2}}</p>

{% endfor %}
    </form>