python主題建模可視化LDA和T-SNE交互式可視化

原文鏈接：http://tecdat.cn/?p=6917

 

我嘗試使用Latent Dirichlet分配LDA來提取一些主題。 本教程以端到端的自然語言處理流程為特色，從原始數據開始，貫穿准備，建模，可視化論文。

我們將涉及以下幾點

使用LDA進行主題建模
使用pyLDAvis可視化主題模型
使用t-SNE和散景可視化LDA結果

In [1]:

from scipy import sparse as sp

Populating the interactive namespace from numpy and matplotlib

In [2]:

docs = array(p_df['PaperText'])

 預處理和矢量化文檔

In [3]:

from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize import RegexpTokenizer

def docs_preprocessor(docs):
    tokenizer = RegexpTokenizer(r'\w+')
    for idx in range(len(docs)):
        docs[idx] = docs[idx].lower()  # Convert to lowercase.
        docs[idx] = tokenizer.tokenize(docs[idx])  # Split into words.

    # Remove numbers, but not words that contain numbers.
    docs = [[token for token in doc if not token.isdigit()] for doc in docs]
    
    # Remove words that are only one character.
    docs = [[token for token in doc if len(token) > 3] for doc in docs]
    
    # Lemmatize all words in documents.
    lemmatizer = WordNetLemmatizer()
    docs = [[lemmatizer.lemmatize(token) for token in doc] for doc in docs]
  
    return docs

In [4]:

docs = docs_preprocessor(docs)

 計算雙字母組/三元組：

 正弦主題非常相似，可以區分它們是短語而不是單個/單個單詞。

In [5]:

from gensim.models import Phrases
# Add bigrams and trigrams to docs (only ones that appear 10 times or more).
bigram = Phrases(docs, min_count=10)
trigram = Phrases(bigram[docs])

for idx in range(len(docs)):
    for token in bigram[docs[idx]]:
        if '_' in token:
            # Token is a bigram, add to document.
            docs[idx].append(token)
    for token in trigram[docs[idx]]:
        if '_' in token:
            # Token is a bigram, add to document.
            docs[idx].append(token)

Using TensorFlow backend.
/opt/conda/lib/python3.6/site-packages/gensim/models/phrases.py:316: UserWarning: For a faster implementation, use the gensim.models.phrases.Phraser class
  warnings.warn("For a faster implementation, use the gensim.models.phrases.Phraser class")

刪除

In [6]:

from gensim.corpora import Dictionary

# Create a dictionary representation of the documents.
dictionary = Dictionary(docs)
print('Number of unique words in initital documents:', len(dictionary))

# Filter out words that occur less than 10 documents, or more than 20% of the documents.
dictionary.filter_extremes(no_below=10, no_above=0.2)
print('Number of unique words after removing rare and common words:', len(dictionary))

Number of unique words in initital documents: 39534
Number of unique words after removing rare and common words: 6001

修剪常見和罕見的單詞，我們最終只有大約6％的單詞。

矢量化數據：
第一步是獲得每個文檔的單詞表示。

In [7]:

corpus = [dictionary.doc2bow(doc) for doc in docs]

In [8]:

print('Number of unique tokens: %d' % len(dictionary))
print('Number of documents: %d' % len(corpus))

Number of unique tokens: 6001
Number of documents: 403

通過詞袋語料庫，我們可以繼續從文檔中學習我們的主題模型。

訓練LDA模型 

In [9]:

from gensim.models import LdaModel

In [10]:

%time model = LdaModel(corpus=corpus, id2word=id2word, chunksize=chunksize, \
                       alpha='auto', eta='auto', \
                       iterations=iterations, num_topics=num_topics, \
                       passes=passes, eval_every=eval_every)

CPU times: user 3min 58s, sys: 348 ms, total: 3min 58s
Wall time: 3min 59s

如何選擇主題數量？

LDA是一種無監督的技術，這意味着我們在運行模型之前不知道在我們的語料庫中有多少主題存在。 主題連貫性是用於確定主題數量的主要技術之一。 

但是，我使用了LDA可視化工具pyLDAvis，嘗試了幾個主題並比較了結果。 四個似乎是最能分離主題的最佳主題數量。

In [11]:

import pyLDAvis.gensim
pyLDAvis.enable_notebook()

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning) 

In [12]:

pyLDAvis.gensim.prepare(model, corpus, dictionary)

Out[12]:

我們在這看到什么？

左側面板，標記為Intertopic Distance Map，圓圈表示不同的主題以及它們之間的距離。類似的主題看起來更近，而不同的主題更遠。圖中主題圓的相對大小對應於語料庫中主題的相對頻率。 

如何評估我們的模型？ 

將每個文檔分成兩部分，看看分配給它們的主題是否類似。 =>越相似越好

將隨機選擇的文檔相互比較。 =>越不相似越好

In [13]:

from sklearn.metrics.pairwise import cosine_similarity

p_df['tokenz'] = docs

docs1 = p_df['tokenz'].apply(lambda l: l[:int0(len(l)/2)])
docs2 = p_df['tokenz'].apply(lambda l: l[int0(len(l)/2):])

Transform the data

In [14]:

corpus1 = [dictionary.doc2bow(doc) for doc in docs1]
corpus2 = [dictionary.doc2bow(doc) for doc in docs2]

# Using the corpus LDA model tranformation
lda_corpus1 = model[corpus1]
lda_corpus2 = model[corpus2]

In [15]:

from collections import OrderedDict
def get_doc_topic_dist(model, corpus, kwords=False):
    
    '''
    LDA transformation, for each doc only returns topics with non-zero weight
    This function makes a matrix transformation of docs in the topic space.
    '''
    top_dist =[]
    keys = []

    for d in corpus:
        tmp = {i:0 for i in range(num_topics)}
        tmp.update(dict(model[d]))
        vals = list(OrderedDict(tmp).values())
        top_dist += [array(vals)]
        if kwords:
            keys += [array(vals).argmax()]

    return array(top_dist), keys

Intra similarity: cosine similarity for corresponding parts of a doc(higher is better):
0.906086532099
Inter similarity: cosine similarity between random parts (lower is better):
0.846485334252

 讓我們看一下每個主題中出現的術語。

In [17]:

def explore_topic(lda_model, topic_number, topn, output=True):
    """
    accept a ldamodel, atopic number and topn vocabs of interest
    prints a formatted list of the topn terms
    """
    terms = []
    for term, frequency in lda_model.show_topic(topic_number, topn=topn):
        terms += [term]
        if output:
            print(u'{:20} {:.3f}'.format(term, round(frequency, 3)))
    
    return terms

In [18]:

term                 frequency

Topic 0 |---------------------

data_set             0.006
embedding            0.004
query                0.004
document             0.003
tensor               0.003
multi_label          0.003
graphical_model      0.003
singular_value       0.003
topic_model          0.003
margin               0.003
Topic 1 |---------------------

policy               0.007
regret               0.007
bandit               0.006
reward               0.006
active_learning      0.005
agent                0.005
vertex               0.005
item                 0.005
reward_function      0.005
submodular           0.004
Topic 2 |---------------------

convolutional        0.005
generative_model     0.005
variational_inference 0.005
recurrent            0.004
gaussian_process     0.004
fully_connected      0.004
recurrent_neural     0.004
hidden_unit          0.004
deep_learning        0.004
hidden_layer         0.004
Topic 3 |---------------------

convergence_rate     0.007
step_size            0.006
matrix_completion    0.006
rank_matrix          0.005
gradient_descent     0.005
regret               0.004
sample_complexity    0.004
strongly_convex      0.004
line_search          0.003
sample_size          0.003

 從上面可以檢查每個主題並為其分配一個人類可解釋的標簽。 在這里我將它們標記如下：

In [19]:

top_labels = {0: 'Statistics', 1:'Numerical Analysis', 2:'Online Learning', 3:'Deep Learning'}

In [20]:

    '''
    # 1. Remove non-letters
    paper_text = re.sub("[^a-zA-Z]"," ", paper)
    # 2. Convert words to lower case and split them
    words = paper_text.lower().split()
    # 3. Remove stop words
    words = [w for w in words if not w in stops]
    # 4. Remove short words
    words = [t for t in words if len(t) > 2]
    # 5. lemmatizing
    words = [nltk.stem.WordNetLemmatizer().lemmatize(t) for t in words]

In [21]:

from sklearn.feature_extraction.text import TfidfVectorizer

tvectorizer = TfidfVectorizer(input='content', analyzer = 'word', lowercase=True, stop_words='english',\
                                  tokenizer=paper_to_wordlist, ngram_range=(1, 3), min_df=40, max_df=0.20,\
                                  norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=True)

dtm = tvectorizer.fit_transform(p_df['PaperText']).toarray()

In [22]:

top_dist =[]
for d in corpus:
    tmp = {i:0 for i in range(num_topics)}
    tmp.update(dict(model[d]))
    vals = list(OrderedDict(tmp).values())
    top_dist += [array(vals)]

In [23]:

top_dist, lda_keys= get_doc_topic_dist(model, corpus, True)
features = tvectorizer.get_feature_names()

In [24]:

top_ws = []
for n in range(len(dtm)):
    inds = int0(argsort(dtm[n])[::-1][:4])
    tmp = [features[i] for i in inds]
    
    top_ws += [' '.join(tmp)]
    


cluster_colors = {0: 'blue', 1: 'green', 2: 'yellow', 3: 'red', 4: 'skyblue', 5:'salmon', 6:'orange', 7:'maroon', 8:'crimson', 9:'black', 10:'gray'}

p_df['colors'] = p_df['clusters'].apply(lambda l: cluster_colors[l])

In [25]:

from sklearn.manifold import TSNE
tsne = TSNE(n_components=2)
X_tsne = tsne.fit_transform(top_dist)

In [26]:

p_df['X_tsne'] =X_tsne[:, 0]
p_df['Y_tsne'] =X_tsne[:, 1]

In [27]:

from bokeh.plotting import figure, show, output_notebook, save#, output_file
from bokeh.models import HoverTool, value, LabelSet, Legend, ColumnDataSource
output_notebook()

 BokehJS 0.12.5成功加載。

In [28]:

source = ColumnDataSource(dict(
    x=p_df['X_tsne'],
    y=p_df['Y_tsne'],
    color=p_df['colors'],
    label=p_df['clusters'].apply(lambda l: top_labels[l]),
#     msize= p_df['marker_size'],
    topic_key= p_df['clusters'],
    title= p_df[u'Title'],
    content = p_df['Text_Rep']
))

In [29]:

title = 'T-SNE visualization of topics'



plot_lda.scatter(x='x', y='y', legend='label', source=source,
                 color='color', alpha=0.8, size=10)#'msize', )

show(plot_lda)

​

 

 

如果您有任何疑問，請在下面發表評論。