TEXT CLASSIFICATION WITH TORCHTEXT

本文譯自PYTORCH官網TEXT系列。本節主要利用torchtext中的文本分類數據集，包括：

這個例子展示了如何利用這些TextClassfication數據集中的一個來訓練監督學習算法。

 

用ngrams加載數據

一個ngrams包特性被用來捕獲一些關於本地詞序的部分信息。在實際應用中，雙字元（bi-gram）或三字元(tri-gram)作為詞組比只使用一個詞更有益處。例如：

TextClassfication 數據集支持ngrams方法。通過設定ngrams為2，數據集中的text將包含單個單詞和bi-grams字符串。

import torch
import torchtext
from torchtext.datasets import text_classification
NGRAMS = 2
import os
if not os.path.isdir('./.data'):
    os.mkdir('./.data')
train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](
    root='./.data', ngrams=NGRAMS, vocab=None)
BATCH_SIZE = 16
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

 

定義模型

模型由EmbeddingBag層和線性層組成。nn.EmbeddingBag計算embedding包的均值。不同的文本長度是不同的。nn.EmbeddingBag無需padding，因為文本長度在offsets已保存。

此外，因為nn.EmbeddingBag動態累加嵌入的平均值，nn.EmbeddingBag動可以提高性能和記憶效率來處理tensor序列。

一個例子

AG_NEWS數據集含有四個標簽，因此類別數為4。

 vocab大小等於vocab長度(包括單詞和ngrams)。類別數目等於標簽數，在AG_NEWS這個例子中為4。

 

批量生成

由於文本長度不同，一個定制的generate_batch()用來生成批量和offsets。該函數傳到collate_fn里面嗎（不會用collate_fn函數的參考這里->）。collate_fn的輸入為：batch_size大小的列表，將其大包圍mini-batch。整個文本輸入批量被整理為list，並連接作為單一tensor作為nn.EmbeddingBag的輸入。offset是一個tensor表征文本tensor中隊里序列的起始索引。Label是tensor保留了整個文本的標簽。

def generate_batch(batch):
    label = torch.tensor([entry[0] for entry in batch])
    text = [entry[1] for entry in batch]
    offsets = [0] + [len(entry) for entry in text]
    # torch.Tensor.cumsum returns the cumulative sum
    # of elements in the dimension dim.
    # torch.Tensor([1.0, 2.0, 3.0]).cumsum(dim=0)

    offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
    text = torch.cat(text)
    return text, offsets, label

 

函數定義與模型訓練評估

推薦pytorch boys利用torch.utils.data.DataLoader來整，並且並行處理也方便（a tutorial is here）。我們利用DataLoader來加載AG_NEWS數據集並用來訓練/評估。

from torch.utils.data import DataLoader

def train_func(sub_train_):

    # Train the model
    train_loss = 0
    train_acc = 0
    data = DataLoader(sub_train_, batch_size=BATCH_SIZE, shuffle=True,
                      collate_fn=generate_batch)
    for i, (text, offsets, cls) in enumerate(data):
        optimizer.zero_grad()
        text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
        output = model(text, offsets)
        loss = criterion(output, cls)
        train_loss += loss.item()
        loss.backward()
        optimizer.step()
        train_acc += (output.argmax(1) == cls).sum().item()

    # Adjust the learning rate
    scheduler.step()

    return train_loss / len(sub_train_), train_acc / len(sub_train_)

def test(data_):
    loss = 0
    acc = 0
    data = DataLoader(data_, batch_size=BATCH_SIZE, collate_fn=generate_batch)
    for text, offsets, cls in data:
        text, offsets, cls = text.to(device), offsets.to(device), cls.to(device)
        with torch.no_grad():
            output = model(text, offsets)
            loss = criterion(output, cls)
            loss += loss.item()
            acc += (output.argmax(1) == cls).sum().item()

    return loss / len(data_), acc / len(data_)

 

划分數據集並訓練模型

因為原始AG_NEWS數據沒有驗證集，將訓練集分為訓練/驗證集合，比例為0.95/0.05。利用pytorch庫中的 torch.utils.data.dataset.random_spllt實現。利用CrossEntropyLoss集合 nn.LogSoftmax() and nn.NLLLoss() 在一個類里。對於訓練C個類別的分類任務是有用的。SGD作為優化器。初始學習率為4.0。StepLR用來調整學習率。

import time
from torch.utils.data.dataset import random_split
N_EPOCHS = 5
min_valid_loss = float('inf')

criterion = torch.nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=4.0)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)

train_len = int(len(train_dataset) * 0.95)
sub_train_, sub_valid_ = \
    random_split(train_dataset, [train_len, len(train_dataset) - train_len])

for epoch in range(N_EPOCHS):

    start_time = time.time()
    train_loss, train_acc = train_func(sub_train_)
    valid_loss, valid_acc = test(sub_valid_)

    secs = int(time.time() - start_time)
    mins = secs / 60
    secs = secs % 60

    print('Epoch: %d' %(epoch + 1), " | time in %d minutes, %d seconds" %(mins, secs))
    print(f'\tLoss: {train_loss:.4f}(train)\t|\tAcc: {train_acc * 100:.1f}%(train)')
    print(f'\tLoss: {valid_loss:.4f}(valid)\t|\tAcc: {valid_acc * 100:.1f}%(valid)')

 

 

利用測試集評估

print('Checking the results of test dataset...')
test_loss, test_acc = test(test_dataset)
print(f'\tLoss: {test_loss:.4f}(test)\t|\tAcc: {test_acc * 100:.1f}%(test)')

 

隨機進行測試

利用最好的模型進行測試，標簽信息：

import re
from torchtext.data.utils import ngrams_iterator
from torchtext.data.utils import get_tokenizer

ag_news_label = {1 : "World",
                 2 : "Sports",
                 3 : "Business",
                 4 : "Sci/Tec"}

def predict(text, model, vocab, ngrams):
    tokenizer = get_tokenizer("basic_english")
    with torch.no_grad():
        text = torch.tensor([vocab[token]
                            for token in ngrams_iterator(tokenizer(text), ngrams)])
        output = model(text, torch.tensor([0]))
        return output.argmax(1).item() + 1

ex_text_str = "MEMPHIS, Tenn. – Four days ago, Jon Rahm was \
    enduring the season’s worst weather conditions on Sunday at The \
    Open on his way to a closing 75 at Royal Portrush, which \
    considering the wind and the rain was a respectable showing. \
    Thursday’s first round at the WGC-FedEx St. Jude Invitational \
    was another story. With temperatures in the mid-80s and hardly any \
    wind, the Spaniard was 13 strokes better in a flawless round. \
    Thanks to his best putting performance on the PGA Tour, Rahm \
    finished with an 8-under 62 for a three-stroke lead, which \
    was even more impressive considering he’d never played the \
    front nine at TPC Southwind."

vocab = train_dataset.get_vocab()
model = model.to("cpu")

print("This is a %s news" %ag_news_label[predict(ex_text_str, model, vocab, 2)])