碼上快樂

相關內容    簡體    繁體

配置、使用transformers包

本文轉載自   查看原文   2019-11-28 10:56   2978 

配置、使用transformers包

一、transformers

transformers包又名pytorch-transformers或者pytorch-pretrained-bert。它提供了一些列的STOA模型的實現，包括(Bert、XLNet、RoBERTa等)。下面介紹該包的使用方法：

1、如何安裝

transformers的安裝十分簡單，通過pip命令即可

pip install transformers

也可通過其他方式來安裝，具體可以參考：https://github.com/huggingface/transformers

2、如何使用

使用transformers前需要下載好pytorch(版本>=1.0)或者tensorflow2.0。下面以pytorch為例，來演示使用方法

1、若要導入所有包可以輸入：

import torch
from transformers import *

2、若要導入指定的包可以輸入：

import torch
from transformers import BertModel

3、加載預訓練權重和詞表

UNCASED = './bert-base-uncased'
bert = BertModel.from_pretrained(UNCASED)

注意：加載預訓練權重時需要下載好預訓練的權重文件，一般來說，當緩存文件中沒有所需文件時(第一次使用)，只要網絡沒有問題，就會自動下載。當網絡出現問題的時候，就需要手動下載預訓練權重了。

當緩存中不存在所需文件時，一般會出現提示：
bert-base-uncased-pytorch_model.bin not found in cache

二、手動下載、本地加載預訓練權重

在使用transformers的時候，由於Bert、XLNet的文件都在AWS上存儲，transformers的默認下載地址指向的是AWS，因此在國內下載速度非常慢。需要我們自己手動下載。

1、下載.txt、.json、.bin文件到本地

以Bert為例，相關的.bin文件(預訓練權重)下載地址如下所示：

BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
    'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin",
    'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-pytorch_model.bin",
    'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin",
    'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-pytorch_model.bin",
    'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin",
    'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-pytorch_model.bin",
    'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-pytorch_model.bin",
    'bert-base-german-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-pytorch_model.bin",
    'bert-large-uncased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-pytorch_model.bin",
    'bert-large-cased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-pytorch_model.bin",
    'bert-large-uncased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin",
    'bert-large-cased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-pytorch_model.bin",
    'bert-base-cased-finetuned-mrpc': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-pytorch_model.bin",
    'bert-base-german-dbmdz-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-pytorch_model.bin",
    'bert-base-german-dbmdz-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-pytorch_model.bin",
}

若需要下載.json文件，則下載地址為：

BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json",
    'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-config.json",
    'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json",
    'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-config.json",
    'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json",
    'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json",
    'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-config.json",
    'bert-base-german-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-config.json",
    'bert-large-uncased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-config.json",
    'bert-large-cased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-config.json",
    'bert-large-uncased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json",
    'bert-large-cased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-config.json",
    'bert-base-cased-finetuned-mrpc': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-config.json",
    'bert-base-german-dbmdz-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json",
    'bert-base-german-dbmdz-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json",
}

.txt相關文件(詞表文件)下載地址如下：

PRETRAINED_VOCAB_FILES_MAP = {
    'vocab_file':
    {
        'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt",
        'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt",
        'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt",
        'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt",
        'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-vocab.txt",
        'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt",
        'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt",
        'bert-base-german-cased': "https://int-deepset-models-bert.s3.eu-central-1.amazonaws.com/pytorch/bert-base-german-cased-vocab.txt",
        'bert-large-uncased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-vocab.txt",
        'bert-large-cased-whole-word-masking': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-vocab.txt",
        'bert-large-uncased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-vocab.txt",
        'bert-large-cased-whole-word-masking-finetuned-squad': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-vocab.txt",
        'bert-base-cased-finetuned-mrpc': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-vocab.txt",
        'bert-base-german-dbmdz-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-vocab.txt",
        'bert-base-german-dbmdz-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt",
    }
}

本地使用預訓練的權重文件

假設我們下載好了如下三個文件：
enter description here
首先我們建立一個文件夾，命名為bert-base-uncased，然后將這個三個文件放入這個文件夾，並且對文件進行重命名，重命名時將bert-base-uncased-去除即可。

假設我們訓練文件夾名字為 train.py，我們需要將上面的bert-base-uncased文件夾放到與train.py同級的目錄下面。
若不改名以及調整文件夾位置將會出現:
vocab.txt not found；pytorch_model.bin not found；Model name 'xxx/pytorch_model.bin ' was not found in model name list等錯誤。

之后使用下面的代碼進行測試即可：

UNCASED = './bert-base-uncased'
VOCAB = 'vocab.txt'
tokenizer=BertTokenizer.from_pretrained(os.path.join(UNCASED,VOCAB))
bert = BertModel.from_pretrained(UNCASED)


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 transformers中，關於PreTrainedTokenizer的使用 huggingface 🤗 Transformers的簡單使用 NLP（三十四）：huggingface transformers預訓練模型如何下載至本地，並使用？ Pytorch-Bert預訓練模型的使用（調用transformers） HuggingFace-transformers系列的介紹以及在下游任務中的使用 如何使用BERT預訓練模型提取文本特征？—基於transformers HuggingFace-transformers系列的介紹以及在下游任務中的使用 Pytorch-Bert預訓練模型的使用（調用transformers） Transformers 簡介（上） 關於miniconda的安裝，配置以及包批量安裝和使用
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM