tokenizer

class DebertaTokenizer(vocab_file, merges_file, errors='replace', max_len=None, bos_token='[CLS]', eos_token='[SEP]', sep_token='[SEP]', cls_token='[CLS]', unk_token='[UNK]', pad_token='[PAD]', mask_token='[MASK]', add_prefix_space=False, add_bos_token=False, **kwargs)

基类：PretrainedTokenizer

Constructs a DeBERTa tokenizer based on byte-level Byte-Pair-Encoding.

This tokenizer inherits from PretrainedTokenizer which contains most of the main methods. For more information regarding those methods, please refer to this superclass.

参数:

• vocab_file (str) -- Path to the vocab file. The vocab file contains a mapping from vocabulary strings to indices.

• merges_file (str) -- Path to the merge file. The merge file is used to split the input sentence into "subword" units. The vocab file is then used to encode those units as intices.

• errors (str) -- Paradigm to follow when decoding bytes to UTF-8. Defaults to 'replace'.

• max_len (int, optional) -- The maximum value of the input sequence length. Defaults to None.

示例

from paddlenlp.transformers import DebertaTokenizer

tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base')
print(tokenizer('Welcome to use PaddlePaddle and PaddleNLP'))

'''
{'input_ids': [1, 25194, 7, 304, 221, 33151, 510, 33151, 8, 221, 33151, 487, 21992, 2],
'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
'''

property vocab_size

Returns the size of vocabulary.

返回:

The sum of size of vocabulary and the size of speical tokens.

返回类型:

int

convert_ids_to_string(ids)

Converts a single index or a sequence of indices to texts.

参数:

ids (int|List[int]) -- The token id (or token ids) to be converted to text.

返回:

The decoded text.

返回类型:

str

示例

from paddlenlp.transformers import DebertaTokenizer
tokenizer = DebertaTokenizer.from_pretrained('deberta-base')
print(tokenizer.convert_ids_to_string(tokenizer.convert_ids_to_string([14618, 284, 779, 350, 37382, 47, 37382, 290, 350, 37382, 45, 19930]))
# 'Welcome to use PaddlePaddle and PaddleNLP'

create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

A ERNIE sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

参数:

• token_ids_0 (List[int]) -- A list of inputs_ids for the first sequence.

• token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

返回:

List of token_type_id according to the given sequence(s).

返回类型:

List[int]

save_resources(save_directory)

Saves SentencePiece file (ends with '.spm') under save_directory.

参数:

save_directory (str) -- Directory to save files into.

convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.

get_vocab()

Returns the vocabulary as a dictionary of token to index.

tokenizer.get_vocab()[token] is equivalent to tokenizer.convert_tokens_to_ids(token) when token is in the vocab.

返回:

The vocabulary.

返回类型:

Dict[str, int]

prepare_for_tokenization(text, is_split_into_words=False, **kwargs)

Performs any necessary transformations before tokenization.

This method should pop the arguments from kwargs and return the remaining kwargs as well. We test the kwargs at the end of the encoding process to be sure all the arguments have been used.

参数:

• text (str) -- The text to prepare.

• is_split_into_words (bool, optional, defaults to False) -- Whether or not the input is already pre-tokenized (e.g., split into words). If set to True, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification.

• kwargs -- Keyword arguments to use for the tokenization.

返回:

The prepared text and the unused kwargs.

返回类型:

Tuple[str, Dict[str, Any]]

build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

• single sequence: [CLS] X [SEP]

• pair of sequences: [CLS] A [SEP] B [SEP]

参数:

• token_ids_0 (List[int]) -- List of IDs to which the special tokens will be added.

• token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

返回:

List of input_id with the appropriate special tokens.

返回类型:

List[int]

get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer encode methods. :param token_ids_0: List of ids of the first sequence. :type token_ids_0: List[int] :param token_ids_1: Optional second list of IDs for sequence pairs.

Defaults to None.

参数:

already_has_special_tokens (str, optional) -- Whether or not the token list is already formatted with special tokens for the model. Defaults to False.

返回:

The list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

返回类型:

List[int]

build_offset_mapping_with_special_tokens(offset_mapping_0, offset_mapping_1=None)

Build offset map from a pair of offset map by concatenating and adding offsets of special tokens.

A BERT offset_mapping has the following format:

• single sequence: (0,0) X (0,0)

• pair of sequences: (0,0) A (0,0) B (0,0)

参数:

• offset_mapping_ids_0 (List[tuple]) -- List of wordpiece offsets to which the special tokens will be added.

• offset_mapping_ids_1 (List[tuple], optional) -- Optional second list of wordpiece offsets for offset mapping pairs. Defaults to None.

返回:

A list of wordpiece offsets with the appropriate offsets of special tokens.

返回类型:

List[tuple]