tokenizer

class BigBirdTokenizer(sentencepiece_model_file, do_lower_case=True, encoding='utf8', unk_token='<unk>', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]')[源代码]

基类:paddlenlp.transformers.tokenizer_utils.PretrainedTokenizer

Constructs an BigBird tokenizer based on SentencePiece.

This tokenizer inherits from PretrainedTokenizer which contains most of the main methods. For more information regarding those methods, please refer to this superclass.

参数
  • sentencepiece_model_file (str) -- The vocabulary file (ends with '.spm') required to instantiate a SentencePiece tokenizer.

  • do_lower_case (bool) -- Whether the text strips accents and convert to Whether or not to lowercase the input when tokenizing. Defaults to`True`.

  • unk_token (str) -- A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to "[UNK]".

  • sep_token (str) -- A special token separating two different sentences in the same input. Defaults to "[SEP]".

  • pad_token (str) -- A special token used to make arrays of tokens the same size for batching purposes. Defaults to "[PAD]".

  • cls_token (str) -- A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to "[CLS]".

  • mask_token (str) -- A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to "[MASK]".

引发

ValueError -- If file sentencepiece_model_file doesn't exist.

property vocab_size

Return the size of vocabulary.

返回

The size of vocabulary.

返回类型

int

convert_tokens_to_string(tokens)[源代码]

Converts a sequence of tokens (list of string) to a single string. Since the usage of WordPiece introducing ## to concat subwords, also removes ## when converting.

参数

tokens (list) -- A list of string representing tokens to be converted.

返回

Converted string from tokens.

返回类型

str

实际案例

from paddlenlp.transformers import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer('He was a puppeteer')
strings = tokenizer.convert_tokens_to_string(tokens)
encode(text, max_seq_len=None, max_pred_len=None, masked_lm_prob=0.15)[源代码]

Returns a tuple containing the encoded sequence and mask information.

参数
  • text (str,list[str] or list[int]) -- The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the tokenize method) or a list of integers (tokenized string ids using the convert_tokens_to_ids method)

  • max_seq_len (int, optional) -- If set to a number, will limit the total sequence returned so that it has a maximum length. If set to None, will not limit the total sequence. Defaults to None.

  • max_pred_len (int, optional) -- If set to a number, will limit the mask sequence returned so that it has a maximum prediction length. If set to None, will not limit the mask sequence.

  • masked_lm_prob (float, optional) -- The probability of the token to be masked. Defaults to 0.15.

返回

Returns tuple (span_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights).

返回类型

tuple

num_special_tokens_to_add(pair=False)[源代码]

Returns the number of added tokens when encoding a sequence with special tokens.

参数

pair (bool) -- Whether the input is a sequence pair or a single sequence. Defaults to False and the input is a single sequence.

返回

Number of tokens added to sequences.

返回类型

int

build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)[源代码]

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

A BigBird sequence has the following format:

  • single sequence: [CLS] X [SEP]

  • pair of sequences: [CLS] A [SEP] B [SEP]

参数
  • token_ids_0 (List[int]) -- List of IDs to which the special tokens will be added.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

返回

List of input_id with the appropriate special tokens.

返回类型

List[int]