class BigBirdTokenizer(sentencepiece_model_file, do_lower_case=False, remove_space=True, keep_accents=True, eos_token='</s>', unk_token='<unk>', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', extra_ids=100, additional_special_tokens=[], sp_model_kwargs=None, encoding='utf8', **kwargs)[源代码]


Constructs an BigBird tokenizer based on SentencePiece.

This tokenizer inherits from PretrainedTokenizer which contains most of the main methods. For more information regarding those methods, please refer to this superclass.

  • sentencepiece_model_file (str) -- The vocabulary file (ends with '.spm') required to instantiate a SentencePiece tokenizer.

  • do_lower_case (bool) -- Whether the text strips accents and convert to Whether or not to lowercase the input when tokenizing. Defaults to`True`.

  • unk_token (str) -- A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to "[UNK]".

  • sep_token (str) -- A special token separating two different sentences in the same input. Defaults to "[SEP]".

  • pad_token (str) -- A special token used to make arrays of tokens the same size for batching purposes. Defaults to "[PAD]".

  • cls_token (str) -- A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to "[CLS]".

  • mask_token (str) -- A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to "[MASK]".


ValueError -- If file sentencepiece_model_file doesn't exist.

property vocab_size

Size of the base vocabulary (without the added tokens).



build_inputs_with_special_tokens(token_ids_0, token_ids_1)[源代码]

Build model inputs from a sequence or a pair of sequence.

An BigBird sequence has the following format:

  • single sequence: X </s>

  • pair of sequences: A </s> B </s>

  • token_ids_0 (List[int]) -- List of IDs to which the special tokens will be added.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.


List of input_id with the appropriate special tokens.



build_offset_mapping_with_special_tokens(offset_mapping_0, offset_mapping_1=None)[源代码]

Build offset map from a pair of offset map by concatenating and adding offsets of special tokens.

Should be overridden in a subclass if the model has a special way of building those.

  • offset_mapping_0 (List[tuple]) -- List of char offsets to which the special tokens will be added.

  • offset_mapping_1 (List[tuple], optional) -- Optional second list of char offsets for offset mapping pairs.


List of char offsets with the appropriate offsets of special tokens.



create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)[源代码]

Create a mask from the two sequences.

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

  • token_ids_0 (List[int]) -- List of IDs.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs.


List of token_type_id according to the given sequence(s).



get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[源代码]

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer encode methods.

  • token_ids_0 (List[int]) -- List of ids of the first sequence.

  • token_ids_1 (List[int], optional) -- List of ids of the second sequence.

  • already_has_special_tokens (bool, optional) -- Whether or not the token list is already formatted with special tokens for the model. Defaults to None.


The list of integers in the range [0, 1]:

1 for a special token, 0 for a sequence token.




Converts a sequence of tokens (string) in a single string.