tokenizer

tokenizer#

class DalleBartTokenizer(vocab_file, merges_file, wiki_word_frequency_file, normalize_text=True, errors='replace', max_len=None, bos_token='<s>', eos_token='</s>', cls_token='<s>', sep_token='</s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', **kwargs)[source]#

Bases: GPTTokenizer

Construct a DalleBart tokenizer based on byte-level Byte-Pair-Encoding.

This tokenizer inherits from GPTTokenizer. For more information regarding those methods, please refer to this superclass.

Parameters:

  • vocab_file (str) – Path to the vocabulary file. The vocab file contains a mapping from vocabulary strings to indices.

  • merges_file (str) – Path to the merge file. The merge file is used to split the input sentence into “subword” units. The vocab file is then used to encode those units as intices.

  • wiki_word_frequency_file (str) – Path to the wiki_word_frequency file when we need normlize text.

  • errors (str) – Paradigm to follow when decoding bytes to UTF-8. Defaults to 'replace'.

  • max_len (int, optional) – The maximum value of the input sequence length. Defaults to None.

  • bos_token (str, optional) – The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. Defaults to "<s>".

  • eos_token (str, optional) – A special token representing the end of a sequence that was used during pretraining. Defaults to "</s>".

  • cls_token (str, optional) – A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to "<s>".

  • sep_token (str, optional) – A special token separating two different sentences in the same input. Defaults to "</s>".

  • unk_token (str, optional) – A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to "<unk>".

  • pad_token (str, optional) – A special token used to make arrays of tokens the same size for batching purposes. Defaults to "<pad>".

  • mask_token (str, optional) – A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to "<mask>".

Examples

from paddlenlp.transformers import DalleBartTokenizer

tokenizer = DalleBartTokenizer.from_pretrained('dalle-mini')
print(tokenizer('Donald Trump in Animal Crossing'))

# {'input_ids': [0, 7083, 3252, 91, 2203, 7807, 2]}
build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)[source]#

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[source]#

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer encode methods.

create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)[source]#

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

Contents