tokenizer#

class CTRLTokenizer(vocab_file, merges_file, max_len=None, unk_token='<unk>', **kwargs)[source]#

Bases: PretrainedTokenizer

Constructs a CTRL tokenizer based on byte-level Byte-Pair-Encoding.

This tokenizer inherits from PretrainedTokenizer which contains most of the main methods. For more information regarding those methods, please refer to this superclass.

Parameters:

vocab_file (str) – Path to the vocab file. The vocab file contains a mapping from vocabulary strings to indices.
merges_file (str) – Path to the merge file. The merge file is used to split the input sentence into “subword” units. The vocab file is then used to encode those units as intices.
max_len (int, optional) – The maximum value of the input sequence length. Defaults to None.
unk_token (str) – A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to “<unk>”.

property vocab_size#

Size of the base vocabulary (without the added tokens).

Type:: int

get_vocab()[source]#

Returns the vocabulary as a dictionary of token to index.

tokenizer.get_vocab()[token] is equivalent to tokenizer.convert_tokens_to_ids(token) when token is in the vocab.

Returns:: The vocabulary.
Return type:: Dict[str, int]

tokenize(text)[source]#

Converts a string to a list of tokens.

Parameters:: text (str) – The text to be tokenized.
Returns:: A list of string representing converted tokens.
Return type:: List[str]

Example

from paddlenlp.transformers import CTRLTokenizer

tokenizer = CTRLTokenizer.from_pretrained('ctrl')
print(tokenizer.tokenize('Welcome to use PaddlePaddle and PaddleNLP'))
# ['Welcome', 'to', 'use', 'Padd@@', 'le@@', 'Padd@@', 'le', 'and', 'Padd@@', 'le@@', 'N@@', 'LP']

convert_tokens_to_string(tokens)[source]#

Converts a sequence of tokens (list of string) to a single string.

Parameters:: tokens (List[str]) – A sequence of tokens.
Returns:: Converted string.
Return type:: str

Example

from paddlenlp.transformers import CTRLTokenizer

tokenizer = CTRLTokenizer.from_pretrained('crtl')
print(tokenizer.convert_tokens_to_string(['Welcome', 'to', 'use', 'Padd@@', 'le@@', 'Padd@@', 'le', 'and', 'Padd@@', 'le@@', 'N@@', 'LP']))
# 'Welcome to use PaddlePaddle and PaddleNLP'

convert_tokens_to_ids(tokens)[source]#

Converts a single token or a sequence of tokens to an index or a sequence of indices using the vocab.

Parameters:: tokens (str|List[str]|tuple(str)) – A single token or a sequence of tokens.
Returns:: The converted token id or token ids.
Return type:: int|List[int]

Example

from paddlenlp.transformers import CTRLTokenizer

tokenizer = CTRLTokenizer.from_pretrained('crtl')
print(tokenizer.convert_tokens_to_ids(['Welcome', 'to', 'use', 'Padd@@', 'le@@', 'Padd@@', 'le', 'and', 'Padd@@', 'le@@', 'N@@', 'LP']))
# [41116, 3, 191, 40324, 1162, 40324, 992, 2, 40324, 1162, 633, 11135]

convert_ids_to_tokens(ids, skip_special_tokens=False)[source]#

Converts an index or a sequence indices to a single token or a sequence of tokens.

Parameters:

ids (int|List[int]) – The token id (or token ids) to be converted to text.
skip_special_tokens (bool, optional) – Whether or not to skip the special tokens. Defaults to False, which means we don’t skip the special tokens.

Returns:

The converted token or the sequence of tokens.

Return type:

str|List[str]

Example

from paddlenlp.transformers import CTRLTokenizer

tokenizer = CTRLTokenizer.from_pretrained('ctrl')
print(tokenizer.convert_ids_to_tokens([41116, 3, 191, 40324, 1162, 40324, 992, 2, 40324, 1162, 633, 11135]))
# ['Welcome', 'to', 'use', 'Padd@@', 'le@@', 'Padd@@', 'le', 'and', 'Padd@@', 'le@@', 'N@@', 'LP']

save_resources(save_directory)[source]#

Save tokenizer related resources to files under save_directory.

Parameters:: save_directory (str) – Directory to save files into.

tokenizer

Contents

tokenizer#