tokenizer

Tokenization class for ALBERT model.

class AlbertTokenizer(vocab_file, sentencepiece_model_file, do_lower_case=True, remove_space=True, keep_accents=False, bos_token='[CLS]', eos_token='[SEP]', unk_token='<unk>', sep_token='[SEP]', pad_token='<pad>', cls_token='[CLS]', mask_token='[MASK]', **kwargs)[source]

Bases: paddlenlp.transformers.tokenizer_utils.PretrainedTokenizer

Constructs an Albert tokenizer based on SentencePiece or BertTokenizer.

This tokenizer inherits from PretrainedTokenizer which contains most of the main methods. For more information regarding those methods, please refer to this superclass.

Parameters

  • vocab_file (str) – The vocabulary file path (ends with ‘.txt’) required to instantiate a WordpieceTokenizer.

  • sentence_model_file (str) – The vocabulary file (ends with ‘.spm’) required to instantiate a SentencePiece tokenizer.

  • do_lower_case (bool) – Whether or not to lowercase the input when tokenizing. Defaults to True.

  • remove_space (bool) – Whether or note to remove space when tokenizing. Defaults to True.

  • keep_accents (bool) – Whether or note to keep accents when tokenizing. Defaults to False.

  • unk_token (str) – A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to “[UNK]”.

  • sep_token (str) – A special token separating two different sentences in the same input. Defaults to “[SEP]”.

  • pad_token (str) – A special token used to make arrays of tokens the same size for batching purposes. Defaults to “[PAD]”.

  • cls_token (str) – A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to “[CLS]”.

  • mask_token (str) – A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to “[MASK]”.

Examples

from paddlenlp.transformers import AlbertTokenizer
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v1')
tokens = tokenizer('He was a puppeteer')
'''
{'input_ids': [2, 24, 23, 21, 10956, 7911, 3],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0]}
'''
property vocab_size

Return the size of vocabulary.

Returns

The size of vocabulary.

Return type

int

tokenize(text)[source]

Converts a string to a list of tokens.

Parameters

text (str) – The text to be tokenized.

Returns

A list of string representing converted tokens.

Return type

List(str)

Examples

from paddlenlp.transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-wwm-ext')
tokens = tokenizer.tokenize('He was a puppeteer')
convert_tokens_to_string(tokens)[source]

Converts a sequence of tokens (list of string) to a single string.

Parameters

tokens (list) – A list of string representing tokens to be converted.

Returns

Converted string from tokens.

Return type

str

Examples

from paddlenlp.transformers import AlbertTokenizer

tokenizer = AlbertTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer.tokenize('He was a puppeteer')
'''
['▁he', '▁was', '▁a', '▁puppet', 'eer']
'''
strings = tokenizer.convert_tokens_to_string(tokens)
'''
he was a puppeteer
'''
num_special_tokens_to_add(pair=False)[source]

Returns the number of added tokens when encoding a sequence with special tokens.

Parameters

pair (bool) – Whether the input is a sequence pair or a single sequence. Defaults to False and the input is a single sequence.

Returns

Number of tokens added to sequences.

Return type

int

build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)[source]

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

An Albert sequence has the following format:

  • single sequence: [CLS] X [SEP]

  • pair of sequences: [CLS] A [SEP] B [SEP]

Parameters

  • token_ids_0 (List[int]) – List of IDs to which the special tokens will be added.

  • token_ids_1 (List[int], optional) – Optional second list of IDs for sequence pairs. Defaults to None.

Returns

List of input_id with the appropriate special tokens.

Return type

List[int]

build_offset_mapping_with_special_tokens(offset_mapping_0, offset_mapping_1=None)[source]

Build offset map from a pair of offset map by concatenating and adding offsets of special tokens.

A Albert offset_mapping has the following format:

  • single sequence: (0,0) X (0,0)

  • pair of sequences: (0,0) A (0,0) B (0,0)

Parameters

  • offset_mapping_ids_0 (List[tuple]) – List of wordpiece offsets to which the special tokens will be added.

  • offset_mapping_ids_1 (List[tuple], optional) – Optional second list of wordpiece offsets for offset mapping pairs. Defaults to None.

Returns

A list of wordpiece offsets with the appropriate offsets of special tokens.

Return type

List[tuple]

get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[source]

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer encode methods.

Parameters

  • token_ids_0 (List[int]) – A list of inputs_ids for the first sequence.

  • token_ids_1 (List[int], optional) – Optional second list of IDs for sequence pairs. Defaults to None.

  • already_has_special_tokens (bool, optional) – Whether or not the token list is already formatted with special tokens for the model. Defaults to None.

Returns

The list of integers either be 0 or 1: 1 for a special token, 0 for a sequence token.

Return type

List[int]

create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)[source]

Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids)

Should be overridden in a subclass if the model has a special way of building those.

Parameters

  • token_ids_0 (List[int]) – The first tokenized sequence.

  • token_ids_1 (List[int], optional) – The second tokenized sequence.

Returns

The token type ids.

Return type

List[int]

save_resources(save_directory)[source]

Save tokenizer related resources to resource_files_names indicating files under save_directory by copying directly. Override it if necessary.

Parameters

save_directory (str) – Directory to save files into.