tokenizer¶
-
class
ErnieMTokenizer
(vocab_file, sentencepiece_model_file, do_lower_case=False, encoding='utf8', unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', **kwargs)[source]¶ Bases:
paddlenlp.transformers.tokenizer_utils.PretrainedTokenizer
Constructs a ErnieM tokenizer. It uses the
sentencepiece
tools to cut the words to sub-words.- Parameters
vocab_file (str) – The file path of the vocabulary.
sentencepiece_model_file (str) – The file path of sentencepiece model.
do_lower_case (str, optional) – Whether or not to lowercase the input when tokenizing. Defaults to`True`.
unk_token (str, optional) – A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be
unk_token
inorder to be converted to an ID. Defaults to “[UNK]”.sep_token (str, optional) – A special token separating two different sentences in the same input. Defaults to “[SEP]”.
pad_token (str, optional) – A special token used to make arrays of tokens the same size for batching purposes. Defaults to “[PAD]”.
cls_token (str, optional) – A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to “[CLS]”.
mask_token (str, optional) – A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to “[MASK]”.
-
get_offset_mapping
(text)[source]¶ Returns the map of tokens and the start and end index of their start and end character. Modified from https://github.com/bojone/bert4keras/blob/master/bert4keras/tokenizers.py#L372 :param text: Input text. :type text: str :param split_tokens: the tokens which has been split which can accelerate the operation. :type split_tokens: Optional[List[str]]
- Returns
The offset map of input text.
- Return type
list
-
property
vocab_size
¶ Return the size of vocabulary.
- Returns
The size of vocabulary.
- Return type
int
-
get_vocab
()[source]¶ Returns the vocabulary as a dictionary of token to index.
tokenizer.get_vocab()[token]
is equivalent totokenizer.convert_tokens_to_ids(token)
whentoken
is in the vocab.- Returns
The vocabulary.
- Return type
Dict[str, int]
-
convert_tokens_to_string
(tokens)[source]¶ Converts a sequence of tokens (strings for sub-words) in a single string.
-
convert_ids_to_string
(ids)[source]¶ Converts a sequence of tokens (strings for sub-words) in a single string.
-
build_inputs_with_special_tokens
(token_ids_0, token_ids_1=None)[source]¶ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.
An ERNIE-M sequence has the following format: - single sequence:
[CLS] X [SEP]
- pair of sequences:[CLS] A [SEP] [SEP] B [SEP]
:param token_ids_0: List of IDs to which the special tokens will be added. :type token_ids_0: List[int] :param token_ids_1: Optional second list of IDs for sequence pairs.Defaults to
None
.- Returns
List of input_id with the appropriate special tokens.
- Return type
List[int]
-
build_offset_mapping_with_special_tokens
(offset_mapping_0, offset_mapping_1=None)[source]¶ Build offset map from a pair of offset map by concatenating and adding offsets of special tokens.
An ERNIE-M offset_mapping has the following format: - single sequence:
(0,0) X (0,0)
- pair of sequences:(0,0) A (0,0) (0,0) B (0,0)
- Parameters
offset_mapping_ids_0 (List[tuple]) – List of char offsets to which the special tokens will be added.
offset_mapping_ids_1 (List[tuple], optional) – Optional second list of wordpiece offsets for offset mapping pairs. Defaults to
None
.
- Returns
List of wordpiece offsets with the appropriate offsets of special tokens.
- Return type
List[tuple]
-
get_special_tokens_mask
(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[source]¶ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer
encode
methods. :param token_ids_0: List of ids of the first sequence. :type token_ids_0: List[int] :param token_ids_1: Optional second list of IDs for sequence pairs.Defaults to
None
.- Parameters
already_has_special_tokens (str, optional) – Whether or not the token list is already formatted with special tokens for the model. Defaults to
False
.- Returns
The list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
- Return type
List[int]
-
create_token_type_ids_from_sequences
(token_ids_0: List[int], token_ids_1: Optional[List[int]] = None) → List[int][source]¶ Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids)
Should be overridden in a subclass if the model has a special way of building those.
- Parameters
token_ids_0 (
List[int]
) – The first tokenized sequence.token_ids_1 (
List[int]
, optional) – The second tokenized sequence.
- Returns
The token type ids.
- Return type
List[int]