tokenizer#
- class ErnieCtmTokenizer(vocab_file, do_lower_case=True, do_basic_tokenize=True, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token_template='[CLS{}]', cls_num=1, mask_token='[MASK]', **kwargs)[源代码]#
-
Construct an ERNIE-CTM tokenizer.
This tokenizer inherits from
PretrainedTokenizer
which contains most of the main methods. For more information regarding those methods, please refer to this superclass.- 参数:
vocab_file (str) -- File path of the vocabulary.
do_lower_case (bool, optional) -- Whether or not to lowercase the input when tokenizing. Defaults to
True
do_basic_tokenize (bool, optional) -- Whether or not to do basic tokenization before WordPiece. Defaults to
True
unk_token (str, optional) -- A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be
unk_token
inorder to be converted to an ID. Defaults to "[UNK]".sep_token (str, optional) -- A special token separating two different sentences in the same input. Defaults to "[SEP]".
pad_token (str, optional) -- A special token used to make arrays of tokens the same size for batching purposes. Defaults to "[PAD]".
cls_token_template (str, optional) -- The template of summary token for multiple summary placeholders. Defaults to
"[CLS{}]"
cls_num (int, optional) -- Summary placeholder used in ernie-ctm model. For catching a sentence global feature from multiple aware. Defaults to
1
.mask_token (str, optional) -- A special token representing a masked token. This is the token used in the masked language modeling task. This is the token which the model will try to predict the original unmasked ones. Defaults to
"[MASK]"
.strip_accents -- (bool, optional): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for
lowercase
(as in the original BERT).
示例
from paddlenlp.transformers import ErnieCtmTokenizer tokenizer = ErnieCtmTokenizer.from_pretrained('ernie-ctm') encoded_inputs = tokenizer('He was a puppeteer') # encoded_inputs: # {'input_ids': [101, 98, 153, 150, 99, 168, 146, 164, 99, 146, 99, 161, 166, 161, # 161, 150, 165, 150, 150, 163, 102], # 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
- property vocab_size#
Return the size of vocabulary.
- 返回:
The size of vocabulary.
- 返回类型:
int
- convert_tokens_to_string(tokens)[源代码]#
Converts a sequence of tokens (list of string) in a single string. Since the usage of WordPiece introducing
##
to concat subwords, also remove##
when converting.- 参数:
tokens (List[str]) -- A list of string representing tokens to be converted.
- 返回:
Converted string from tokens.
- 返回类型:
str
示例
from paddlenlp.transformers import ErnieCtmTokenizer tokenizer = ErnieCtmTokenizer.from_pretrained('ernie-ctm') tokens = tokenizer.tokenize('He was a puppeteer') strings = tokenizer.convert_tokens_to_string(tokens) #he was a puppeteer
- build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)[源代码]#
Build model inputs from a sequence or a pair of sequences for sequence classification tasks by concatenating and add special tokens.
A ERNIE-CTM sequence has the following format:
single sequence: [CLS0][CLS1]... X [SEP]
pair of sequences: [CLS0][CLS1]... X [SEP] X [SEP]
- 参数:
token_ids_0 (List) -- List of IDs to which the special tokens will be added.
token_ids_1 (List, optional) -- Optional second list of IDs for sequence pairs. Defaults to
None
.
- 返回:
The input_id with the appropriate special tokens.
- 返回类型:
List[int]
- get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[源代码]#
Creates a special tokens mask from the input sequences. This method is called when adding special tokens using the tokenizer
encode
method.- 参数:
token_ids_0 (List[int]) -- A list of
inputs_ids
for the first sequence.token_ids_1 (List[int], optional) -- Optional second list of
inputs_ids
for the second sequence. Defaults toNone
.already_has_special_tokens (bool, optional) -- Whether or not the token list already contains special tokens for the model. Defaults to
False
.
- 返回:
A list of integers which is either 0 or 1: 1 for a special token, 0 for a sequence token.
- 返回类型:
List[int]
- create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)[源代码]#
Creates a token_type mask from the input sequences.
If
token_ids_1
is notNone
, then a sequence pair token_type mask has the following format:0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 | first sequence | second sequence |
Else if
token_ids_1
isNone
, then a single sequence token_type mask has the following format:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 | first sequence |
0 stands for the segment id of first segment tokens,
1 stands for the segment id of second segment tokens,
2 stands for the segment id of cls_token.
- 参数:
token_ids_0 (List[int]) -- A list of
inputs_ids
for the first sequence.token_ids_1 (List[int], optional) -- Optional second list of
inputs_ids
for the second sequence. Defaults toNone
.
- 返回:
List of token type IDs according to the given sequence(s).
- 返回类型:
List[int]
- num_special_tokens_to_add(pair=False)[源代码]#
Returns the number of added tokens when encoding a sequence with special tokens.
备注
This encodes inputs and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop.
- 参数:
pair (bool, optional) -- Whether the input is a sequence pair or a single sequence. Defaults to
False
and the input is a single sequence.- 返回:
Number of tokens added to sequences.
- 返回类型:
int