tokenizer¶
-
class
MPNetTokenizer
(vocab_file, do_lower_case=True, bos_token='<s>', eos_token='</s>', unk_token='[UNK]', sep_token='</s>', pad_token='<pad>', cls_token='<s>', mask_token='<mask>', **kwargs)[source]¶ Bases:
paddlenlp.transformers.bert.tokenizer.BertTokenizer
Construct a MPNet tokenizer which is almost identical to
BertTokenizer
. For more information regarding those methods, please refer to this superclass.-
build_inputs_with_special_tokens
(token_ids_0, token_ids_1=None)[source]¶ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.
A MPNet sequence has the following format:
single sequence:
<s> X </s>
pair of sequences:
<s> A </s></s> B </s>
- Parameters
token_ids_0 (List[int]) – List of IDs to which the special tokens will be added.
token_ids_1 (List[int], optional) – Optional second list of IDs for sequence pairs. Defaults to None.
- Returns
List of input_id with the appropriate special tokens.
- Return type
List[int]
-
get_special_tokens_mask
(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[source]¶ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer
encode
methods.- Parameters
token_ids_0 (List[int]) – A list of
inputs_ids
for the first sequence.token_ids_1 (List[int], optional) – Optional second list of IDs for sequence pairs. Defaults to None.
already_has_special_tokens (bool, optional) – Whether or not the token list is already formatted with special tokens for the model. Defaults to None.
- Returns
The list of integers either be 0 or 1: 1 for a special token, 0 for a sequence token.
- Return type
List[int]
-
create_token_type_ids_from_sequences
(token_ids_0, token_ids_1=None)[source]¶ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. MPNet does not make use of token type ids, therefore a list of zeros is returned.
- Parameters
token_ids_0 (List[int]) – A list of
inputs_ids
for the first sequence.token_ids_1 (List[int], optional) – Optional second list of IDs for sequence pairs. Defaults to None.
- Returns
List of token_type_id according to the given sequence(s).
- Return type
List[int]
-