tokenizer#

class MPNetTokenizer(vocab_file, do_lower_case=True, bos_token='<s>', eos_token='</s>', unk_token='[UNK]', sep_token='</s>', pad_token='<pad>', cls_token='<s>', mask_token='<mask>', **kwargs)[源代码]#

基类：BertTokenizer

Construct a MPNet tokenizer which is almost identical to BertTokenizer. For more information regarding those methods, please refer to this superclass.

build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)[源代码]#

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens.

A MPNet sequence has the following format:

  • single sequence: <s> X </s>

  • pair of sequences: <s> A </s></s> B </s>

参数:

  • token_ids_0 (List[int]) -- List of IDs to which the special tokens will be added.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

返回:

List of input_id with the appropriate special tokens.

返回类型:

List[int]

get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)[源代码]#

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer encode methods.

参数:

  • token_ids_0 (List[int]) -- A list of inputs_ids for the first sequence.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

  • already_has_special_tokens (bool, optional) -- Whether or not the token list is already formatted with special tokens for the model. Defaults to None.

返回:

The list of integers either be 0 or 1: 1 for a special token, 0 for a sequence token.

返回类型:

List[int]

create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)[源代码]#

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. MPNet does not make use of token type ids, therefore a list of zeros is returned.

参数:

  • token_ids_0 (List[int]) -- A list of inputs_ids for the first sequence.

  • token_ids_1 (List[int], optional) -- Optional second list of IDs for sequence pairs. Defaults to None.

返回:

List of token_type_id according to the given sequence(s).

返回类型:

List[int]

build_offset_mapping_with_special_tokens(offset_mapping_0, offset_mapping_1=None)[源代码]#

Build offset map from a pair of offset map by concatenating and adding offsets of special tokens.

A BERT offset_mapping has the following format:

  • single sequence: (0,0) X (0,0)

  • pair of sequences: (0,0) A (0,0) B (0,0)

参数:

  • offset_mapping_ids_0 (List[tuple]) -- List of wordpiece offsets to which the special tokens will be added.

  • offset_mapping_ids_1 (List[tuple], optional) -- Optional second list of wordpiece offsets for offset mapping pairs. Defaults to None.

返回:

A list of wordpiece offsets with the appropriate offsets of special tokens.

返回类型:

List[tuple]