tokenizer¶
-
class
ConvBertTokenizer
(vocab_file, do_lower_case=True, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', **kwargs)[源代码]¶ 基类:
paddlenlp.transformers.electra.tokenizer.ElectraTokenizer
Construct a ConvBERT tokenizer.
ConvBertTokenizer
is identical toElectraTokenizer
. For more information regarding those methods, please refer to this superclass.