tokenizer

tokenizer#

class TinyBertTokenizer(vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', tokenize_chinese_chars=True, strip_accents=None, **kwargs)[source]#

Bases: BertTokenizer

Constructs a TinyBert tokenizer. The usage of TinyBertTokenizer is the same as BertTokenizer. For more information regarding those methods, please refer to this superclass.