luke#
- modeling
- tokenizer
LukeTokenizerLukeTokenizer.sep_token_idLukeTokenizer.cls_token_idLukeTokenizer.pad_token_idLukeTokenizer.unk_token_idLukeTokenizer.get_entity_vocab()LukeTokenizer.tokenize()LukeTokenizer.convert_tokens_to_string()LukeTokenizer.add_special_tokens()LukeTokenizer.convert_entity_to_id()LukeTokenizer.entity_encode()LukeTokenizer.get_offset_mapping()LukeTokenizer.create_token_type_ids_from_sequences()LukeTokenizer.num_special_tokens_to_add()LukeTokenizer.build_inputs_with_special_tokens()