roformer#
- modeling
- tokenizer
RoFormerTokenizer
RoFormerTokenizer.vocab_size
RoFormerTokenizer.convert_tokens_to_string()
RoFormerTokenizer.num_special_tokens_to_add()
RoFormerTokenizer.build_inputs_with_special_tokens()
RoFormerTokenizer.build_offset_mapping_with_special_tokens()
RoFormerTokenizer.create_token_type_ids_from_sequences()
RoFormerTokenizer.get_special_tokens_mask()
RoFormerTokenizer.get_vocab()
JiebaBasicTokenizer