paddlenlp.transformers
- albert
- modeling
- tokenizer
AlbertTokenizerAlbertTokenizer.vocab_sizeAlbertTokenizer.tokenize()AlbertTokenizer.convert_tokens_to_string()AlbertTokenizer.num_special_tokens_to_add()AlbertTokenizer.build_inputs_with_special_tokens()AlbertTokenizer.build_offset_mapping_with_special_tokens()AlbertTokenizer.get_special_tokens_mask()AlbertTokenizer.create_token_type_ids_from_sequences()AlbertTokenizer.save_resources()
- artist
- auto
- modeling
AutoBackboneAutoModelAutoModelForPretrainingAutoModelForSequenceClassificationAutoModelForTokenClassificationAutoModelForQuestionAnsweringAutoModelForMultipleChoiceAutoModelForMaskedLMAutoModelForCausalLMAutoEncoderAutoDecoderAutoGeneratorAutoDiscriminatorAutoModelForConditionalGeneration
- tokenizer
- modeling
- bart
- modeling
- tokenizer
BartTokenizerBartTokenizer.build_inputs_with_special_tokens()BartTokenizer.get_special_tokens_mask()BartTokenizer.create_token_type_ids_from_sequences()BartTokenizer.get_vocab()BartTokenizer.vocab_sizeBartTokenizer.convert_ids_to_string()BartTokenizer.save_resources()BartTokenizer.convert_tokens_to_string()BartTokenizer.build_offset_mapping_with_special_tokens()
- bert
- fast_tokenizer
- modeling
- tokenizer
BasicTokenizerBertTokenizerBertTokenizer.vocab_sizeBertTokenizer.get_vocab()BertTokenizer.convert_tokens_to_string()BertTokenizer.num_special_tokens_to_add()BertTokenizer.build_inputs_with_special_tokens()BertTokenizer.build_offset_mapping_with_special_tokens()BertTokenizer.create_token_type_ids_from_sequences()BertTokenizer.get_special_tokens_mask()
WordpieceTokenizer
- bert_japanese
- bigbird
- blenderbot
- blenderbot_small
- chinesebert
- codegen
- convbert
- modeling
ConvBertModelConvBertForMaskedLMConvBertPretrainedModelConvBertForTotalPretrainingConvBertDiscriminatorConvBertGeneratorConvBertClassificationHeadConvBertForSequenceClassificationConvBertForTokenClassificationConvBertPretrainingCriterionConvBertForQuestionAnsweringConvBertForMultipleChoiceConvBertForPretraining
- tokenizer
- modeling
- ctrl
- dallebart
- distilbert
- electra
- modeling
ElectraModelElectraPretrainedModelElectraForTotalPretrainingElectraDiscriminatorElectraGeneratorElectraClassificationHeadElectraForSequenceClassificationElectraForTokenClassificationElectraPretrainingCriterionElectraForMultipleChoiceElectraForQuestionAnsweringElectraForMaskedLMElectraForPretrainingErnieHealthForTotalPretrainingErnieHealthPretrainingCriterionErnieHealthDiscriminator
- tokenizer
ElectraTokenizerElectraTokenizer.vocab_sizeElectraTokenizer.get_vocab()ElectraTokenizer.convert_tokens_to_string()ElectraTokenizer.num_special_tokens_to_add()ElectraTokenizer.build_inputs_with_special_tokens()ElectraTokenizer.build_offset_mapping_with_special_tokens()ElectraTokenizer.create_token_type_ids_from_sequences()ElectraTokenizer.get_special_tokens_mask()
- modeling
- ernie
- fast_tokenizer
- modeling
- tokenizer
ErnieTokenizerErnieTokenizer.vocab_sizeErnieTokenizer.extend_chinese_char()ErnieTokenizer.get_vocab()ErnieTokenizer.convert_tokens_to_string()ErnieTokenizer.num_special_tokens_to_add()ErnieTokenizer.build_inputs_with_special_tokens()ErnieTokenizer.build_offset_mapping_with_special_tokens()ErnieTokenizer.create_token_type_ids_from_sequences()ErnieTokenizer.get_special_tokens_mask()
ErnieTinyTokenizerErnieTinyTokenizer.vocab_sizeErnieTinyTokenizer.convert_tokens_to_string()ErnieTinyTokenizer.save_resources()ErnieTinyTokenizer.num_special_tokens_to_add()ErnieTinyTokenizer.build_inputs_with_special_tokens()ErnieTinyTokenizer.build_offset_mapping_with_special_tokens()ErnieTinyTokenizer.create_token_type_ids_from_sequences()ErnieTinyTokenizer.get_special_tokens_mask()ErnieTinyTokenizer.get_vocab()
- ernie_ctm
- ernie_doc
- ernie_gen
- ernie_gram
- ernie_m
- modeling
- tokenizer
ErnieMTokenizerErnieMTokenizer.get_offset_mapping()ErnieMTokenizer.vocab_sizeErnieMTokenizer.get_vocab()ErnieMTokenizer.clean_text()ErnieMTokenizer.convert_tokens_to_string()ErnieMTokenizer.convert_ids_to_string()ErnieMTokenizer.build_inputs_with_special_tokens()ErnieMTokenizer.build_offset_mapping_with_special_tokens()ErnieMTokenizer.get_special_tokens_mask()ErnieMTokenizer.create_token_type_ids_from_sequences()ErnieMTokenizer.is_ch_char()ErnieMTokenizer.is_alpha()ErnieMTokenizer.is_punct()ErnieMTokenizer.is_whitespace()
- fnet
- funnel
- gau_alpha
- modeling
- tokenizer
GAUAlphaTokenizerGAUAlphaTokenizer.vocab_sizeGAUAlphaTokenizer.convert_tokens_to_string()GAUAlphaTokenizer.num_special_tokens_to_add()GAUAlphaTokenizer.build_inputs_with_special_tokens()GAUAlphaTokenizer.build_offset_mapping_with_special_tokens()GAUAlphaTokenizer.create_token_type_ids_from_sequences()GAUAlphaTokenizer.get_special_tokens_mask()GAUAlphaTokenizer.get_vocab()
- gpt
- layoutlm
- layoutlmv2
- layoutxlm
- modeling
- tokenizer
LayoutXLMTokenizerLayoutXLMTokenizer.build_inputs_with_special_tokens()LayoutXLMTokenizer.get_special_tokens_mask()LayoutXLMTokenizer.create_token_type_ids_from_sequences()LayoutXLMTokenizer.vocab_sizeLayoutXLMTokenizer.get_vocab()LayoutXLMTokenizer.convert_tokens_to_string()LayoutXLMTokenizer.num_special_tokens_to_add()
- visual_backbone
- luke
- modeling
- tokenizer
LukeTokenizerLukeTokenizer.sep_token_idLukeTokenizer.cls_token_idLukeTokenizer.pad_token_idLukeTokenizer.unk_token_idLukeTokenizer.get_entity_vocab()LukeTokenizer.tokenize()LukeTokenizer.convert_tokens_to_string()LukeTokenizer.add_special_tokens()LukeTokenizer.convert_entity_to_id()LukeTokenizer.entity_encode()LukeTokenizer.get_offset_mapping()LukeTokenizer.create_token_type_ids_from_sequences()LukeTokenizer.num_special_tokens_to_add()LukeTokenizer.build_inputs_with_special_tokens()
- mbart
- modeling
- tokenizer
MBartTokenizerMBartTokenizer.save_resources()MBartTokenizer.vocab_sizeMBartTokenizer.get_vocab()MBartTokenizer.convert_tokens_to_string()MBartTokenizer.convert_ids_to_string()MBartTokenizer.get_special_tokens_mask()MBartTokenizer.build_inputs_with_special_tokens()MBartTokenizer.build_offset_mapping_with_special_tokens()MBartTokenizer.set_src_lang_special_tokens()MBartTokenizer.set_tgt_lang_special_tokens()
MBart50TokenizerMBart50Tokenizer.save_resources()MBart50Tokenizer.get_vocab()MBart50Tokenizer.vocab_sizeMBart50Tokenizer.convert_tokens_to_string()MBart50Tokenizer.convert_ids_to_string()MBart50Tokenizer.get_special_tokens_mask()MBart50Tokenizer.build_inputs_with_special_tokens()MBart50Tokenizer.build_offset_mapping_with_special_tokens()MBart50Tokenizer.set_src_lang_special_tokens()MBart50Tokenizer.set_tgt_lang_special_tokens()
- megatronbert
- modeling
- tokenizer
- mobilebert
- mpnet
- nezha
- modeling
- tokenizer
NeZhaTokenizerNeZhaTokenizer.vocab_sizeNeZhaTokenizer.convert_tokens_to_string()NeZhaTokenizer.num_special_tokens_to_add()NeZhaTokenizer.build_inputs_with_special_tokens()NeZhaTokenizer.build_offset_mapping_with_special_tokens()NeZhaTokenizer.create_token_type_ids_from_sequences()NeZhaTokenizer.get_special_tokens_mask()NeZhaTokenizer.get_vocab()
- opt
- ppminilm
- modeling
- tokenizer
PPMiniLMTokenizerPPMiniLMTokenizer.vocab_sizePPMiniLMTokenizer.get_vocab()PPMiniLMTokenizer.convert_tokens_to_string()PPMiniLMTokenizer.num_special_tokens_to_add()PPMiniLMTokenizer.build_inputs_with_special_tokens()PPMiniLMTokenizer.build_offset_mapping_with_special_tokens()PPMiniLMTokenizer.create_token_type_ids_from_sequences()PPMiniLMTokenizer.get_special_tokens_mask()
- prophetnet
- modeling
- tokenizer
load_vocab()ProphetNetTokenizerProphetNetTokenizer.vocab_sizeProphetNetTokenizer.get_vocab()ProphetNetTokenizer.tokenize()ProphetNetTokenizer.convert_tokens_to_ids()ProphetNetTokenizer.convert_ids_to_tokens()ProphetNetTokenizer.convert_tokens_to_string()ProphetNetTokenizer.get_special_tokens_mask()ProphetNetTokenizer.create_token_type_ids_from_sequences()ProphetNetTokenizer.build_inputs_with_special_tokens()ProphetNetTokenizer.save_vocabulary()
- reformer
- rembert
- roberta
- modeling
- tokenizer
RobertaTokenizerRobertaChineseTokenizerRobertaChineseTokenizer.vocab_sizeRobertaChineseTokenizer.get_vocab()RobertaChineseTokenizer.convert_tokens_to_string()RobertaChineseTokenizer.num_special_tokens_to_add()RobertaChineseTokenizer.build_inputs_with_special_tokens()RobertaChineseTokenizer.build_offset_mapping_with_special_tokens()RobertaChineseTokenizer.create_token_type_ids_from_sequences()RobertaChineseTokenizer.get_special_tokens_mask()
RobertaBPETokenizerRobertaBPETokenizer.get_vocab()RobertaBPETokenizer.build_inputs_with_special_tokens()RobertaBPETokenizer.get_offset_mapping()RobertaBPETokenizer.build_offset_mapping_with_special_tokens()RobertaBPETokenizer.get_special_tokens_mask()RobertaBPETokenizer.create_token_type_ids_from_sequences()RobertaBPETokenizer.convert_tokens_to_string()RobertaBPETokenizer.num_special_tokens_to_add()RobertaBPETokenizer.prepare_for_tokenization()
- roformer
- modeling
- tokenizer
RoFormerTokenizerRoFormerTokenizer.vocab_sizeRoFormerTokenizer.convert_tokens_to_string()RoFormerTokenizer.num_special_tokens_to_add()RoFormerTokenizer.build_inputs_with_special_tokens()RoFormerTokenizer.build_offset_mapping_with_special_tokens()RoFormerTokenizer.create_token_type_ids_from_sequences()RoFormerTokenizer.get_special_tokens_mask()RoFormerTokenizer.get_vocab()
JiebaBasicTokenizer
- roformerv2
- modeling
- tokenizer
RoFormerv2TokenizerRoFormerv2Tokenizer.vocab_sizeRoFormerv2Tokenizer.convert_tokens_to_string()RoFormerv2Tokenizer.num_special_tokens_to_add()RoFormerv2Tokenizer.build_inputs_with_special_tokens()RoFormerv2Tokenizer.build_offset_mapping_with_special_tokens()RoFormerv2Tokenizer.create_token_type_ids_from_sequences()RoFormerv2Tokenizer.get_special_tokens_mask()RoFormerv2Tokenizer.get_vocab()
- semantic_search
- skep
- modeling
- tokenizer
SkepTokenizerSkepTokenizer.vocab_sizeSkepTokenizer.num_special_tokens_to_add()SkepTokenizer.build_offset_mapping_with_special_tokens()SkepTokenizer.build_inputs_with_special_tokens()SkepTokenizer.create_token_type_ids_from_sequences()SkepTokenizer.save_resources()SkepTokenizer.convert_tokens_to_string()SkepTokenizer.get_special_tokens_mask()SkepTokenizer.get_vocab()
- squeezebert
- modeling
- tokenizer
SqueezeBertTokenizerSqueezeBertTokenizer.vocab_sizeSqueezeBertTokenizer.get_vocab()SqueezeBertTokenizer.convert_tokens_to_string()SqueezeBertTokenizer.num_special_tokens_to_add()SqueezeBertTokenizer.build_inputs_with_special_tokens()SqueezeBertTokenizer.build_offset_mapping_with_special_tokens()SqueezeBertTokenizer.create_token_type_ids_from_sequences()SqueezeBertTokenizer.get_special_tokens_mask()
- t5
- modeling
- tokenizer
T5TokenizerT5Tokenizer.vocab_sizeT5Tokenizer.build_inputs_with_special_tokens()T5Tokenizer.build_offset_mapping_with_special_tokens()T5Tokenizer.create_token_type_ids_from_sequences()T5Tokenizer.get_special_tokens_mask()T5Tokenizer.convert_tokens_to_string()T5Tokenizer.batch_decode()T5Tokenizer.clean_up_tokenization()
- tinybert
- transformer
- unified_transformer
- convert
- modeling
- tokenizer
UnifiedTransformerTokenizerUnifiedTransformerTokenizer.vocab_sizeUnifiedTransformerTokenizer.get_vocab()UnifiedTransformerTokenizer.convert_tokens_to_string()UnifiedTransformerTokenizer.convert_ids_to_string()UnifiedTransformerTokenizer.num_special_tokens_to_add()UnifiedTransformerTokenizer.build_inputs_with_special_tokens()UnifiedTransformerTokenizer.build_offset_mapping_with_special_tokens()UnifiedTransformerTokenizer.create_token_type_ids_from_sequences()UnifiedTransformerTokenizer.get_special_tokens_mask()UnifiedTransformerTokenizer.save_resources()UnifiedTransformerTokenizer.load_vocabulary()UnifiedTransformerTokenizer.dialogue_encode()
- unimo
- modeling
- tokenizer
UNIMOTokenizerUNIMOTokenizer.vocab_sizeUNIMOTokenizer.load_vocabulary()UNIMOTokenizer.get_vocab()UNIMOTokenizer.convert_tokens_to_string()UNIMOTokenizer.num_special_tokens_to_add()UNIMOTokenizer.build_inputs_with_special_tokens()UNIMOTokenizer.merge_subword()UNIMOTokenizer.build_offset_mapping_with_special_tokens()UNIMOTokenizer.create_token_type_ids_from_sequences()UNIMOTokenizer.gen_encode()
- xlm
- modeling
- tokenizer
XLMTokenizerXLMTokenizer.ja_tokenize()XLMTokenizer.vocab_sizeXLMTokenizer.get_vocab()XLMTokenizer.tokenize()XLMTokenizer.convert_tokens_to_string()XLMTokenizer.build_inputs_with_special_tokens()XLMTokenizer.create_token_type_ids_from_sequences()XLMTokenizer.get_special_tokens_mask()XLMTokenizer.save_resources()
- xlnet
- modeling
- tokenizer
XLNetTokenizerXLNetTokenizer.sp_modelXLNetTokenizer.vocab_sizeXLNetTokenizer.get_vocab()XLNetTokenizer.convert_tokens_to_string()XLNetTokenizer.num_special_tokens_to_add()XLNetTokenizer.build_inputs_with_special_tokens()XLNetTokenizer.build_offset_mapping_with_special_tokens()XLNetTokenizer.get_special_tokens_mask()XLNetTokenizer.create_token_type_ids_from_sequences()XLNetTokenizer.save_resources()
- attention_utils
- convert_slow_tokenizer
- distill_utils
- export
- generation_utils
- model_outputs
tuple_output()convert_encoder_output()ModelOutputBaseModelOutputBaseModelOutputWithNoAttentionBaseModelOutputWithPoolingBaseModelOutputWithPastBaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPoolingAndCrossAttentionsSequenceClassifierOutputTokenClassifierOutputQuestionAnsweringModelOutputMultipleChoiceModelOutputMaskedLMOutputCausalLMOutputWithPastCausalLMOutputWithCrossAttentionsSeq2SeqModelOutputSeq2SeqLMOutputSeq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputSequenceClassifierOutputWithPastBackboneOutputBaseModelOutputWithPoolingAndNoAttentionImageClassifierOutputWithNoAttentionDepthEstimatorOutputSemanticSegmenterOutput
- model_utils
PretrainedModelPretrainedModel.init_weights()PretrainedModel.base_modelPretrainedModel.model_name_listPretrainedModel.get_input_embeddings()PretrainedModel.set_input_embeddings()PretrainedModel.get_output_embeddings()PretrainedModel.tie_weights()PretrainedModel.resize_position_embeddings()PretrainedModel.constructed_from_pretrained_config()PretrainedModel.save_model_config()PretrainedModel.save_to_hf_hub()PretrainedModel.resize_token_embeddings()PretrainedModel.from_pretrained()PretrainedModel.save_pretrained()
register_base_model()
- optimization
- sentencepiece_model_pb2
- tokenizer_utils
PretrainedTokenizerPretrainedTokenizer.vocab_sizePretrainedTokenizer.get_added_vocab()PretrainedTokenizer.prepare_for_tokenization()PretrainedTokenizer.tokenize()PretrainedTokenizer.convert_tokens_to_string()PretrainedTokenizer.load_vocabulary()PretrainedTokenizer.save_vocabulary()PretrainedTokenizer.get_special_tokens_mask()PretrainedTokenizer.num_special_tokens_to_add()PretrainedTokenizer.get_offset_mapping()
BPETokenizertokenize_chinese_chars()is_chinese_char()normalize_chars()tokenize_special_chars()convert_to_unicode()
- tokenizer_utils_base
AddedTokenFastEncodingExplicitEnumPaddingStrategyTensorTypeto_py_obj()TruncationStrategyCharSpanTokenSpanBatchEncodingBatchEncoding.n_sequencesBatchEncoding.is_fastBatchEncoding.keys()BatchEncoding.values()BatchEncoding.items()BatchEncoding.encodingsBatchEncoding.tokens()BatchEncoding.sequence_ids()BatchEncoding.words()BatchEncoding.word_ids()BatchEncoding.token_to_sequence()BatchEncoding.token_to_word()BatchEncoding.word_to_tokens()BatchEncoding.token_to_chars()BatchEncoding.char_to_token()BatchEncoding.word_to_chars()BatchEncoding.char_to_word()BatchEncoding.convert_to_tensors()
SpecialTokensMixinSpecialTokensMixin.sanitize_special_tokens()SpecialTokensMixin.add_special_tokens()SpecialTokensMixin.add_tokens()SpecialTokensMixin.bos_tokenSpecialTokensMixin.eos_tokenSpecialTokensMixin.unk_tokenSpecialTokensMixin.sep_tokenSpecialTokensMixin.pad_tokenSpecialTokensMixin.cls_tokenSpecialTokensMixin.mask_tokenSpecialTokensMixin.additional_special_tokensSpecialTokensMixin.pad_token_type_idSpecialTokensMixin.bos_token_idSpecialTokensMixin.eos_token_idSpecialTokensMixin.unk_token_idSpecialTokensMixin.sep_token_idSpecialTokensMixin.pad_token_idSpecialTokensMixin.cls_token_idSpecialTokensMixin.mask_token_idSpecialTokensMixin.additional_special_tokens_idsSpecialTokensMixin.special_tokens_mapSpecialTokensMixin.special_tokens_map_extendedSpecialTokensMixin.all_special_tokensSpecialTokensMixin.all_special_tokens_extendedSpecialTokensMixin.all_special_ids
PretrainedTokenizerBasePretrainedTokenizerBase.max_len_single_sentencePretrainedTokenizerBase.max_len_sentences_pairPretrainedTokenizerBase.get_vocab()PretrainedTokenizerBase.from_pretrained()PretrainedTokenizerBase.save_pretrained()PretrainedTokenizerBase.save_resources()PretrainedTokenizerBase.save_to_hf_hub()PretrainedTokenizerBase.tokenize()PretrainedTokenizerBase.__call__()PretrainedTokenizerBase.encode()PretrainedTokenizerBase.batch_encode()PretrainedTokenizerBase.pad()PretrainedTokenizerBase.create_token_type_ids_from_sequences()PretrainedTokenizerBase.build_inputs_with_special_tokens()PretrainedTokenizerBase.build_offset_mapping_with_special_tokens()PretrainedTokenizerBase.prepare_for_model()PretrainedTokenizerBase.truncate_sequences()PretrainedTokenizerBase.convert_tokens_to_string()PretrainedTokenizerBase.batch_decode()PretrainedTokenizerBase.decode()PretrainedTokenizerBase.get_special_tokens_mask()PretrainedTokenizerBase.clean_up_tokenization()
- tokenizer_utils_fast
- utils