Skip to main content

Ctrl+K

快速开始

安装PaddleNLP
10分钟完成高精度中文情感分析
对话模板

数据准备

整体介绍
数据集列表
加载数据集
自定义数据集
数据处理

模型库

Transformer预训练模型
- ALBERT
- BART
- BERT
- BigBird
- Blenderbot
- Blenderbot-Small
- ChineseBert
- ConvBert
- CTRL
- DistilBert
- ELECTRA
- ERNIE
- ERNIE-CTM
- ERNIE-DOC
- ERNIE-GEN
- ERNIE-GRAM
- ERNIE-M
- FNet
- Funnel
- GPT
- LayoutLM
- LayoutLMV2
- LayoutXLM
- Luke
- MBart
- MegatronBert
- MobileBert
- MPNet
- NeZha
- PPMiniLM
- ProphetNet
- Reformer
- RemBert
- RoBERTa
- RoFormer
- SKEP
- SqueezeBert
- T5
- TinyBert
- UnifiedTransformer
- UNIMO
- XLNet
使用Trainer API训练
使用Trainer API进行模型压缩
一键预测功能
预训练词向量

评价指标

评价指标

实践教程

AI Studio Notebook

进阶指南

模型压缩
文本生成高性能加速
- Transformer高性能加速
大规模分布式训练

社区交流共建

如何贡献模型
如何贡献数据集
- 分享你的数据集
- 创建 DatasetBuilder
如何贡献文档案例
如何加入兴趣小组

FAQ

PaddleNLP常见问题汇总（持续更新）

API Reference

paddlenlp.data
- collate
- data_collator
- sampler
- tokenizer
- vocab
paddlenlp.datasets
- dataset
paddlenlp.embeddings
- token_embedding
paddlenlp.layers
- crf
- sequence
- tcn
paddlenlp.losses
- rdrop
paddlenlp.metrics
- bleu
- chunk
- distinct
- dureader
- glue
- perplexity
- rouge
- sighan
- span
- squad
- utils
paddlenlp.ops
- distributed
  - utils
    - random
    - topo
  - parallel
- fast_transformer
  - transformer
- optimizer
  - adamwdl
  - ema
- einsum
- ext_utils
paddlenlp.seq2vec
- encoder
paddlenlp.taskflow
paddlenlp.trainer
paddlenlp.transformers
- albert
  - modeling
  - tokenizer
- artist
  - modeling
  - tokenizer
- auto
  - modeling
  - tokenizer
- bart
  - modeling
  - tokenizer
- bert
- bert_japanese
  - tokenizer
- bigbird
  - modeling
  - tokenizer
- blenderbot
  - modeling
  - tokenizer
- blenderbot_small
  - modeling
  - tokenizer
- chinesebert
  - modeling
  - tokenizer
- codegen
  - modeling
  - tokenizer
- convbert
  - modeling
  - tokenizer
- ctrl
  - modeling
  - tokenizer
- dallebart
  - modeling
  - tokenizer
- distilbert
  - modeling
  - tokenizer
- electra
  - modeling
  - tokenizer
- ernie
- ernie_ctm
  - modeling
  - tokenizer
- ernie_doc
  - modeling
  - tokenizer
- ernie_gen
  - modeling
- ernie_gram
- ernie_m
  - modeling
  - tokenizer
- fnet
  - modeling
  - tokenizer
- funnel
  - modeling
  - tokenizer
- gau_alpha
  - modeling
  - tokenizer
- gpt
  - modeling
  - tokenizer
- layoutlm
  - modeling
  - tokenizer
- layoutlmv2
  - modeling
  - tokenizer
- layoutxlm
- luke
  - modeling
  - tokenizer
- mbart
  - modeling
  - tokenizer
- megatronbert
  - modeling
  - tokenizer
- mobilebert
  - modeling
  - tokenizer
- mpnet
  - modeling
  - tokenizer
- nezha
  - modeling
  - tokenizer
- opt
  - modeling
- ppminilm
  - modeling
  - tokenizer
- prophetnet
  - modeling
  - tokenizer
- reformer
  - modeling
  - tokenizer
- rembert
  - modeling
  - tokenizer
- roberta
  - modeling
  - tokenizer
- roformer
  - modeling
  - tokenizer
- roformerv2
  - modeling
  - tokenizer
- semantic_search
  - modeling
- skep
  - modeling
  - tokenizer
- squeezebert
  - modeling
  - tokenizer
- t5
  - modeling
  - tokenizer
- tinybert
- transformer
  - modeling
- unified_transformer
- unimo
  - modeling
  - tokenizer
- xlm
  - modeling
  - tokenizer
- xlnet
  - modeling
  - tokenizer
- attention_utils
- convert_slow_tokenizer
- distill_utils
- export
- generation_utils
- model_outputs
- model_utils
- optimization
- sentencepiece_model_pb2
- tokenizer_utils
- tokenizer_utils_base
- tokenizer_utils_fast
- utils
paddlenlp.utils
- batch_sampler
- downloader
- env
- file_lock
- import_utils
- log
- profiler
- tools

.rst

tokenizer_utils_base

tokenizer_utils_base#

By PaddleNLP