BERT模型汇总

下表汇总介绍了目前PaddleNLP支持的BERT模型对应预训练权重。 关于模型的具体细节可以参考对应链接。

Pretrained Weight	Language	Details of the model
bert-base-uncased	English	12-layer, 768-hidden, 12-heads, 110M parameters. Trained on lower-cased English text.
bert-large-uncased	English	24-layer, 1024-hidden, 16-heads, 336M parameters. Trained on lower-cased English text.
bert-base-cased	English	12-layer, 768-hidden, 12-heads, 109M parameters. Trained on cased English text.
bert-large-cased	English	24-layer, 1024-hidden, 16-heads, 335M parameters. Trained on cased English text.
bert-base-multilingual-uncased	Multilingual	12-layer, 768-hidden, 12-heads, 168M parameters. Trained on lower-cased text in the top 102 languages with the largest Wikipedias.
bert-base-multilingual-cased	Multilingual	12-layer, 768-hidden, 12-heads, 179M parameters. Trained on cased text in the top 104 languages with the largest Wikipedias.
bert-base-chinese	Chinese	12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text.
bert-wwm-chinese	Chinese	12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text using Whole-Word-Masking.
bert-wwm-ext-chinese	Chinese	12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text using Whole-Word-Masking with extented data.
uer/chinese-roberta-base	Chinese	Please refer to: uer/chinese_roberta_L-12_H-768
uer/chinese-roberta-medium	Chinese	Please refer to: uer/chinese_roberta_L-8_H-512
uer/chinese-roberta-small	Chinese	Please refer to: uer/chinese_roberta_L-4_H-512
uer/chinese-roberta-mini	Chinese	Please refer to: uer/chinese_roberta_L-4_H-256
uer/chinese-roberta-tiny	Chinese	Please refer to: uer/chinese_roberta_L-2_H-128
uer/chinese-roberta-6l-768h	Chinese	Please refer to: uer/chinese_roberta_L-6_H-768
ckiplab/bert-base-chinese-pos	Chinese	Please refer to: ckiplab/bert-base-chinese-pos
tbs17/MathBERT	English	Please refer to: tbs17/MathBERT
macbert-base-chinese	Chinese	12-layer, 768-hidden, 12-heads, 102M parameters. Trained with novel MLM as correction pre-training task.
macbert-large-chinese	Chinese	24-layer, 1024-hidden, 16-heads, 326M parameters. Trained with novel MLM as correction pre-training task.
simbert-base-chinese	Chinese	12-layer, 768-hidden, 12-heads, 108M parameters. Trained on 22 million pairs of similar sentences crawed from Baidu Know.
Langboat/mengzi-bert-base	Chinese	12-layer, 768-hidden, 12-heads, 102M parameters. Trained on 300G Chinese Corpus Datasets.
Langboat/mengzi-bert-base-fin	Chinese	12-layer, 768-hidden, 12-heads, 102M parameters. Trained on 20G Finacial Corpus, based on Langboat/mengzi-bert-base.
cross-encoder/ms-marco-MiniLM-L-12-v2	English	Please refer to: cross-encoder/ms-marco-MiniLM-L-12-v2
cl-tohoku/bert-base-japanese-char	Japanese	Please refer to: cl-tohoku/bert-base-japanese-char
cl-tohoku/bert-base-japanese-whole-word-masking	Japanese	Please refer to: cl-tohoku/bert-base-japanese-whole-word-masking
cl-tohoku/bert-base-japanese	Japanese	Please refer to: cl-tohoku/bert-base-japanese
nlptown/bert-base-multilingual-uncased-sentiment	Multilingual	Please refer to: nlptown/bert-base-multilingual-uncased-sentiment
bert-large-uncased-whole-word-masking-finetuned-squad	English	Please refer to: bert-large-uncased-whole-word-masking-finetuned-squad
finiteautomata/beto-sentiment-analysis	Spanish	Please refer to: finiteautomata/beto-sentiment-analysis
hfl/chinese-bert-wwm-ext	Chinese	Please refer to: hfl/chinese-bert-wwm-ext
emilyalsentzer/Bio_ClinicalBERT	English	Please refer to: emilyalsentzer/Bio_ClinicalBERT
dslim/bert-base-NER	English	Please refer to: dslim/bert-base-NER
deepset/bert-large-uncased-whole-word-masking-squad2	English	Please refer to: deepset/bert-large-uncased-whole-word-masking-squad2
neuralmind/bert-base-portuguese-cased	Portuguese	Please refer to: neuralmind/bert-base-portuguese-cased
SpanBERT/spanbert-large-cased	English	Please refer to: SpanBERT/spanbert-large-cased
dslim/bert-large-NER	English	Please refer to: dslim/bert-large-NER
bert-base-german-cased	German	Please refer to: bert-base-german-cased
deepset/sentence_bert	English	Please refer to: deepset/sentence_bert
ProsusAI/finbert	English	Please refer to: ProsusAI/finbert
oliverguhr/german-sentiment-bert	German	Please refer to: oliverguhr/german-sentiment-bert
google/bert_uncased_L-2_H-128_A-2	English	Please refer to: google/bert_uncased_L-2_H-128_A-2
microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract	English	Please refer to: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
DeepPavlov/rubert-base-cased	Russian	Please refer to: DeepPavlov/rubert-base-cased
wietsedv/bert-base-dutch-cased	Dutch	Please refer to: wietsedv/bert-base-dutch-cased
monologg/bert-base-cased-goemotions-original	English	Please refer to: monologg/bert-base-cased-goemotions-original
allenai/scibert_scivocab_uncased	English	Please refer to: allenai/scibert_scivocab_uncased
dbmdz/bert-large-cased-finetuned-conll03-english	English	Please refer to: dbmdz/bert-large-cased-finetuned-conll03-english
microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext	English	Please refer to: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
bert-large-uncased-whole-word-masking	English	Please refer to: bert-large-uncased-whole-word-masking
dccuchile/bert-base-spanish-wwm-uncased	Spanish	Please refer to: dccuchile/bert-base-spanish-wwm-uncased
google/bert_uncased_L-6_H-256_A-4	English	Please refer to: google/bert_uncased_L-6_H-256_A-4
google/bert_uncased_L-4_H-512_A-8	English	Please refer to: google/bert_uncased_L-4_H-512_A-8
FPTAI/vibert-base-cased	English	Please refer to: FPTAI/vibert-base-cased
cointegrated/rubert-tiny	Russian	Please refer to: cointegrated/rubert-tiny
bert-base-german-dbmdz-uncased	German	Please refer to: bert-base-german-dbmdz-uncased
dbmdz/bert-base-turkish-128k-cased	Turkish	Please refer to: dbmdz/bert-base-turkish-128k-cased
dbmdz/bert-base-german-uncased	German	Please refer to: dbmdz/bert-base-german-uncased
deepset/minilm-uncased-squad2	English	Please refer to: deepset/minilm-uncased-squad2
HooshvareLab/bert-base-parsbert-uncased	Persian	Please refer to: HooshvareLab/bert-base-parsbert-uncased
textattack/bert-base-uncased-ag-news	English	Please refer to: textattack/bert-base-uncased-ag-news
cl-tohoku/bert-base-japanese-v2	Japanese	Please refer to: cl-tohoku/bert-base-japanese-v2
emilyalsentzer/Bio_Discharge_Summary_BERT	English	Please refer to: emilyalsentzer/Bio_Discharge_Summary_BERT
KoichiYasuoka/bert-base-japanese-upos	Japanese	Please refer to: KoichiYasuoka/bert-base-japanese-upos
dbmdz/bert-base-italian-xxl-cased	Italian	Please refer to: dbmdz/bert-base-italian-xxl-cased
deepset/bert-base-cased-squad2	English	Please refer to: deepset/bert-base-cased-squad2
beomi/kcbert-large	English	Please refer to: beomi/kcbert-large
bert-large-cased-whole-word-masking-finetuned-squad	English	Please refer to: bert-large-cased-whole-word-masking-finetuned-squad
neuralmind/bert-large-portuguese-cased	Portuguese	Please refer to: neuralmind/bert-large-portuguese-cased
Luyu/co-condenser-marco	English	Please refer to: Luyu/co-condenser-marco
Sahajtomar/German_Zeroshot	German	Please refer to: Sahajtomar/German_Zeroshot
indolem/indobert-base-uncased	Indonesian	Please refer to: indolem/indobert-base-uncased
shibing624/text2vec-base-chinese	Chinese	Please refer to: shibing624/text2vec-base-chinese
cointegrated/LaBSE-en-ru	English and Russian	Please refer to: cointegrated/LaBSE-en-ru
prithivida/parrot_fluency_on_BERT	English	Please refer to: prithivida/parrot_fluency_on_BERT
textattack/bert-base-uncased-SST-2	English	Please refer to: textattack/bert-base-uncased-SST-2
textattack/bert-base-uncased-snli	English	Please refer to: textattack/bert-base-uncased-snli
klue/bert-base	English	Please refer to: klue/bert-base
asafaya/bert-base-arabic	Arabic	Please refer to: asafaya/bert-base-arabic
textattack/bert-base-uncased-MRPC	English	Please refer to: textattack/bert-base-uncased-MRPC
textattack/bert-base-uncased-imdb	English	Please refer to: textattack/bert-base-uncased-imdb
cross-encoder/ms-marco-TinyBERT-L-2	English	Please refer to: cross-encoder/ms-marco-TinyBERT-L-2
mrm8488/bert-tiny-finetuned-sms-spam-detection	English	Please refer to: mrm8488/bert-tiny-finetuned-sms-spam-detection
felflare/bert-restore-punctuation	English	Please refer to: felflare/bert-restore-punctuation
sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english	English	Please refer to: sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english
textattack/bert-base-uncased-rotten-tomatoes	English	Please refer to: textattack/bert-base-uncased-rotten-tomatoes
nlpaueb/legal-bert-base-uncased	English	Please refer to: nlpaueb/legal-bert-base-uncased
hf-internal-testing/tiny-bert-for-token-classification	English	Please refer to: hf-internal-testing/tiny-bert-for-token-classification
cointegrated/rubert-tiny2	Russian	Please refer to: cointegrated/rubert-tiny2
kykim/bert-kor-base	Korean	Please refer to: kykim/bert-kor-base
cl-tohoku/bert-base-japanese-char-v2	Japanese	Please refer to: cl-tohoku/bert-base-japanese-char-v2
mrm8488/bert-small-finetuned-squadv2	English	Please refer to: mrm8488/bert-small-finetuned-squadv2
beomi/kcbert-base	English	Please refer to: beomi/kcbert-base
textattack/bert-base-uncased-MNLI	English	Please refer to: textattack/bert-base-uncased-MNLI
textattack/bert-base-uncased-WNLI	English	Please refer to: textattack/bert-base-uncased-WNLI
dbmdz/bert-base-turkish-cased	Turkish	Please refer to: dbmdz/bert-base-turkish-cased
huawei-noah/TinyBERT_General_4L_312D	English	Please refer to: huawei-noah/TinyBERT_General_4L_312D
textattack/bert-base-uncased-QQP	English	Please refer to: textattack/bert-base-uncased-QQP
textattack/bert-base-uncased-STS-B	English	Please refer to: textattack/bert-base-uncased-STS-B
allenai/scibert_scivocab_cased	English	Please refer to: allenai/scibert_scivocab_cased
mrm8488/bert-medium-finetuned-squadv2	English	Please refer to: mrm8488/bert-medium-finetuned-squadv2
TurkuNLP/bert-base-finnish-cased-v1	Finnish	Please refer to: TurkuNLP/bert-base-finnish-cased-v1
textattack/bert-base-uncased-RTE	English	Please refer to: textattack/bert-base-uncased-RTE
uer/roberta-base-chinese-extractive-qa	Chinese	Please refer to: uer/roberta-base-chinese-extractive-qa
textattack/bert-base-uncased-QNLI	English	Please refer to: textattack/bert-base-uncased-QNLI
textattack/bert-base-uncased-CoLA	English	Please refer to: textattack/bert-base-uncased-CoLA
dmis-lab/biobert-base-cased-v1.2	English	Please refer to: dmis-lab/biobert-base-cased-v1.2
pierreguillou/bert-base-cased-squad-v1.1-portuguese	Portuguese	Please refer to: pierreguillou/bert-base-cased-squad-v1.1-portuguese
KB/bert-base-swedish-cased	Swedish	Please refer to: KB/bert-base-swedish-cased
uer/roberta-base-finetuned-cluener2020-chinese	Chinese	Please refer to: uer/roberta-base-finetuned-cluener2020-chinese
onlplab/alephbert-base	Hebrew	Please refer to: onlplab/alephbert-base
mrm8488/bert-spanish-cased-finetuned-ner	Spanish	Please refer to: mrm8488/bert-spanish-cased-finetuned-ner
alvaroalon2/biobert_chemical_ner	English	Please refer to: alvaroalon2/biobert_chemical_ner
bert-base-cased-finetuned-mrpc	English	Please refer to: bert-base-cased-finetuned-mrpc
unitary/toxic-bert	English	Please refer to: unitary/toxic-bert
nlpaueb/bert-base-greek-uncased-v1	Greek	Please refer to: nlpaueb/bert-base-greek-uncased-v1
HooshvareLab/bert-fa-base-uncased-sentiment-snappfood	Persian	Please refer to: HooshvareLab/bert-fa-base-uncased-sentiment-snappfood
Maltehb/danish-bert-botxo	Danish	Please refer to: Maltehb/danish-bert-botxo
shahrukhx01/bert-mini-finetune-question-detection	English	Please refer to: shahrukhx01/bert-mini-finetune-question-detection
GroNLP/bert-base-dutch-cased	Dutch	Please refer to: GroNLP/bert-base-dutch-cased
SpanBERT/spanbert-base-cased	English	Please refer to: SpanBERT/spanbert-base-cased
dbmdz/bert-base-italian-uncased	Italian	Please refer to: dbmdz/bert-base-italian-uncased
dbmdz/bert-base-german-cased	Germanh	Please refer to: dbmdz/bert-base-german-cased
cl-tohoku/bert-large-japanese	Japanese	Please refer to: cl-tohoku/bert-large-japanese
hfl/chinese-bert-wwm	Chinese	Please refer to: hfl/chinese-bert-wwm
hfl/chinese-macbert-large	Chinese	Please refer to: hfl/chinese-macbert-large
dslim/bert-base-NER-uncased	English	Please refer to: dslim/bert-base-NER-uncased
amberoad/bert-multilingual-passage-reranking-msmarco	Multilingual	Please refer to: amberoad/bert-multilingual-passage-reranking-msmarco
aubmindlab/bert-base-arabertv02	Arabic	Please refer to: aubmindlab/bert-base-arabertv02
google/bert_uncased_L-4_H-256_A-4	English	Please refer to: google/bert_uncased_L-4_H-256_A-4
DeepPavlov/rubert-base-cased-conversational	Russian	Please refer to: DeepPavlov/rubert-base-cased-conversational
dccuchile/bert-base-spanish-wwm-cased	Spanish	Please refer to: dccuchile/bert-base-spanish-wwm-cased
ckiplab/bert-base-chinese-ws	Chinese	Please refer to: ckiplab/bert-base-chinese-ws
daigo/bert-base-japanese-sentiment	Japanese	Please refer to: daigo/bert-base-japanese-sentiment
SZTAKI-HLT/hubert-base-cc	Hungarian	Please refer to: SZTAKI-HLT/hubert-base-cc
nlpaueb/legal-bert-small-uncased	English	Please refer to: nlpaueb/legal-bert-small-uncased
dumitrescustefan/bert-base-romanian-uncased-v1	Romanian	Please refer to: dumitrescustefan/bert-base-romanian-uncased-v1
google/muril-base-cased	Indian	Please refer to: google/muril-base-cased
dkleczek/bert-base-polish-uncased-v1	Polish	Please refer to: dkleczek/bert-base-polish-uncased-v1
ckiplab/bert-base-chinese-ner	Chinese	Please refer to: ckiplab/bert-base-chinese-ner
savasy/bert-base-turkish-sentiment-cased	Turkish	Please refer to: savasy/bert-base-turkish-sentiment-cased
mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es	Spanish	Please refer to: mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es
KB/bert-base-swedish-cased-ner	Swedish	Please refer to: KB/bert-base-swedish-cased-ner
hfl/rbt3	Chinese	Please refer to: hfl/rbt3
remotejob/gradientclassification_v0	English	Please refer to: remotejob/gradientclassification_v0
Recognai/bert-base-spanish-wwm-cased-xnli	Spanish	Please refer to: Recognai/bert-base-spanish-wwm-cased-xnli
HooshvareLab/bert-fa-zwnj-base	Persian	Please refer to: HooshvareLab/bert-fa-zwnj-base
monologg/bert-base-cased-goemotions-group	English	Please refer to: monologg/bert-base-cased-goemotions-group
blanchefort/rubert-base-cased-sentiment	Russian	Please refer to: blanchefort/rubert-base-cased-sentiment
shibing624/macbert4csc-base-chinese	Chinese	Please refer to: shibing624/macbert4csc-base-chinese
google/bert_uncased_L-8_H-512_A-8	English	Please refer to: google/bert_uncased_L-8_H-512_A-8
bert-large-cased-whole-word-masking	English	Please refer to: bert-large-cased-whole-word-masking
alvaroalon2/biobert_diseases_ner	English	Please refer to: alvaroalon2/biobert_diseases_ner
philschmid/BERT-Banking77	English	Please refer to: philschmid/BERT-Banking77
dbmdz/bert-base-turkish-uncased	Turkish	Please refer to: dbmdz/bert-base-turkish-uncased
vblagoje/bert-english-uncased-finetuned-pos	English	Please refer to: vblagoje/bert-english-uncased-finetuned-pos
dumitrescustefan/bert-base-romanian-cased-v1	Romanian	Please refer to: dumitrescustefan/bert-base-romanian-cased-v1
nreimers/BERT-Tiny_L-2_H-128_A-2	English	Please refer to: nreimers/BERT-Tiny_L-2_H-128_A-2
digitalepidemiologylab/covid-twitter-bert-v2	English	Please refer to: digitalepidemiologylab/covid-twitter-bert-v2
UBC-NLP/MARBERT	(DA) and MSA	Please refer to: UBC-NLP/MARBERT
pierreguillou/bert-large-cased-squad-v1.1-portuguese	Portuguese	Please refer to: pierreguillou/bert-large-cased-squad-v1.1-portuguese
alvaroalon2/biobert_genetic_ner	English	Please refer to: alvaroalon2/biobert_genetic_ner
bvanaken/clinical-assertion-negation-bert	English	Please refer to: bvanaken/clinical-assertion-negation-bert
cross-encoder/stsb-TinyBERT-L-4	English	Please refer to: cross-encoder/stsb-TinyBERT-L-4
sshleifer/tiny-distilbert-base-cased	English	Please refer to: sshleifer/tiny-distilbert-base-cased
ckiplab/bert-base-chinese	Chinese	Please refer to: ckiplab/bert-base-chinese
fabriceyhc/bert-base-uncased-amazon_polarity	English	Please refer to: fabriceyhc/bert-base-uncased-amazon_polarity