BERT模型汇总#

下表汇总介绍了目前PaddleNLP支持的BERT模型对应预训练权重。 关于模型的具体细节可以参考对应链接。

Pretrained Weight

Language

Details of the model

bert-base-uncased

English

12-layer, 768-hidden, 12-heads, 110M parameters. Trained on lower-cased English text.

bert-large-uncased

English

24-layer, 1024-hidden, 16-heads, 336M parameters. Trained on lower-cased English text.

bert-base-cased

English

12-layer, 768-hidden, 12-heads, 109M parameters. Trained on cased English text.

bert-large-cased

English

24-layer, 1024-hidden, 16-heads, 335M parameters. Trained on cased English text.

bert-base-multilingual-uncased

Multilingual

12-layer, 768-hidden, 12-heads, 168M parameters. Trained on lower-cased text in the top 102 languages with the largest Wikipedias.

bert-base-multilingual-cased

Multilingual

12-layer, 768-hidden, 12-heads, 179M parameters. Trained on cased text in the top 104 languages with the largest Wikipedias.

bert-base-chinese

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text.

bert-wwm-chinese

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text using Whole-Word-Masking.

bert-wwm-ext-chinese

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on cased Chinese Simplified and Traditional text using Whole-Word-Masking with extented data.

uer/chinese-roberta-base

Chinese

Please refer to: uer/chinese_roberta_L-12_H-768

uer/chinese-roberta-medium

Chinese

Please refer to: uer/chinese_roberta_L-8_H-512

uer/chinese-roberta-small

Chinese

Please refer to: uer/chinese_roberta_L-4_H-512

uer/chinese-roberta-mini

Chinese

Please refer to: uer/chinese_roberta_L-4_H-256

uer/chinese-roberta-tiny

Chinese

Please refer to: uer/chinese_roberta_L-2_H-128

uer/chinese-roberta-6l-768h

Chinese

Please refer to: uer/chinese_roberta_L-6_H-768

ckiplab/bert-base-chinese-pos

Chinese

Please refer to: ckiplab/bert-base-chinese-pos

tbs17/MathBERT

English

Please refer to: tbs17/MathBERT

macbert-base-chinese

Chinese

12-layer, 768-hidden, 12-heads, 102M parameters. Trained with novel MLM as correction pre-training task.

macbert-large-chinese

Chinese

24-layer, 1024-hidden, 16-heads, 326M parameters. Trained with novel MLM as correction pre-training task.

simbert-base-chinese

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on 22 million pairs of similar sentences crawed from Baidu Know.

Langboat/mengzi-bert-base

Chinese

12-layer, 768-hidden, 12-heads, 102M parameters. Trained on 300G Chinese Corpus Datasets.

Langboat/mengzi-bert-base-fin

Chinese

12-layer, 768-hidden, 12-heads, 102M parameters. Trained on 20G Finacial Corpus, based on Langboat/mengzi-bert-base.

cross-encoder/ms-marco-MiniLM-L-12-v2

English

Please refer to: cross-encoder/ms-marco-MiniLM-L-12-v2

cl-tohoku/bert-base-japanese-char

Japanese

Please refer to: cl-tohoku/bert-base-japanese-char

cl-tohoku/bert-base-japanese-whole-word-masking

Japanese

Please refer to: cl-tohoku/bert-base-japanese-whole-word-masking

cl-tohoku/bert-base-japanese

Japanese

Please refer to: cl-tohoku/bert-base-japanese

nlptown/bert-base-multilingual-uncased-sentiment

Multilingual

Please refer to: nlptown/bert-base-multilingual-uncased-sentiment

bert-large-uncased-whole-word-masking-finetuned-squad

English

Please refer to: bert-large-uncased-whole-word-masking-finetuned-squad

finiteautomata/beto-sentiment-analysis

Spanish

Please refer to: finiteautomata/beto-sentiment-analysis

hfl/chinese-bert-wwm-ext

Chinese

Please refer to: hfl/chinese-bert-wwm-ext

emilyalsentzer/Bio_ClinicalBERT

English

Please refer to: emilyalsentzer/Bio_ClinicalBERT

dslim/bert-base-NER

English

Please refer to: dslim/bert-base-NER

deepset/bert-large-uncased-whole-word-masking-squad2

English

Please refer to: deepset/bert-large-uncased-whole-word-masking-squad2

neuralmind/bert-base-portuguese-cased

Portuguese

Please refer to: neuralmind/bert-base-portuguese-cased

SpanBERT/spanbert-large-cased

English

Please refer to: SpanBERT/spanbert-large-cased

dslim/bert-large-NER

English

Please refer to: dslim/bert-large-NER

bert-base-german-cased

German

Please refer to: bert-base-german-cased

deepset/sentence_bert

English

Please refer to: deepset/sentence_bert

ProsusAI/finbert

English

Please refer to: ProsusAI/finbert

oliverguhr/german-sentiment-bert

German

Please refer to: oliverguhr/german-sentiment-bert

google/bert_uncased_L-2_H-128_A-2

English

Please refer to: google/bert_uncased_L-2_H-128_A-2

microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract

English

Please refer to: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract

DeepPavlov/rubert-base-cased

Russian

Please refer to: DeepPavlov/rubert-base-cased

wietsedv/bert-base-dutch-cased

Dutch

Please refer to: wietsedv/bert-base-dutch-cased

monologg/bert-base-cased-goemotions-original

English

Please refer to: monologg/bert-base-cased-goemotions-original

allenai/scibert_scivocab_uncased

English

Please refer to: allenai/scibert_scivocab_uncased

dbmdz/bert-large-cased-finetuned-conll03-english

English

Please refer to: dbmdz/bert-large-cased-finetuned-conll03-english

microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext

English

Please refer to: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext

bert-large-uncased-whole-word-masking

English

Please refer to: bert-large-uncased-whole-word-masking

dccuchile/bert-base-spanish-wwm-uncased

Spanish

Please refer to: dccuchile/bert-base-spanish-wwm-uncased

google/bert_uncased_L-6_H-256_A-4

English

Please refer to: google/bert_uncased_L-6_H-256_A-4

google/bert_uncased_L-4_H-512_A-8

English

Please refer to: google/bert_uncased_L-4_H-512_A-8

FPTAI/vibert-base-cased

English

Please refer to: FPTAI/vibert-base-cased

cointegrated/rubert-tiny

Russian

Please refer to: cointegrated/rubert-tiny

bert-base-german-dbmdz-uncased

German

Please refer to: bert-base-german-dbmdz-uncased

dbmdz/bert-base-turkish-128k-cased

Turkish

Please refer to: dbmdz/bert-base-turkish-128k-cased

dbmdz/bert-base-german-uncased

German

Please refer to: dbmdz/bert-base-german-uncased

deepset/minilm-uncased-squad2

English

Please refer to: deepset/minilm-uncased-squad2

HooshvareLab/bert-base-parsbert-uncased

Persian

Please refer to: HooshvareLab/bert-base-parsbert-uncased

textattack/bert-base-uncased-ag-news

English

Please refer to: textattack/bert-base-uncased-ag-news

cl-tohoku/bert-base-japanese-v2

Japanese

Please refer to: cl-tohoku/bert-base-japanese-v2

emilyalsentzer/Bio_Discharge_Summary_BERT

English

Please refer to: emilyalsentzer/Bio_Discharge_Summary_BERT

KoichiYasuoka/bert-base-japanese-upos

Japanese

Please refer to: KoichiYasuoka/bert-base-japanese-upos

dbmdz/bert-base-italian-xxl-cased

Italian

Please refer to: dbmdz/bert-base-italian-xxl-cased

deepset/bert-base-cased-squad2

English

Please refer to: deepset/bert-base-cased-squad2

beomi/kcbert-large

English

Please refer to: beomi/kcbert-large

bert-large-cased-whole-word-masking-finetuned-squad

English

Please refer to: bert-large-cased-whole-word-masking-finetuned-squad

neuralmind/bert-large-portuguese-cased

Portuguese

Please refer to: neuralmind/bert-large-portuguese-cased

Luyu/co-condenser-marco

English

Please refer to: Luyu/co-condenser-marco

Sahajtomar/German_Zeroshot

German

Please refer to: Sahajtomar/German_Zeroshot

indolem/indobert-base-uncased

Indonesian

Please refer to: indolem/indobert-base-uncased

shibing624/text2vec-base-chinese

Chinese

Please refer to: shibing624/text2vec-base-chinese

cointegrated/LaBSE-en-ru

English and Russian

Please refer to: cointegrated/LaBSE-en-ru

prithivida/parrot_fluency_on_BERT

English

Please refer to: prithivida/parrot_fluency_on_BERT

textattack/bert-base-uncased-SST-2

English

Please refer to: textattack/bert-base-uncased-SST-2

textattack/bert-base-uncased-snli

English

Please refer to: textattack/bert-base-uncased-snli

klue/bert-base

English

Please refer to: klue/bert-base

asafaya/bert-base-arabic

Arabic

Please refer to: asafaya/bert-base-arabic

textattack/bert-base-uncased-MRPC

English

Please refer to: textattack/bert-base-uncased-MRPC

textattack/bert-base-uncased-imdb

English

Please refer to: textattack/bert-base-uncased-imdb

cross-encoder/ms-marco-TinyBERT-L-2

English

Please refer to: cross-encoder/ms-marco-TinyBERT-L-2

mrm8488/bert-tiny-finetuned-sms-spam-detection

English

Please refer to: mrm8488/bert-tiny-finetuned-sms-spam-detection

felflare/bert-restore-punctuation

English

Please refer to: felflare/bert-restore-punctuation

sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english

English

Please refer to: sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english

textattack/bert-base-uncased-rotten-tomatoes

English

Please refer to: textattack/bert-base-uncased-rotten-tomatoes

nlpaueb/legal-bert-base-uncased

English

Please refer to: nlpaueb/legal-bert-base-uncased

hf-internal-testing/tiny-bert-for-token-classification

English

Please refer to: hf-internal-testing/tiny-bert-for-token-classification

cointegrated/rubert-tiny2

Russian

Please refer to: cointegrated/rubert-tiny2

kykim/bert-kor-base

Korean

Please refer to: kykim/bert-kor-base

cl-tohoku/bert-base-japanese-char-v2

Japanese

Please refer to: cl-tohoku/bert-base-japanese-char-v2

mrm8488/bert-small-finetuned-squadv2

English

Please refer to: mrm8488/bert-small-finetuned-squadv2

beomi/kcbert-base

English

Please refer to: beomi/kcbert-base

textattack/bert-base-uncased-MNLI

English

Please refer to: textattack/bert-base-uncased-MNLI

textattack/bert-base-uncased-WNLI

English

Please refer to: textattack/bert-base-uncased-WNLI

dbmdz/bert-base-turkish-cased

Turkish

Please refer to: dbmdz/bert-base-turkish-cased

huawei-noah/TinyBERT_General_4L_312D

English

Please refer to: huawei-noah/TinyBERT_General_4L_312D

textattack/bert-base-uncased-QQP

English

Please refer to: textattack/bert-base-uncased-QQP

textattack/bert-base-uncased-STS-B

English

Please refer to: textattack/bert-base-uncased-STS-B

allenai/scibert_scivocab_cased

English

Please refer to: allenai/scibert_scivocab_cased

mrm8488/bert-medium-finetuned-squadv2

English

Please refer to: mrm8488/bert-medium-finetuned-squadv2

TurkuNLP/bert-base-finnish-cased-v1

Finnish

Please refer to: TurkuNLP/bert-base-finnish-cased-v1

textattack/bert-base-uncased-RTE

English

Please refer to: textattack/bert-base-uncased-RTE

uer/roberta-base-chinese-extractive-qa

Chinese

Please refer to: uer/roberta-base-chinese-extractive-qa

textattack/bert-base-uncased-QNLI

English

Please refer to: textattack/bert-base-uncased-QNLI

textattack/bert-base-uncased-CoLA

English

Please refer to: textattack/bert-base-uncased-CoLA

dmis-lab/biobert-base-cased-v1.2

English

Please refer to: dmis-lab/biobert-base-cased-v1.2

pierreguillou/bert-base-cased-squad-v1.1-portuguese

Portuguese

Please refer to: pierreguillou/bert-base-cased-squad-v1.1-portuguese

KB/bert-base-swedish-cased

Swedish

Please refer to: KB/bert-base-swedish-cased

uer/roberta-base-finetuned-cluener2020-chinese

Chinese

Please refer to: uer/roberta-base-finetuned-cluener2020-chinese

onlplab/alephbert-base

Hebrew

Please refer to: onlplab/alephbert-base

mrm8488/bert-spanish-cased-finetuned-ner

Spanish

Please refer to: mrm8488/bert-spanish-cased-finetuned-ner

alvaroalon2/biobert_chemical_ner

English

Please refer to: alvaroalon2/biobert_chemical_ner

bert-base-cased-finetuned-mrpc

English

Please refer to: bert-base-cased-finetuned-mrpc

unitary/toxic-bert

English

Please refer to: unitary/toxic-bert

nlpaueb/bert-base-greek-uncased-v1

Greek

Please refer to: nlpaueb/bert-base-greek-uncased-v1

HooshvareLab/bert-fa-base-uncased-sentiment-snappfood

Persian

Please refer to: HooshvareLab/bert-fa-base-uncased-sentiment-snappfood

Maltehb/danish-bert-botxo

Danish

Please refer to: Maltehb/danish-bert-botxo

shahrukhx01/bert-mini-finetune-question-detection

English

Please refer to: shahrukhx01/bert-mini-finetune-question-detection

GroNLP/bert-base-dutch-cased

Dutch

Please refer to: GroNLP/bert-base-dutch-cased

SpanBERT/spanbert-base-cased

English

Please refer to: SpanBERT/spanbert-base-cased

dbmdz/bert-base-italian-uncased

Italian

Please refer to: dbmdz/bert-base-italian-uncased

dbmdz/bert-base-german-cased

Germanh

Please refer to: dbmdz/bert-base-german-cased

cl-tohoku/bert-large-japanese

Japanese

Please refer to: cl-tohoku/bert-large-japanese

hfl/chinese-bert-wwm

Chinese

Please refer to: hfl/chinese-bert-wwm

hfl/chinese-macbert-large

Chinese

Please refer to: hfl/chinese-macbert-large

dslim/bert-base-NER-uncased

English

Please refer to: dslim/bert-base-NER-uncased

amberoad/bert-multilingual-passage-reranking-msmarco

Multilingual

Please refer to: amberoad/bert-multilingual-passage-reranking-msmarco

aubmindlab/bert-base-arabertv02

Arabic

Please refer to: aubmindlab/bert-base-arabertv02

google/bert_uncased_L-4_H-256_A-4

English

Please refer to: google/bert_uncased_L-4_H-256_A-4

DeepPavlov/rubert-base-cased-conversational

Russian

Please refer to: DeepPavlov/rubert-base-cased-conversational

dccuchile/bert-base-spanish-wwm-cased

Spanish

Please refer to: dccuchile/bert-base-spanish-wwm-cased

ckiplab/bert-base-chinese-ws

Chinese

Please refer to: ckiplab/bert-base-chinese-ws

daigo/bert-base-japanese-sentiment

Japanese

Please refer to: daigo/bert-base-japanese-sentiment

SZTAKI-HLT/hubert-base-cc

Hungarian

Please refer to: SZTAKI-HLT/hubert-base-cc

nlpaueb/legal-bert-small-uncased

English

Please refer to: nlpaueb/legal-bert-small-uncased

dumitrescustefan/bert-base-romanian-uncased-v1

Romanian

Please refer to: dumitrescustefan/bert-base-romanian-uncased-v1

google/muril-base-cased

Indian

Please refer to: google/muril-base-cased

dkleczek/bert-base-polish-uncased-v1

Polish

Please refer to: dkleczek/bert-base-polish-uncased-v1

ckiplab/bert-base-chinese-ner

Chinese

Please refer to: ckiplab/bert-base-chinese-ner

savasy/bert-base-turkish-sentiment-cased

Turkish

Please refer to: savasy/bert-base-turkish-sentiment-cased

mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es

Spanish

Please refer to: mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es

KB/bert-base-swedish-cased-ner

Swedish

Please refer to: KB/bert-base-swedish-cased-ner

hfl/rbt3

Chinese

Please refer to: hfl/rbt3

remotejob/gradientclassification_v0

English

Please refer to: remotejob/gradientclassification_v0

Recognai/bert-base-spanish-wwm-cased-xnli

Spanish

Please refer to: Recognai/bert-base-spanish-wwm-cased-xnli

HooshvareLab/bert-fa-zwnj-base

Persian

Please refer to: HooshvareLab/bert-fa-zwnj-base

monologg/bert-base-cased-goemotions-group

English

Please refer to: monologg/bert-base-cased-goemotions-group

blanchefort/rubert-base-cased-sentiment

Russian

Please refer to: blanchefort/rubert-base-cased-sentiment

shibing624/macbert4csc-base-chinese

Chinese

Please refer to: shibing624/macbert4csc-base-chinese

google/bert_uncased_L-8_H-512_A-8

English

Please refer to: google/bert_uncased_L-8_H-512_A-8

bert-large-cased-whole-word-masking

English

Please refer to: bert-large-cased-whole-word-masking

alvaroalon2/biobert_diseases_ner

English

Please refer to: alvaroalon2/biobert_diseases_ner

philschmid/BERT-Banking77

English

Please refer to: philschmid/BERT-Banking77

dbmdz/bert-base-turkish-uncased

Turkish

Please refer to: dbmdz/bert-base-turkish-uncased

vblagoje/bert-english-uncased-finetuned-pos

English

Please refer to: vblagoje/bert-english-uncased-finetuned-pos

dumitrescustefan/bert-base-romanian-cased-v1

Romanian

Please refer to: dumitrescustefan/bert-base-romanian-cased-v1

nreimers/BERT-Tiny_L-2_H-128_A-2

English

Please refer to: nreimers/BERT-Tiny_L-2_H-128_A-2

digitalepidemiologylab/covid-twitter-bert-v2

English

Please refer to: digitalepidemiologylab/covid-twitter-bert-v2

UBC-NLP/MARBERT

(DA) and MSA

Please refer to: UBC-NLP/MARBERT

pierreguillou/bert-large-cased-squad-v1.1-portuguese

Portuguese

Please refer to: pierreguillou/bert-large-cased-squad-v1.1-portuguese

alvaroalon2/biobert_genetic_ner

English

Please refer to: alvaroalon2/biobert_genetic_ner

bvanaken/clinical-assertion-negation-bert

English

Please refer to: bvanaken/clinical-assertion-negation-bert

cross-encoder/stsb-TinyBERT-L-4

English

Please refer to: cross-encoder/stsb-TinyBERT-L-4

sshleifer/tiny-distilbert-base-cased

English

Please refer to: sshleifer/tiny-distilbert-base-cased

ckiplab/bert-base-chinese

Chinese

Please refer to: ckiplab/bert-base-chinese

fabriceyhc/bert-base-uncased-amazon_polarity

English

Please refer to: fabriceyhc/bert-base-uncased-amazon_polarity