ELECTRA模型汇总

ELECTRA模型汇总#

下表汇总介绍了目前PaddleNLP支持的ELECTRA模型对应预训练权重。 关于模型的具体细节可以参考对应链接。

Pretrained Weight

Language

Details of the model

electra-small

English

12-layer, 768-hidden, 4-heads, 14M parameters. Trained on lower-cased English text.

electra-base

English

12-layer, 768-hidden, 12-heads, 109M parameters. Trained on lower-cased English text.

electra-large

English

24-layer, 1024-hidden, 16-heads, 334M parameters. Trained on lower-cased English text.

chinese-electra-small

Chinese

12-layer, 768-hidden, 4-heads, 12M parameters. Trained on Chinese text.

chinese-electra-base

Chinese

12-layer, 768-hidden, 12-heads, 102M parameters. Trained on Chinese text.

ernie-health-chinese

Chinese

12-layer, 768-hidden, 12-heads, 102M parameters. Trained on Chinese medical corpus.

hfl/chinese-electra-180g-base-discriminator

Chinese

Discriminator, 12-layer, 768-hidden, 12-heads, 102M parameters. Trained on 180g Chinese text.

Please refer to: hfl/chinese-electra-180g-base-discriminator

hfl/chinese-electra-180g-small-ex-discriminator

Chinese

Discriminator, 24-layer, 256-hidden, 4-heads, 24M parameters. Trained on 180g Chinese text.

Please refer to: hfl/chinese-electra-180g-small-ex-discriminator

hfl/chinese-legal-electra-small-generator

Chinese

Generator, 12-layer, 64-hidden, 1-heads, 3M parameters. Trained on Chinese legal corpus.

Please refer to: hfl/chinese-legal-electra-small-generator