UnifiedTransformer模型汇总

UnifiedTransformer模型汇总#

下表汇总介绍了目前PaddleNLP支持的UnifiedTransformer模型对应预训练权重。 关于模型的具体细节可以参考对应链接。

Pretrained Weight

Language

Details of the model

unified_transformer-12L-cn

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on Chinese text.

unified_transformer-12L-cn-luge

Chinese

12-layer, 768-hidden, 12-heads, 108M parameters. Trained on Chinese text (LUGE.ai).

plato-mini

Chinese

6-layer, 768-hidden, 12-heads, 66M parameters. Trained on Chinese text.

plato-xl

Chinese

72-layer, 3072-hidden, 32-heads, ?M parameters. Trained on Chinese text.