RoFormer模型汇总#
下表汇总介绍了目前PaddleNLP支持的RoFormer模型对应预训练权重。 关于模型的具体细节可以参考对应链接。
Pretrained Weight |
Language |
Details of the model |
---|---|---|
|
Chinese |
6-layer, 384-hidden, 6-heads, 30M parameters. Roformer Small Chinese model. |
|
Chinese |
12-layer, 768-hidden, 12-heads, 124M parameters. Roformer Base Chinese model. |
|
Chinese |
6-layer, 384-hidden, 6-heads, 15M parameters. Roformer Chinese Char Small model. |
|
Chinese |
12-layer, 768-hidden, 12-heads, 95M parameters. Roformer Chinese Char Base model. |
|
Chinese |
6-layer, 384-hidden, 6-heads, 15M parameters. Roformer Chinese Char Ft Small model. |
|
Chinese |
12-layer, 768-hidden, 12-heads, 95M parameters. Roformer Chinese Char Ft Base model. |
|
Chinese |
6-layer, 384-hidden, 6-heads, 15M parameters. Roformer Chinese Sim Char Small model. |
|
Chinese |
12-layer, 768-hidden, 12-heads, 95M parameters. Roformer Chinese Sim Char Base model. |
|
English |
12-layer, 256-hidden, 4-heads, 13M parameters. Roformer English Small Discriminator. |
|
English |
12-layer, 64-hidden, 1-heads, 5M parameters. Roformer English Small Generator. |