TinyBert模型汇总#
下表汇总介绍了目前PaddleNLP支持的TinyBert模型以及对应预训练权重。 关于模型的具体细节可以参考对应链接。
Pretrained Weight |
Language |
Details of the model |
---|---|---|
|
English |
4-layer, 312-hidden,
12-heads, 14.5M parameters.
The TinyBert model distilled from
the BERT model |
|
English |
6-layer, 768-hidden,
12-heads, 67M parameters.
The TinyBert model distilled from
the BERT model |
|
English |
4-layer, 312-hidden,
12-heads, 14.5M parameters.
The TinyBert model distilled from
the BERT model |
|
English |
6-layer, 768-hidden,
12-heads, 67M parameters.
The TinyBert model distilled from
the BERT model |
|
Chinese |
4-layer, 312-hidden,
12-heads, 14.5M parameters.
The TinyBert model distilled from
the BERT model |
|
Chinese |
6-layer, 768-hidden,
12-heads, 67M parameters.
The TinyBert model distilled from
the BERT model |