class Perplexity(name='Perplexity', *args, **kwargs)[source]#

Bases: Metric

Perplexity is a metric used to judge how good a language model is. We can define perplexity as the inverse probability of the test set, normalised by the number of the words in the test set. Perplexity is calculated using cross entropy. It supports both padding data and no padding data.

If data is not padded, users should provide seq_len for Metric initialization. If data is padded, your label should contain seq_mask, which indicates the actual length of samples.

This Perplexity requires that the output of your network is prediction, label and sequence length (optional). If the Perplexity here doesn’t meet your needs, you could override the compute or update method for calculating Perplexity.

  • seq_len (int) – Sequence length of each sample, it must be provided while data is not padded. Defaults to 20.

  • name (str) – Name of Metric instance. Defaults to ‘Perplexity’.


import paddle
from paddlenlp.transformers import BertTokenizer
from paddlenlp.metrics import Perplexity

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
batch_size, seq_len, vocab_size = 1, 4, tokenizer.vocab_size
logits = paddle.rand([batch_size, seq_len, vocab_size])
labels= paddle.to_tensor([[1,0,1,1]])

perplexity = Perplexity()
correct = perplexity.compute(logits,labels)
res = perplexity.accumulate()
# 48263.528820122105
compute(pred, label, seq_mask=None)[source]#

Computes cross entropy loss.

  • pred (Tensor) – Predictor tensor, and its dtype is float32 or float64, and has a shape of [batch_size, sequence_length, vocab_size].

  • label (Tensor) – Label tensor, and its dtype is int64, and has a shape of [batch_size, sequence_length, 1] or [batch_size, sequence_length].

  • seq_mask (Tensor, optional) – Sequence mask tensor, and its type could be float32, float64, int32 or int64, and has a shape of [batch_size, sequence_length]. It’s used to calculate loss. Defaults to None.


Returns tuple (ce, word_num) if seq_mask is not None. Otherwise, returns tensor ce. ce it the cross entropy loss, its shape is [batch_size, sequence_length] and its data type should be float32.

Return type:

tuple or Tensor

update(ce, word_num=None)[source]#

Updates metric states.

  • ce (numpy.ndarray) – Cross entropy loss, it’s calculated by compute and converted to numpy.ndarray.

  • word_num (numpy.ndarray) – The number of words of sequence, it’s calculated by compute and converted to numpy.ndarray. Defaults to None.


Resets all metric states.


Calculates and returns the value of perplexity.


Returns perplexity, the calculation results.

Return type:



Returns name of the metric instance.


The name of the metric instance.

Return type: