glue

class AccuracyAndF1(topk=(1), pos_label=1, name='acc_and_f1', *args, **kwargs)

Bases: paddle.metric.metrics.Metric

This class encapsulates Accuracy, Precision, Recall and F1 metric logic, and accumulate function returns accuracy, precision, recall and f1. The overview of all metrics could be seen at the document of paddle.metric for details.

Parameters

• topk (int or tuple(int), optional) – Number of top elements to look at for computing accuracy. Defaults to (1,).

• pos_label (int, optional) – The positive label for calculating precision and recall. Defaults to 1.

• name (str, optional) – String name of the metric instance. Defaults to ‘acc_and_f1’.

Example

import paddle
from paddlenlp.metrics import AccuracyAndF1

x = paddle.to_tensor([[0.1, 0.9], [0.5, 0.5], [0.6, 0.4], [0.7, 0.3]])
y = paddle.to_tensor([[1], [0], [1], [1]])

m = AccuracyAndF1()
correct = m.compute(x, y)
m.update(correct)
res = m.accumulate()
print(res) # (0.5, 0.5, 0.3333333333333333, 0.4, 0.45)

compute(pred, label, *args)

Accepts network’s output and the labels, and calculates the top-k (maximum value in topk) indices for accuracy.

Parameters

• pred (Tensor) – Predicted tensor, and its dtype is float32 or float64, and has a shape of [batch_size, num_classes].

• label (Tensor) – The ground truth tensor, and its dtype is is int64, and has a shape of [batch_size, 1] or [batch_size, num_classes] in one hot representation.

Returns

Correct mask, each element indicates whether the prediction equals to the label. Its’ a tensor with a data type of float32 and has a shape of [batch_size, topk].

Return type

Tensor

update(correct, *args)

Updates the metrics states (accuracy, precision and recall), in order to calculate accumulated accuracy, precision and recall of all instances.

Parameters

correct (Tensor) – Correct mask for calculating accuracy, and it’s a tensor with shape [batch_size, topk] and has a dtype of float32.

accumulate()

Calculates and returns the accumulated metric.

Returns

The accumulated metric. A tuple of shape (acc, precision, recall, f1, average_of_acc_and_f1)

With the fields:

• acc (numpy.float64):

  The accumulated accuracy.

• precision (numpy.float64):

  The accumulated precision.

• recall (numpy.float64):

  The accumulated recall.

• f1 (numpy.float64):

  The accumulated f1.

• average_of_acc_and_f1 (numpy.float64):

  The average of accumulated accuracy and f1.

Return type

tuple

reset()

Resets all metric states.

name()

Returns name of the metric instance.

Returns

The name of the metric instance.

Return type

str

class Mcc(name='mcc', *args, **kwargs)

Bases: paddle.metric.metrics.Metric

This class calculates Matthews correlation coefficient .

Parameters

name (str, optional) – String name of the metric instance. Defaults to ‘mcc’.

Example

import paddle
from paddlenlp.metrics import Mcc

x = paddle.to_tensor([[-0.1, 0.12], [-0.23, 0.23], [-0.32, 0.21], [-0.13, 0.23]])
y = paddle.to_tensor([[1], [0], [1], [1]])

m = Mcc()
(preds, label) = m.compute(x, y)
m.update((preds, label))
res = m.accumulate()
print(res) # (0.0,)

compute(pred, label, *args)

Processes the pred tensor, and returns the indices of the maximum of each sample.

Parameters

• pred (Tensor) – The predicted value is a Tensor with dtype float32 or float64. Shape is [batch_size, 1].

• label (Tensor) – The ground truth value is Tensor with dtype int64, and its shape is [batch_size, 1].

Returns

A tuple of preds and label. Each shape is [batch_size, 1], with dtype float32 or float64.

Return type

tuple

update(preds_and_labels)

Calculates states, i.e. the number of true positive, false positive, true negative and false negative samples.

Parameters

preds_and_labels (tuple[Tensor]) – Tuple of predicted value and the ground truth label, with dtype float32 or float64. Each shape is [batch_size, 1].

accumulate()

Calculates and returns the accumulated metric.

Returns

Returns the accumulated metric, a tuple of shape (mcc,), mcc is the accumulated mcc and its data type is float64.

Return type

tuple

reset()

Resets all metric states.

name()

Returns name of the metric instance.

Returns

The name of the metric instance.

Return type

str

class PearsonAndSpearman(name='pearson_and_spearman', *args, **kwargs)

Bases: paddle.metric.metrics.Metric

The class calculates Pearson correlation coefficient and Spearman’s rank correlation coefficient .

Parameters

name (str, optional) – String name of the metric instance. Defaults to ‘pearson_and_spearman’.

Example

import paddle
from paddlenlp.metrics import PearsonAndSpearman

x = paddle.to_tensor([[0.1], [1.0], [2.4], [0.9]])
y = paddle.to_tensor([[0.0], [1.0], [2.9], [1.0]])

m = PearsonAndSpearman()
m.update((x, y))
res = m.accumulate()
print(res) # (0.9985229081857804, 1.0, 0.9992614540928901)

update(preds_and_labels)

Ensures the type of preds and labels is numpy.ndarray and reshapes them into [-1, 1].

Parameters

preds_and_labels (tuple[Tensor] or list[Tensor]) – Tuple or list of predicted value and the ground truth label. Its data type should be float32 or float64 and its shape is [batch_size, d0, …, dN].

accumulate()

Calculates and returns the accumulated metric.

Returns

Returns the accumulated metric, a tuple of (pearson, spearman, the_average_of_pearson_and_spearman).

With the fields:

• pearson (numpy.float64):

  The accumulated pearson.

• spearman (numpy.float64):

  The accumulated spearman.

• the_average_of_pearson_and_spearman (numpy.float64):

  The average of accumulated pearson and spearman correlation coefficient.

Return type

tuple

reset()

Resets all metric states.

name()

Returns name of the metric instance.

Returns

The name of the metric instance.

Return type

str

class MultiLabelsMetric(num_labels, name='multi_labels_metric')

Bases: paddle.metric.metrics.Metric

This class encapsulates Accuracy, Precision, Recall and F1 metric logic in multi-labels setting (also the binary setting). Some codes are taken and modified from sklearn.metrics .

Parameters

• num_labels (int) – The total number of labels which is usually the number of classes

• name (str, optional) – String name of the metric instance. Defaults to ‘multi_labels_metric’.

Example

import paddle
from paddlenlp.metrics import MultiLabelsMetric

x = paddle.to_tensor([[0.1, 0.2, 0.9], [0.5, 0.8, 0.5], [0.6, 1.5, 0.4], [2.8, 0.7, 0.3]])
y = paddle.to_tensor([[2], [1], [2], [1]])

m = MultiLabelsMetric(num_labels=3)
args = m.compute(x, y)
m.update(args)

result1 = m.accumulate(average=None)
# (array([0.0, 0.5, 1.0]), array([0.0, 0.5, 0.5]), array([0.0, 0.5, 0.66666667]))
result2 = m.accumulate(average='binary', pos_label=0)
# (0.0, 0.0, 0.0)
result3 = m.accumulate(average='binary', pos_label=1)
# (0.5, 0.5, 0.5)
result4 = m.accumulate(average='binary', pos_label=2)
# (1.0, 0.5, 0.6666666666666666)
result5 = m.accumulate(average='micro')
# (0.5, 0.5, 0.5)
result6 = m.accumulate(average='macro')
# (0.5, 0.3333333333333333, 0.38888888888888884)
result7 = m.accumulate(average='weighted')
# (0.75, 0.5, 0.5833333333333333)

Note: When zero_division is encountered (details as followed), the corresponding metrics will be set to 0.0

precision is zero_division if there are no positive predictions recall is zero_division if there are no positive labels fscore is zero_division if all labels AND predictions are negative

update(args)

Updates the metrics states (accuracy, precision and recall), in order to calculate accumulated accuracy, precision and recall of all instances.

Parameters

args (tuple of Tensor) – the tuple returned from compute function

accumulate(average=None, pos_label=1)

Calculates and returns the accumulated metric.

Parameters

• average (str in {‘binary’, ‘micro’, ‘macro’, ’weighted’} or None, optional) –

• to None. If None (Defaults) –

• scores for each class are returned. (the) –

• Otherwise –

• determines the type of averaging performed on the data (this) –

• binary (-) – Only report results for the class specified by pos_label.

• micro (-) – Calculate metrics globally by counting the total true positives, false negatives and false positives.

• macro (-) – Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

• weighted (-) – Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters macro to account for label imbalance; it can result in an F-score that is not between precision and recall.

• pos_label (int, optional) – The positive label for calculating precision and recall in binary settings. Noted: Only when average='binary', this arguments will be used. Otherwise, it will be ignored. Defaults to 1.

Returns

The accumulated metric. A tuple of shape (precision, recall, f1)

With the fields:

• precision (numpy.float64 or numpy.ndarray if average=None):

  The accumulated precision.

• recall (numpy.float64 or numpy.ndarray if average=None):

  The accumulated recall.

• f1 (numpy.float64 or numpy.ndarray if average=None):

  The accumulated f1.

Return type

tuple

compute(pred, label)

Accepts network’s output and the labels, and calculates the top-k (maximum value in topk) indices for accuracy.

Parameters

• pred (Tensor) – Predicted tensor, and its dtype is float32 or float64, and has a shape of [batch_size, *, num_labels].

• label (Tensor) – The ground truth tensor, and its dtype is is int64, and has a shape of [batch_size, *] or [batch_size, *, num_labels] in one hot representation.

Returns

it contains two Tensor of shape [*, 1]. The tuple should be passed to update function.

Return type

tuple of Tensor

reset()

Reset states and result

name()

Returns name of the metric instance.

Returns

The name of the metric instance.

Return type

str