R-Drop Loss implementation For more information about R-drop please refer to this paper: https://arxiv.org/abs/2106.14448 Original implementation please refer to this code: https://github.com/dropreg/R-Drop
reduction (str, optional) – Indicate how to average the loss, the candicates are
. If `reduction` is ``'mean', the reduced mean loss is returned; If
'batchmean', the sum loss divided by batch size is returned; If
'sum', the reduced sum loss is returned; If
'none', no reduction will be applied. Defaults to
forward(p, q, pad_mask=None)¶
p (Tensor) – the first forward logits of training examples.
q (Tensor) – the second forward logits of training examples.
pad_mask (Tensor, optional) – The Tensor containing the binary mask to index with, it’s data type is bool.
loss, the rdrop loss of p and q.
- Return type