modeling

class BigBirdModel(num_layers, vocab_size, nhead, attn_dropout=0.1, dim_feedforward=3072, activation='gelu', normalize_before=False, block_size=1, window_size=3, num_global_blocks=1, num_rand_blocks=2, seed=None, pad_token_id=0, hidden_size=768, hidden_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=2, **kwargs)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

The bare BigBird Model outputting raw hidden-states.

This model inherits from PretrainedModel. Refer to the superclass documentation for the generic methods.

This model is also a Paddle paddle.nn.Layer subclass. Use it as a regular Paddle Layer and refer to the Paddle documentation for all matter related to general usage and behavior.

Parameters

  • num_layers (int) – Number of hidden layers in the Transformer encoder.

  • vocab_size (int) – Vocabulary size of inputs_ids in BigBirdModel. Also is the vocab size of token embedding matrix. Defines the number of different tokens that can be represented by the inputs_ids passed when calling BigBirdModel.

  • nhead (int) – Number of attention heads for each attention layer in the Transformer encoder.

  • attn_dropout (float, optional) – The dropout probability used in MultiHeadAttention in all encoder layers to drop some attention target. Defaults to 0.1.

  • dim_feedforward (int, optional) – Dimensionality of the feed-forward (ff) layer in the Transformer encoder. Input tensors to ff layers are firstly projected from hidden_size to intermediate_size, and then projected back to hidden_size. Typically intermediate_size is larger than hidden_size. Defaults to 3072.

  • activation (str, optional) – The non-linear activation function in the feed-forward layer. "gelu", "relu", "silu" and "gelu_new" are supported. Defaults to "gelu".

  • normalize_before (bool, optional) – Indicates whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Defaults to False.

  • block_size (int, optional) – The block size for the attention mask. Defaults to 1.

  • window_size (int, optional) – The number of block in a window. Defaults to 3.

  • num_global_blocks (int, optional) – Number of global blocks per sequence. Defaults to 1.

  • num_rand_blocks (int, optional) – Number of random blocks per row. Defaults to 2.

  • seed (int, optional) – The random seed for generating random block id. Defaults to None.

  • pad_token_id (int, optional) – The index of padding token for BigBird embedding. Defaults to 0.

  • hidden_size (int, optional) – Dimensionality of the embedding layer, encoder layer and pooler layer. Defaults to 768.

  • hidden_dropout_prob (float, optional) – The dropout probability for all fully connected layers in the embeddings and encoder. Defaults to 0.1.

  • max_position_embeddings (int, optional) – The maximum value of the dimensionality of position encoding, which dictates the maximum supported length of an input sequence. Defaults to 512.

  • type_vocab_size (int, optional) – The vocabulary size of the token_type_ids. Defaults to 2.

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)[source]

The BigBirdModel forward method, overrides the __call__() special method.

Parameters

  • input_ids (Tensor) – Indices of input sequence tokens in the vocabulary. Its data type should be int64 and it has a shape of [batch_size, sequence_length].

  • token_type_ids (Tensor, optional) –

    Segment token indices to indicate first and second portions of the inputs. Indices can either be 0 or 1:

    • 0 corresponds to a sentence A token,

    • 1 corresponds to a sentence B token.

    Its data type should be int64 and it has a shape of [batch_size, sequence_length]. Defaults to None, which means we don’t add segment embeddings.

  • attention_mask_list (list, optional) – A list which contains some tensors used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. Its data type can be int, float and bool. When the data type is bool, the masked tokens have False values and the others have True values. When the data type is int, the masked tokens have 0 values and the others have 1 values. When the data type is float, the masked tokens have -INF values and the others have 0 values. It is a tensor with shape broadcasted to [batch_size, n_head, sequence_length, sequence_length]. For example, its shape can be [batch_size, sequence_length], [batch_size, sequence_length, sequence_length], [batch_size, num_attention_heads, sequence_length, sequence_length]. Defaults to None, which means nothing needed to be prevented attention to.

  • rand_mask_idx_list (list, optional) – A list which contains some tensors used in bigbird random block.

Returns

Returns tuple (encoder_output, pooled_output).

With the fields:

  • encoder_output (Tensor):

    Sequence of output at the last layer of the model. Its data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].

  • pooled_output (Tensor):

    The output of first token ([CLS]) in sequence. We “pool” the model by simply taking the hidden state corresponding to the first token. Its data type should be float32 and its shape is [batch_size, hidden_size].

Return type

tuple

Examples

import paddle
from paddlenlp.transformers import BigBirdModel, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdModel.from_pretrained('bigbird-base-uncased')
config = model.config
max_seq_len = 512
input_ids = tokenizer.convert_tokens_to_ids(
    tokenizer(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so"
    ))
input_ids.extend([0] * (max_seq_len - len(input_ids)))
seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
class BigBirdPretrainedModel(*args, **kwargs)[source]

Bases: paddlenlp.transformers.model_utils.PretrainedModel

An abstract class for pretrained BigBird models. It provides BigBird related model_config_file, pretrained_init_configuration, resource_files_names, pretrained_resource_files_map, base_model_prefix for downloading and loading pretrained models. See PretrainedModel for more details.

base_model_class

alias of paddlenlp.transformers.bigbird.modeling.BigBirdModel

class BigBirdForPretraining(bigbird)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with pretraining tasks on top.

Parameters

bigbird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids, token_type_ids=None, position_ids=None, rand_mask_idx_list=None, masked_positions=None)[source]

The BigBirdForPretraining forward method, overrides the __call__() special method.

Parameters

  • input_ids (Tensor) – See BigBirdModel.

  • token_type_ids (Tensor) – See BigBirdModel.

  • attention_mask_list (list) – See BigBirdModel.

  • rand_mask_idx_list (list) – See BigBirdModel.

  • masked_positions (list) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num]. mask_token_num is the number of masked tokens. It should be no bigger than sequence_length. Defaults to None, which means we output hidden-states of all tokens in masked token prediction.

Returns

Returns tuple (prediction_scores, seq_relationship_score).

With the fields:

  • prediction_scores (Tensor):

    The scores of masked token prediction. Its data type should be float32 and its shape is [batch_size, sequence_length, vocab_size].

  • seq_relationship_score (Tensor):

    The scores of next sentence prediction. Its data type should be float32 and its shape is [batch_size, 2].

Return type

tuple

Examples

import paddle
from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
max_seq_len = 512
input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so", max_seq_len=max_seq_len)

seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
print(output)
class BigBirdPretrainingCriterion(vocab_size, use_nsp=False, ignore_index=0)[source]

Bases: paddle.fluid.dygraph.layers.Layer

BigBird Criterion for a pretraining task on top.

Parameters

  • vocab_size (int) – See BigBirdModel.

  • use_nsp (bool, optional) – It decides whether it considers Next Sentence Prediction loss. Defaults to False.

  • ignore_index (int) – Specifies a target value that is ignored and does not contribute to the input gradient. Only valid if soft_label is set to False. Defaults to 0.

forward(prediction_scores, seq_relationship_score, masked_lm_labels, next_sentence_labels, masked_lm_scale, masked_lm_weights)[source]

The BigBirdPretrainingCriterion forward method, overrides the __call__() special method.

Parameters

  • prediction_scores (Tensor) – The scores of masked token prediction. Its data type should be float32. If masked_positions is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].

  • seq_relationship_score (Tensor) – The scores of next sentence prediction. Its data type should be float32 and its shape is [batch_size, 2].

  • masked_lm_labels (Tensor) – The labels of the masked language modeling, its dimensionality is equal to prediction_scores. Its data type should be int64. If masked_positions is None, its shape is [batch_size, sequence_length, 1]. Otherwise, its shape is [batch_size, mask_token_num, 1].

  • next_sentence_labels (Tensor) – The labels of the next sentence prediction task, the dimensionality of next_sentence_labels is equal to seq_relation_labels. Its data type should be int64 and its shape is [batch_size, 1].

  • masked_lm_scale (Tensor or int) – The scale of masked tokens. Used for the normalization of masked language modeling loss. If it is a Tensor, its data type should be int64 and its shape is equal to prediction_scores.

  • masked_lm_weights (Tensor) – The weight of masked tokens. Its data type should be float32 and its shape is [mask_token_num, 1].

Returns

The pretraining loss, equals to the sum of masked_lm_loss plus the mean of next_sentence_loss. Its data type should be float32 and its shape is [1].

Return type

Tensor

Example

import numpy as np
import paddle
from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer, BigBirdPretrainingCriterion
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
criterion = BigBirdPretrainingCriterion(config["vocab_size"], False)
max_seq_len = 512
max_pred_length=75
input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so", max_seq_len=max_seq_len, max_pred_len=max_pred_length)

seq_len = len(input_ids)
masked_lm_positions_tmp = np.full(seq_len, 0, dtype=np.int32)
masked_lm_ids_tmp = np.full([seq_len, 1], -1, dtype=np.int64)
masked_lm_weights_tmp = np.full([seq_len], 0, dtype="float32")

mask_token_num = 0
for i, x in enumerate([input_ids]):
    for j, pos in enumerate(masked_lm_positions):
        masked_lm_positions_tmp[mask_token_num] = i * seq_len + pos
        masked_lm_ids_tmp[mask_token_num] = masked_lm_ids[j]
        masked_lm_weights_tmp[mask_token_num] = masked_lm_weights[j]

masked_lm_positions = masked_lm_positions_tmp
masked_lm_ids = masked_lm_ids_tmp
masked_lm_weights = masked_lm_weights_tmp
print(masked_lm_ids.shape)
input_ids = paddle.to_tensor([input_ids])
masked_lm_positions = paddle.to_tensor(masked_lm_positions)
masked_lm_ids = paddle.to_tensor(masked_lm_ids, dtype='int64')
masked_lm_weights = paddle.to_tensor(masked_lm_weights)
masked_lm_scale = 1.0
next_sentence_labels = paddle.zeros(shape=(1, 1), dtype='int64')

rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
prediction_scores, seq_relationship_score = model(input_ids, rand_mask_idx_list=rand_mask_idx_list, masked_positions=masked_lm_positions)

loss = criterion(prediction_scores, seq_relationship_score,
                masked_lm_ids, next_sentence_labels,
                masked_lm_scale, masked_lm_weights)
print(loss)
class BigBirdForSequenceClassification(bigbird, num_classes=None)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the output layer, designed for sequence classification/regression tasks like GLUE tasks.

Parameters

  • bigbird (BigBirdModel) – An instance of BigBirdModel.

  • num_classes (int, optional) – The number of classes. Defaults to None.

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)[source]

The BigBirdForSequenceClassification forward method, overrides the __call__() special method.

Parameters
Returns

Returns tensor output, a tensor of the input text classification logits. Its data type should be float32 and it has a shape of [batch_size, num_classes].

Return type

Tensor

Examples

import paddle
from paddlenlp.transformers import BigBirdForSequenceClassification, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForSequenceClassification.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
max_seq_len = 512
input_ids = tokenizer.convert_tokens_to_ids(
    tokenizer(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so"
    ))
input_ids.extend([0] * (max_seq_len - len(input_ids)))
seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
print(output)
class BigBirdPretrainingHeads(hidden_size, vocab_size, activation, embedding_weights=None)[source]

Bases: paddle.fluid.dygraph.layers.Layer

The BigBird pretraining heads for a pretraining task.

Parameters

  • hidden_size (int) – See BigBirdModel.

  • vocab_size (int) – See BigBirdModel.

  • activation (str) – See BigBirdModel.

  • embedding_weights (Tensor, optional) – The weight of pretraining embedding layer. Its data type should be float32 and its shape is [hidden_size, vocab_size]. If set to None, use normal distribution to initialize weight. Defaults to None.

forward(sequence_output, pooled_output, masked_positions=None)[source]

The BigBirdPretrainingHeads forward method, overrides the __call__() special method.

Parameters

  • sequence_output (Tensor) – The sequence output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].

  • pooled_output (Tensor) – The pooled output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, hidden_size].

  • masked_positions (Tensor) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num]. mask_token_num is the number of masked tokens. It should be no bigger than sequence_length. Defaults to None, which means we output hidden-states of all tokens in masked token prediction.

Returns

(prediction_scores, seq_relationship_score).

With the fields:

  • prediction_scores (Tensor):

    The scores of masked token prediction. Its data type should be float32. If masked_positions is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].

  • seq_relationship_score (Tensor):

    The logits whether 2 sequences are NSP relationship. Its data type should be float32 and has a shape of [batch_size, 2].

Return type

tuple

class BigBirdForQuestionAnswering(bigbird, dropout=None)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output to compute span_start_logits and span_end_logits, designed for question-answering tasks like SQuAD.

Parameters

  • bigbird (BigBirdModel) – An instance of BigBirdModel.

  • dropout (float, optional) – The dropout probability for output of BigBirdModel. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)[source]

The BigBirdForQuestionAnswering forward method, overrides the __call__() special method.

Parameters
Returns

Returns tuple (start_logits, end_logits).

With the fields:

  • start_logits (Tensor):

    A tensor of the input token classification logits, indicates the start position of the labelled span. Its data type should be float32 and its shape is [batch_size, sequence_length].

  • end_logits (Tensor):

    A tensor of the input token classification logits, indicates the end position of the labelled span. Its data type should be float32 and its shape is [batch_size, sequence_length].

Return type

tuple

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForQuestionAnswering
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForQuestionAnswering.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

start_logits = outputs[0]
end_logits  =outputs[1]
class BigBirdForTokenClassification(bigbird, num_classes=2, dropout=None)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output layer, designed for token classification tasks like NER tasks.

Parameters

  • bigbird (BigBirdModel) – An instance of BigBirdModel.

  • num_classes (int, optional) – The number of classes. Defaults to 2.

  • dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)[source]

The BigBirdForSequenceClassification forward method, overrides the __call__() special method.

Parameters
Returns

Returns tensor logits, a tensor of the input token classification logits. Shape as [batch_size, sequence_length, num_classes] and dtype as float32.

Return type

Tensor

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForTokenClassification
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

logits = outputs
class BigBirdForMultipleChoice(bigbird, num_choices=2, dropout=None)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output layer, designed for multiple choice tasks like RocStories/SWAG tasks .

Parameters

  • bigbird (BigBirdModel) – An instance of BigBirdModel.

  • num_choices (int, optional) – The number of choices. Defaults to 2.

  • dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids, attention_mask_list=None, rand_mask_idx_list=None)[source]

The BigBirdForMultipleChoice forward method, overrides the __call__() special method.

Parameters

  • input_ids (Tensor) – See BigBirdModel and shape as [batch_size, num_choice, sequence_length].

  • attention_mask_list (List) – See BigBirdModel and shape as [batch_size, num_choice, n_head, sequence_length, sequence_length].

  • rand_mask_idx_list (List) – See BigBirdModel.

Returns

Returns tensor logits, a tensor of the input text classification logits. Shape as [batch_size, 1] and dtype as float32.

Return type

Tensor

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForMultipleChoice
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

logits = outputs
class BigBirdForMaskedLM(bigbird)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a language modeling head on top.

Parameters

BigBird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids, attention_mask_list=None, rand_mask_idx_list=None, labels=None)[source]
Parameters

  • input_ids (Tensor) – See BigBirdModel.

  • attention_mask_list (List) – See BigBirdModel.

  • rand_mask_idx_list (List) – See BigBirdModel.

  • labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., vocab_size] Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., vocab_size] Its shape is [batch_size, sequence_length].

Returns

Returns tuple (masked_lm_loss, prediction_scores, ``sequence_output`).

With the fields:

  • masked_lm_loss (Tensor):

    The masked lm loss. Its data type should be float32 and its shape is [1].

  • prediction_scores (Tensor):

    The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].

  • sequence_output (Tensor):

    Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is [batch_size, sequence_length, hidden_size].

Return type

tuple

class BigBirdForCausalLM(bigbird)[source]

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model for casual language model tasks.

Parameters

BigBird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids, attention_mask_list=None, rand_mask_idx_list=None, labels=None)[source]
Parameters

  • input_ids (Tensor) – See BigBirdModel.

  • attention_mask_list (List) – See BigBirdModel.

  • rand_mask_idx_list (List) – See BigBirdModel.

  • labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., vocab_size] Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., vocab_size] Its shape is [batch_size, sequence_length].

Returns

Returns tuple (masked_lm_loss, prediction_scores, ``sequence_output`).

With the fields:

  • masked_lm_loss (Tensor):

    The masked lm loss. Its data type should be float32 and its shape is [1].

  • prediction_scores (Tensor):

    The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].

  • sequence_output (Tensor):

    Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is [batch_size, sequence_length, hidden_size].

Return type

tuple